This is python.info, produced by makeinfo version 7.1 from python.texi.
Python 3.13.7, August 14, 2025
Author name not set
Copyright © 2001-2025, Python Software Foundation
INFO-DIR-SECTION Miscellaneous
START-INFO-DIR-ENTRY
* python: (python.info). One line description of project
END-INFO-DIR-ENTRY
Generated by Sphinx 8.2.3.
File: python.info, Node: Top, Next: What’s New in Python, Up: (dir)
Python
******
Python 3.13.7, August 14, 2025
Author name not set
Copyright © 2001-2025, Python Software Foundation
* Menu:
* What’s New in Python::
* The Python Tutorial::
* Python Setup and Usage::
* The Python Language Reference::
* The Python Standard Library::
* Extending and Embedding the Python Interpreter::
* Python/C API Reference Manual::
* Installing Python Modules::
* Python HOWTOs::
* Python Frequently Asked Questions::
* Deprecations::
* Glossary::
* About this documentation::
* Dealing with Bugs::
* Copyright::
* History and License::
* Python Module Index::
* Index::
-- The Detailed Node Listing --
What’s New in Python
* What’s New In Python 3.13: What’s New In Python 3 13.
* What’s New In Python 3.12: What’s New In Python 3 12.
* What’s New In Python 3.11: What’s New In Python 3 11.
* What’s New In Python 3.10: What’s New In Python 3 10.
* What’s New In Python 3.9: What’s New In Python 3 9.
* What’s New In Python 3.8: What’s New In Python 3 8.
* What’s New In Python 3.7: What’s New In Python 3 7.
* What’s New In Python 3.6: What’s New In Python 3 6.
* What’s New In Python 3.5: What’s New In Python 3 5.
* What’s New In Python 3.4: What’s New In Python 3 4.
* What’s New In Python 3.3: What’s New In Python 3 3.
* What’s New In Python 3.2: What’s New In Python 3 2.
* What’s New In Python 3.1: What’s New In Python 3 1.
* What’s New In Python 3.0: What’s New In Python 3 0.
* What’s New in Python 2.7: What’s New in Python 2 7.
* What’s New in Python 2.6: What’s New in Python 2 6.
* What’s New in Python 2.5: What’s New in Python 2 5.
* What’s New in Python 2.4: What’s New in Python 2 4.
* What’s New in Python 2.3: What’s New in Python 2 3.
* What’s New in Python 2.2: What’s New in Python 2 2.
* What’s New in Python 2.1: What’s New in Python 2 1.
* What’s New in Python 2.0: What’s New in Python 2 0.
* Changelog::
What’s New In Python 3.13
* Summary – Release Highlights::
* New Features::
* Other Language Changes::
* New Modules::
* Improved Modules::
* Optimizations::
* Removed Modules And APIs::
* New Deprecations::
* CPython Bytecode Changes::
* C API Changes::
* Build Changes::
* Porting to Python 3.13: Porting to Python 3 13.
* Regression Test Changes::
* Notable changes in 3.13.1: Notable changes in 3 13 1.
* Notable changes in 3.13.4: Notable changes in 3 13 4.
New Features
* A better interactive interpreter::
* Improved error messages::
* Free-threaded CPython::
* An experimental just-in-time (JIT) compiler: An experimental just-in-time JIT compiler.
* Defined mutation semantics for locals(): Defined mutation semantics for locals.
* Support for mobile platforms::
Improved Modules
* argparse::
* array::
* ast::
* asyncio::
* base64::
* compileall::
* concurrent.futures: concurrent futures.
* configparser::
* copy::
* ctypes::
* dbm::
* dis::
* doctest::
* email::
* enum::
* fractions::
* glob::
* importlib::
* io::
* ipaddress::
* itertools::
* marshal::
* math::
* mimetypes::
* mmap::
* multiprocessing::
* os::
* os.path: os path.
* pathlib::
* pdb::
* queue::
* random::
* re::
* shutil::
* site::
* sqlite3::
* ssl::
* statistics::
* subprocess::
* sys::
* tempfile::
* time::
* tkinter::
* traceback::
* types::
* typing::
* unicodedata::
* venv::
* warnings::
* xml::
* zipimport::
Removed Modules And APIs
* PEP 594; Remove “dead batteries” from the standard library: PEP 594 Remove “dead batteries” from the standard library.
* 2to3::
* builtins::
* configparser: configparser<2>.
* importlib.metadata: importlib metadata.
* locale::
* opcode::
* optparse::
* pathlib: pathlib<2>.
* re: re<2>.
* tkinter.tix: tkinter tix.
* turtle::
* typing: typing<2>.
* unittest::
* urllib::
* webbrowser::
New Deprecations
* Pending Removal in Python 3.14: Pending Removal in Python 3 14.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15.
* Pending removal in Python 3.16: Pending removal in Python 3 16.
* Pending Removal in Future Versions::
C API Changes
* New Features: New Features<2>.
* Changed C APIs::
* Limited C API Changes::
* Removed C APIs::
* Deprecated C APIs::
Deprecated C APIs
* Pending Removal in Python 3.14: Pending Removal in Python 3 14<2>.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15<2>.
* Pending removal in Python 3.16: Pending removal in Python 3 16<2>.
* Pending Removal in Future Versions: Pending Removal in Future Versions<2>.
Porting to Python 3.13
* Changes in the Python API::
* Changes in the C API::
Notable changes in 3.13.1
* sys: sys<2>.
Notable changes in 3.13.4
* os.path: os path<2>.
* tarfile::
What’s New In Python 3.12
* Summary – Release highlights::
* New Features: New Features<3>.
* New Features Related to Type Hints::
* Other Language Changes: Other Language Changes<2>.
* New Modules: New Modules<2>.
* Improved Modules: Improved Modules<2>.
* Optimizations: Optimizations<2>.
* CPython bytecode changes::
* Demos and Tools::
* Deprecated::
* Removed::
* Porting to Python 3.12: Porting to Python 3 12.
* Build Changes: Build Changes<2>.
* C API Changes: C API Changes<2>.
New Features
* PEP 695; Type Parameter Syntax: PEP 695 Type Parameter Syntax.
* PEP 701; Syntactic formalization of f-strings: PEP 701 Syntactic formalization of f-strings.
* PEP 684; A Per-Interpreter GIL: PEP 684 A Per-Interpreter GIL.
* PEP 669; Low impact monitoring for CPython: PEP 669 Low impact monitoring for CPython.
* PEP 688; Making the buffer protocol accessible in Python: PEP 688 Making the buffer protocol accessible in Python.
* PEP 709; Comprehension inlining: PEP 709 Comprehension inlining.
* Improved Error Messages::
New Features Related to Type Hints
* PEP 692; Using TypedDict for more precise **kwargs typing: PEP 692 Using TypedDict for more precise **kwargs typing.
* PEP 698; Override Decorator for Static Typing: PEP 698 Override Decorator for Static Typing.
Improved Modules
* array: array<2>.
* asyncio: asyncio<2>.
* calendar::
* csv::
* dis: dis<2>.
* fractions: fractions<2>.
* importlib.resources: importlib resources.
* inspect::
* itertools: itertools<2>.
* math: math<2>.
* os: os<2>.
* os.path: os path<3>.
* pathlib: pathlib<3>.
* platform::
* pdb: pdb<2>.
* random: random<2>.
* shutil: shutil<2>.
* sqlite3: sqlite3<2>.
* statistics: statistics<2>.
* sys: sys<3>.
* tempfile: tempfile<2>.
* threading::
* tkinter: tkinter<2>.
* tokenize::
* types: types<2>.
* typing: typing<3>.
* unicodedata: unicodedata<2>.
* unittest: unittest<2>.
* uuid::
Deprecated
* Pending Removal in Python 3.13: Pending Removal in Python 3 13.
* Pending Removal in Python 3.14: Pending Removal in Python 3 14<3>.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15<3>.
* Pending removal in Python 3.16: Pending removal in Python 3 16<3>.
* Pending Removal in Future Versions: Pending Removal in Future Versions<3>.
Removed
* asynchat and asyncore::
* configparser: configparser<3>.
* distutils::
* ensurepip::
* enum: enum<2>.
* ftplib::
* gzip::
* hashlib::
* importlib: importlib<2>.
* imp::
* io: io<2>.
* locale: locale<2>.
* smtpd::
* sqlite3: sqlite3<3>.
* ssl: ssl<2>.
* unittest: unittest<3>.
* webbrowser: webbrowser<2>.
* xml.etree.ElementTree: xml etree ElementTree.
* zipimport: zipimport<2>.
* Others::
Porting to Python 3.12
* Changes in the Python API: Changes in the Python API<2>.
C API Changes
* New Features: New Features<4>.
* Porting to Python 3.12: Porting to Python 3 12<2>.
* Deprecated: Deprecated<2>.
* Removed: Removed<2>.
Deprecated
* Pending Removal in Python 3.14: Pending Removal in Python 3 14<4>.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15<4>.
* Pending removal in Python 3.16: Pending removal in Python 3 16<4>.
* Pending Removal in Future Versions: Pending Removal in Future Versions<4>.
What’s New In Python 3.11
* Summary – Release highlights: Summary – Release highlights<2>.
* New Features: New Features<5>.
* New Features Related to Type Hints: New Features Related to Type Hints<2>.
* Other Language Changes: Other Language Changes<3>.
* Other CPython Implementation Changes::
* New Modules: New Modules<3>.
* Improved Modules: Improved Modules<3>.
* Optimizations: Optimizations<3>.
* Faster CPython::
* CPython bytecode changes: CPython bytecode changes<2>.
* Deprecated: Deprecated<3>.
* Pending Removal in Python 3.12: Pending Removal in Python 3 12.
* Removed: Removed<3>.
* Porting to Python 3.11: Porting to Python 3 11.
* Build Changes: Build Changes<3>.
* C API Changes: C API Changes<3>.
* Notable changes in 3.11.4: Notable changes in 3 11 4.
* Notable changes in 3.11.5: Notable changes in 3 11 5.
New Features
* PEP 657; Fine-grained error locations in tracebacks: PEP 657 Fine-grained error locations in tracebacks.
* PEP 654; Exception Groups and except*: PEP 654 Exception Groups and except*.
* PEP 678; Exceptions can be enriched with notes: PEP 678 Exceptions can be enriched with notes.
* Windows py.exe launcher improvements: Windows py exe launcher improvements.
New Features Related to Type Hints
* PEP 646; Variadic generics: PEP 646 Variadic generics.
* PEP 655; Marking individual TypedDict items as required or not-required: PEP 655 Marking individual TypedDict items as required or not-required.
* PEP 673; Self type: PEP 673 Self type.
* PEP 675; Arbitrary literal string type: PEP 675 Arbitrary literal string type.
* PEP 681; Data class transforms: PEP 681 Data class transforms.
* PEP 563 may not be the future::
Improved Modules
* asyncio: asyncio<3>.
* contextlib::
* dataclasses::
* datetime::
* enum: enum<3>.
* fcntl::
* fractions: fractions<3>.
* functools::
* gzip: gzip<2>.
* hashlib: hashlib<2>.
* IDLE and idlelib::
* inspect: inspect<2>.
* locale: locale<3>.
* logging::
* math: math<3>.
* operator::
* os: os<3>.
* pathlib: pathlib<4>.
* re: re<3>.
* shutil: shutil<3>.
* socket::
* sqlite3: sqlite3<4>.
* string::
* sys: sys<4>.
* sysconfig::
* tempfile: tempfile<3>.
* threading: threading<2>.
* time: time<2>.
* tkinter: tkinter<3>.
* traceback: traceback<2>.
* typing: typing<4>.
* unicodedata: unicodedata<3>.
* unittest: unittest<4>.
* venv: venv<2>.
* warnings: warnings<2>.
* zipfile::
Faster CPython
* Faster Startup::
* Faster Runtime::
* Misc::
* FAQ::
* About::
Faster Startup
* Frozen imports / Static code objects::
Faster Runtime
* Cheaper, lazy Python frames: Cheaper lazy Python frames.
* Inlined Python function calls::
* PEP 659; Specializing Adaptive Interpreter: PEP 659 Specializing Adaptive Interpreter.
FAQ
* How should I write my code to utilize these speedups?::
* Will CPython 3.11 use more memory?: Will CPython 3 11 use more memory?.
* I don’t see any speedups in my workload. Why?: I don’t see any speedups in my workload Why?.
* Is there a JIT compiler?::
CPython bytecode changes
* New opcodes::
* Replaced opcodes::
* Changed/removed opcodes::
Deprecated
* Language/Builtins::
* Modules::
* Standard Library::
C API Changes
* New Features: New Features<6>.
* Porting to Python 3.11: Porting to Python 3 11<2>.
* Deprecated: Deprecated<4>.
* Pending Removal in Python 3.12: Pending Removal in Python 3 12<2>.
* Removed: Removed<4>.
Notable changes in 3.11.4
* tarfile: tarfile<2>.
Notable changes in 3.11.5
* OpenSSL::
What’s New In Python 3.10
* Summary – Release highlights: Summary – Release highlights<3>.
* New Features: New Features<7>.
* New Features Related to Type Hints: New Features Related to Type Hints<3>.
* Other Language Changes: Other Language Changes<4>.
* New Modules: New Modules<4>.
* Improved Modules: Improved Modules<4>.
* Optimizations: Optimizations<4>.
* Deprecated: Deprecated<5>.
* Removed: Removed<5>.
* Porting to Python 3.10: Porting to Python 3 10.
* CPython bytecode changes: CPython bytecode changes<3>.
* Build Changes: Build Changes<4>.
* C API Changes: C API Changes<4>.
* Notable security feature in 3.10.7: Notable security feature in 3 10 7.
* Notable security feature in 3.10.8: Notable security feature in 3 10 8.
* Notable changes in 3.10.12: Notable changes in 3 10 12.
New Features
* Parenthesized context managers::
* Better error messages::
* PEP 626; Precise line numbers for debugging and other tools: PEP 626 Precise line numbers for debugging and other tools.
* PEP 634; Structural Pattern Matching: PEP 634 Structural Pattern Matching.
* Optional EncodingWarning and encoding="locale" option::
Better error messages
* SyntaxErrors::
* IndentationErrors::
* AttributeErrors::
* NameErrors::
PEP 634: Structural Pattern Matching
* Syntax and operations::
* Declarative approach::
* Simple pattern; match to a literal: Simple pattern match to a literal.
* Patterns with a literal and variable::
* Patterns and classes::
* Nested patterns::
* Complex patterns and the wildcard::
* Guard::
* Other Key Features::
Simple pattern: match to a literal
* Behavior without the wildcard::
Patterns and classes
* Patterns with positional parameters::
New Features Related to Type Hints
* PEP 604; New Type Union Operator: PEP 604 New Type Union Operator.
* PEP 612; Parameter Specification Variables: PEP 612 Parameter Specification Variables.
* PEP 613; TypeAlias: PEP 613 TypeAlias.
* PEP 647; User-Defined Type Guards: PEP 647 User-Defined Type Guards.
Improved Modules
* asyncio: asyncio<4>.
* argparse: argparse<2>.
* array: array<3>.
* asynchat, asyncore, smtpd: asynchat asyncore smtpd.
* base64: base64<2>.
* bdb::
* bisect::
* codecs::
* collections.abc: collections abc.
* contextlib: contextlib<2>.
* curses::
* dataclasses: dataclasses<2>.
* distutils: distutils<2>.
* doctest: doctest<2>.
* encodings::
* enum: enum<4>.
* fileinput::
* faulthandler::
* gc::
* glob: glob<2>.
* hashlib: hashlib<3>.
* hmac::
* IDLE and idlelib: IDLE and idlelib<2>.
* importlib.metadata: importlib metadata<2>.
* inspect: inspect<3>.
* itertools: itertools<3>.
* linecache::
* os: os<4>.
* os.path: os path<4>.
* pathlib: pathlib<5>.
* platform: platform<2>.
* pprint::
* py_compile::
* pyclbr::
* shelve::
* statistics: statistics<3>.
* site: site<2>.
* socket: socket<2>.
* ssl: ssl<3>.
* sqlite3: sqlite3<5>.
* sys: sys<5>.
* _thread::
* threading: threading<3>.
* traceback: traceback<3>.
* types: types<3>.
* typing: typing<5>.
* unittest: unittest<5>.
* urllib.parse: urllib parse.
* xml: xml<2>.
* zipimport: zipimport<3>.
dataclasses
* __slots__::
* Keyword-only fields::
Porting to Python 3.10
* Changes in the Python syntax::
* Changes in the Python API: Changes in the Python API<3>.
* Changes in the C API: Changes in the C API<2>.
C API Changes
* PEP 652; Maintaining the Stable ABI: PEP 652 Maintaining the Stable ABI.
* New Features: New Features<8>.
* Porting to Python 3.10: Porting to Python 3 10<2>.
* Deprecated: Deprecated<6>.
* Removed: Removed<6>.
Notable changes in 3.10.12
* tarfile: tarfile<3>.
What’s New In Python 3.9
* Summary – Release highlights: Summary – Release highlights<4>.
* You should check for DeprecationWarning in your code::
* New Features: New Features<9>.
* Other Language Changes: Other Language Changes<5>.
* New Modules: New Modules<5>.
* Improved Modules: Improved Modules<5>.
* Optimizations: Optimizations<5>.
* Deprecated: Deprecated<7>.
* Removed: Removed<7>.
* Porting to Python 3.9: Porting to Python 3 9.
* Build Changes: Build Changes<5>.
* C API Changes: C API Changes<5>.
* Notable changes in Python 3.9.1: Notable changes in Python 3 9 1.
* Notable changes in Python 3.9.2: Notable changes in Python 3 9 2.
* Notable changes in Python 3.9.3: Notable changes in Python 3 9 3.
* Notable changes in Python 3.9.5: Notable changes in Python 3 9 5.
* Notable security feature in 3.9.14: Notable security feature in 3 9 14.
* Notable changes in 3.9.17: Notable changes in 3 9 17.
New Features
* Dictionary Merge & Update Operators::
* New String Methods to Remove Prefixes and Suffixes::
* Type Hinting Generics in Standard Collections::
* New Parser::
New Modules
* zoneinfo::
* graphlib::
Improved Modules
* ast: ast<2>.
* asyncio: asyncio<5>.
* compileall: compileall<2>.
* concurrent.futures: concurrent futures<2>.
* curses: curses<2>.
* datetime: datetime<2>.
* distutils: distutils<3>.
* fcntl: fcntl<2>.
* ftplib: ftplib<2>.
* gc: gc<2>.
* hashlib: hashlib<4>.
* http::
* IDLE and idlelib: IDLE and idlelib<3>.
* imaplib::
* importlib: importlib<3>.
* inspect: inspect<4>.
* ipaddress: ipaddress<2>.
* math: math<4>.
* multiprocessing: multiprocessing<2>.
* nntplib::
* os: os<5>.
* pathlib: pathlib<6>.
* pdb: pdb<3>.
* poplib::
* pprint: pprint<2>.
* pydoc::
* random: random<3>.
* signal::
* smtplib::
* socket: socket<3>.
* time: time<3>.
* sys: sys<6>.
* tracemalloc::
* typing: typing<6>.
* unicodedata: unicodedata<4>.
* venv: venv<3>.
* xml: xml<3>.
Porting to Python 3.9
* Changes in the Python API: Changes in the Python API<4>.
* Changes in the C API: Changes in the C API<3>.
* CPython bytecode changes: CPython bytecode changes<4>.
C API Changes
* New Features: New Features<10>.
* Porting to Python 3.9: Porting to Python 3 9<2>.
* Removed: Removed<8>.
Notable changes in Python 3.9.1
* typing: typing<7>.
* macOS 11.0 (Big Sur) and Apple Silicon Mac support: macOS 11 0 Big Sur and Apple Silicon Mac support.
Notable changes in Python 3.9.2
* collections.abc: collections abc<2>.
* urllib.parse: urllib parse<2>.
Notable changes in Python 3.9.5
* urllib.parse: urllib parse<3>.
Notable changes in 3.9.17
* tarfile: tarfile<4>.
What’s New In Python 3.8
* Summary – Release highlights: Summary – Release highlights<5>.
* New Features: New Features<11>.
* Other Language Changes: Other Language Changes<6>.
* New Modules: New Modules<6>.
* Improved Modules: Improved Modules<6>.
* Optimizations: Optimizations<6>.
* Build and C API Changes::
* Deprecated: Deprecated<8>.
* API and Feature Removals::
* Porting to Python 3.8: Porting to Python 3 8.
* Notable changes in Python 3.8.1: Notable changes in Python 3 8 1.
* Notable changes in Python 3.8.2: Notable changes in Python 3 8 2.
* Notable changes in Python 3.8.3: Notable changes in Python 3 8 3.
* Notable changes in Python 3.8.8: Notable changes in Python 3 8 8.
* Notable changes in Python 3.8.9: Notable changes in Python 3 8 9.
* Notable changes in Python 3.8.10: Notable changes in Python 3 8 10.
* Notable changes in Python 3.8.10: Notable changes in Python 3 8 10<2>.
* Notable changes in Python 3.8.12: Notable changes in Python 3 8 12.
* Notable security feature in 3.8.14: Notable security feature in 3 8 14.
* Notable changes in 3.8.17: Notable changes in 3 8 17.
New Features
* Assignment expressions::
* Positional-only parameters::
* Parallel filesystem cache for compiled bytecode files::
* Debug build uses the same ABI as release build::
* f-strings support = for self-documenting expressions and debugging::
* PEP 578; Python Runtime Audit Hooks: PEP 578 Python Runtime Audit Hooks.
* PEP 587; Python Initialization Configuration: PEP 587 Python Initialization Configuration.
* PEP 590; Vectorcall; a fast calling protocol for CPython: PEP 590 Vectorcall a fast calling protocol for CPython.
* Pickle protocol 5 with out-of-band data buffers::
Improved Modules
* ast: ast<3>.
* asyncio: asyncio<6>.
* builtins: builtins<2>.
* collections::
* cProfile::
* csv: csv<2>.
* curses: curses<3>.
* ctypes: ctypes<2>.
* datetime: datetime<3>.
* functools: functools<2>.
* gc: gc<3>.
* gettext::
* gzip: gzip<3>.
* IDLE and idlelib: IDLE and idlelib<4>.
* inspect: inspect<5>.
* io: io<3>.
* itertools: itertools<4>.
* json.tool: json tool.
* logging: logging<2>.
* math: math<5>.
* mmap: mmap<2>.
* multiprocessing: multiprocessing<3>.
* os: os<6>.
* os.path: os path<5>.
* pathlib: pathlib<7>.
* pickle::
* plistlib::
* pprint: pprint<3>.
* py_compile: py_compile<2>.
* shlex::
* shutil: shutil<4>.
* socket: socket<4>.
* ssl: ssl<4>.
* statistics: statistics<4>.
* sys: sys<7>.
* tarfile: tarfile<5>.
* threading: threading<4>.
* tokenize: tokenize<2>.
* tkinter: tkinter<4>.
* time: time<4>.
* typing: typing<8>.
* unicodedata: unicodedata<5>.
* unittest: unittest<6>.
* venv: venv<4>.
* weakref::
* xml: xml<4>.
* xmlrpc::
Porting to Python 3.8
* Changes in Python behavior::
* Changes in the Python API: Changes in the Python API<5>.
* Changes in the C API: Changes in the C API<4>.
* CPython bytecode changes: CPython bytecode changes<5>.
* Demos and Tools: Demos and Tools<2>.
Notable changes in Python 3.8.10
* macOS 11.0 (Big Sur) and Apple Silicon Mac support: macOS 11 0 Big Sur and Apple Silicon Mac support<2>.
Notable changes in Python 3.8.10
* urllib.parse: urllib parse<4>.
Notable changes in Python 3.8.12
* Changes in the Python API: Changes in the Python API<6>.
Notable changes in 3.8.17
* tarfile: tarfile<6>.
What’s New In Python 3.7
* Summary – Release Highlights: Summary – Release Highlights<2>.
* New Features: New Features<12>.
* Other Language Changes: Other Language Changes<7>.
* New Modules: New Modules<7>.
* Improved Modules: Improved Modules<7>.
* C API Changes: C API Changes<6>.
* Build Changes: Build Changes<6>.
* Optimizations: Optimizations<7>.
* Other CPython Implementation Changes: Other CPython Implementation Changes<2>.
* Deprecated Python Behavior::
* Deprecated Python modules, functions and methods: Deprecated Python modules functions and methods.
* Deprecated functions and types of the C API::
* Platform Support Removals::
* API and Feature Removals: API and Feature Removals<2>.
* Module Removals::
* Windows-only Changes::
* Porting to Python 3.7: Porting to Python 3 7.
* Notable changes in Python 3.7.1: Notable changes in Python 3 7 1.
* Notable changes in Python 3.7.2: Notable changes in Python 3 7 2.
* Notable changes in Python 3.7.6: Notable changes in Python 3 7 6.
* Notable changes in Python 3.7.10: Notable changes in Python 3 7 10.
* Notable changes in Python 3.7.11: Notable changes in Python 3 7 11.
* Notable security feature in 3.7.14: Notable security feature in 3 7 14.
New Features
* PEP 563; Postponed Evaluation of Annotations: PEP 563 Postponed Evaluation of Annotations.
* PEP 538; Legacy C Locale Coercion: PEP 538 Legacy C Locale Coercion.
* PEP 540; Forced UTF-8 Runtime Mode: PEP 540 Forced UTF-8 Runtime Mode.
* PEP 553; Built-in breakpoint(): PEP 553 Built-in breakpoint.
* PEP 539; New C API for Thread-Local Storage: PEP 539 New C API for Thread-Local Storage.
* PEP 562; Customization of Access to Module Attributes: PEP 562 Customization of Access to Module Attributes.
* PEP 564; New Time Functions With Nanosecond Resolution: PEP 564 New Time Functions With Nanosecond Resolution.
* PEP 565; Show DeprecationWarning in __main__: PEP 565 Show DeprecationWarning in __main__.
* PEP 560; Core Support for typing module and Generic Types: PEP 560 Core Support for typing module and Generic Types.
* PEP 552; Hash-based .pyc Files: PEP 552 Hash-based pyc Files.
* PEP 545; Python Documentation Translations: PEP 545 Python Documentation Translations.
* Python Development Mode (-X dev): Python Development Mode -X dev.
New Modules
* contextvars::
* dataclasses: dataclasses<3>.
* importlib.resources: importlib resources<2>.
Improved Modules
* argparse: argparse<3>.
* asyncio: asyncio<7>.
* binascii::
* calendar: calendar<2>.
* collections: collections<2>.
* compileall: compileall<3>.
* concurrent.futures: concurrent futures<3>.
* contextlib: contextlib<3>.
* cProfile: cProfile<2>.
* crypt::
* datetime: datetime<4>.
* dbm: dbm<2>.
* decimal::
* dis: dis<3>.
* distutils: distutils<4>.
* enum: enum<5>.
* functools: functools<3>.
* gc: gc<4>.
* hmac: hmac<2>.
* http.client: http client.
* http.server: http server.
* idlelib and IDLE::
* importlib: importlib<4>.
* io: io<4>.
* ipaddress: ipaddress<3>.
* itertools: itertools<5>.
* locale: locale<4>.
* logging: logging<3>.
* math: math<6>.
* mimetypes: mimetypes<2>.
* msilib::
* multiprocessing: multiprocessing<4>.
* os: os<7>.
* pathlib: pathlib<8>.
* pdb: pdb<4>.
* py_compile: py_compile<3>.
* pydoc: pydoc<2>.
* queue: queue<2>.
* re: re<4>.
* signal: signal<2>.
* socket: socket<5>.
* socketserver::
* sqlite3: sqlite3<6>.
* ssl: ssl<5>.
* string: string<2>.
* subprocess: subprocess<2>.
* sys: sys<8>.
* time: time<5>.
* tkinter: tkinter<5>.
* tracemalloc: tracemalloc<2>.
* types: types<4>.
* unicodedata: unicodedata<6>.
* unittest: unittest<7>.
* unittest.mock: unittest mock.
* urllib.parse: urllib parse<5>.
* uu::
* uuid: uuid<2>.
* warnings: warnings<3>.
* xml: xml<5>.
* xml.etree: xml etree.
* xmlrpc.server: xmlrpc server.
* zipapp::
* zipfile: zipfile<2>.
Deprecated Python modules, functions and methods
* aifc::
* asyncio: asyncio<8>.
* collections: collections<3>.
* dbm: dbm<3>.
* enum: enum<6>.
* gettext: gettext<2>.
* importlib: importlib<5>.
* locale: locale<5>.
* macpath::
* threading: threading<5>.
* socket: socket<6>.
* ssl: ssl<6>.
* sunau::
* sys: sys<9>.
* wave::
Porting to Python 3.7
* Changes in Python Behavior::
* Changes in the Python API: Changes in the Python API<7>.
* Changes in the C API: Changes in the C API<5>.
* CPython bytecode changes: CPython bytecode changes<6>.
* Windows-only Changes: Windows-only Changes<2>.
* Other CPython implementation changes::
What’s New In Python 3.6
* Summary – Release highlights: Summary – Release highlights<6>.
* New Features: New Features<13>.
* Other Language Changes: Other Language Changes<8>.
* New Modules: New Modules<8>.
* Improved Modules: Improved Modules<8>.
* Optimizations: Optimizations<8>.
* Build and C API Changes: Build and C API Changes<2>.
* Other Improvements::
* Deprecated: Deprecated<9>.
* Removed: Removed<9>.
* Porting to Python 3.6: Porting to Python 3 6.
* Notable changes in Python 3.6.2: Notable changes in Python 3 6 2.
* Notable changes in Python 3.6.4: Notable changes in Python 3 6 4.
* Notable changes in Python 3.6.5: Notable changes in Python 3 6 5.
* Notable changes in Python 3.6.7: Notable changes in Python 3 6 7.
* Notable changes in Python 3.6.10: Notable changes in Python 3 6 10.
* Notable changes in Python 3.6.13: Notable changes in Python 3 6 13.
* Notable changes in Python 3.6.14: Notable changes in Python 3 6 14.
New Features
* PEP 498; Formatted string literals: PEP 498 Formatted string literals.
* PEP 526; Syntax for variable annotations: PEP 526 Syntax for variable annotations.
* PEP 515; Underscores in Numeric Literals: PEP 515 Underscores in Numeric Literals.
* PEP 525; Asynchronous Generators: PEP 525 Asynchronous Generators.
* PEP 530; Asynchronous Comprehensions: PEP 530 Asynchronous Comprehensions.
* PEP 487; Simpler customization of class creation: PEP 487 Simpler customization of class creation.
* PEP 487; Descriptor Protocol Enhancements: PEP 487 Descriptor Protocol Enhancements.
* PEP 519; Adding a file system path protocol: PEP 519 Adding a file system path protocol.
* PEP 495; Local Time Disambiguation: PEP 495 Local Time Disambiguation.
* PEP 529; Change Windows filesystem encoding to UTF-8: PEP 529 Change Windows filesystem encoding to UTF-8.
* PEP 528; Change Windows console encoding to UTF-8: PEP 528 Change Windows console encoding to UTF-8.
* PEP 520; Preserving Class Attribute Definition Order: PEP 520 Preserving Class Attribute Definition Order.
* PEP 468; Preserving Keyword Argument Order: PEP 468 Preserving Keyword Argument Order.
* New dict implementation::
* PEP 523; Adding a frame evaluation API to CPython: PEP 523 Adding a frame evaluation API to CPython.
* PYTHONMALLOC environment variable::
* DTrace and SystemTap probing support::
New Modules
* secrets::
Improved Modules
* array: array<4>.
* ast: ast<4>.
* asyncio: asyncio<9>.
* binascii: binascii<2>.
* cmath::
* collections: collections<4>.
* concurrent.futures: concurrent futures<4>.
* contextlib: contextlib<4>.
* datetime: datetime<5>.
* decimal: decimal<2>.
* distutils: distutils<5>.
* email: email<2>.
* encodings: encodings<2>.
* enum: enum<7>.
* faulthandler: faulthandler<2>.
* fileinput: fileinput<2>.
* hashlib: hashlib<5>.
* http.client: http client<2>.
* idlelib and IDLE: idlelib and IDLE<2>.
* importlib: importlib<6>.
* inspect: inspect<6>.
* json::
* logging: logging<4>.
* math: math<7>.
* multiprocessing: multiprocessing<5>.
* os: os<8>.
* pathlib: pathlib<9>.
* pdb: pdb<5>.
* pickle: pickle<2>.
* pickletools::
* pydoc: pydoc<3>.
* random: random<4>.
* re: re<5>.
* readline::
* rlcompleter::
* shlex: shlex<2>.
* site: site<3>.
* sqlite3: sqlite3<7>.
* socket: socket<7>.
* socketserver: socketserver<2>.
* ssl: ssl<7>.
* statistics: statistics<5>.
* struct::
* subprocess: subprocess<3>.
* sys: sys<10>.
* telnetlib::
* time: time<6>.
* timeit::
* tkinter: tkinter<6>.
* traceback: traceback<4>.
* tracemalloc: tracemalloc<3>.
* typing: typing<9>.
* unicodedata: unicodedata<7>.
* unittest.mock: unittest mock<2>.
* urllib.request: urllib request.
* urllib.robotparser: urllib robotparser.
* venv: venv<5>.
* warnings: warnings<4>.
* winreg::
* winsound::
* xmlrpc.client: xmlrpc client.
* zipfile: zipfile<3>.
* zlib::
Deprecated
* New Keywords::
* Deprecated Python behavior::
* Deprecated Python modules, functions and methods: Deprecated Python modules functions and methods<2>.
* xml: xml<6>.
* Deprecated functions and types of the C API: Deprecated functions and types of the C API<2>.
* Deprecated Build Options::
Deprecated Python modules, functions and methods
* asynchat::
* asyncore::
* dbm: dbm<4>.
* distutils: distutils<6>.
* grp::
* importlib: importlib<7>.
* os: os<9>.
* re: re<6>.
* ssl: ssl<8>.
* tkinter: tkinter<7>.
* venv: venv<6>.
Removed
* API and Feature Removals: API and Feature Removals<3>.
Porting to Python 3.6
* Changes in ‘python’ Command Behavior::
* Changes in the Python API: Changes in the Python API<8>.
* Changes in the C API: Changes in the C API<6>.
* CPython bytecode changes: CPython bytecode changes<7>.
Notable changes in Python 3.6.2
* New make regen-all build target::
* Removal of make touch build target::
What’s New In Python 3.5
* Summary – Release highlights: Summary – Release highlights<7>.
* New Features: New Features<14>.
* Other Language Changes: Other Language Changes<9>.
* New Modules: New Modules<9>.
* Improved Modules: Improved Modules<9>.
* Other module-level changes::
* Optimizations: Optimizations<9>.
* Build and C API Changes: Build and C API Changes<3>.
* Deprecated: Deprecated<10>.
* Removed: Removed<10>.
* Porting to Python 3.5: Porting to Python 3 5.
* Notable changes in Python 3.5.4: Notable changes in Python 3 5 4.
New Features
* PEP 492 - Coroutines with async and await syntax::
* PEP 465 - A dedicated infix operator for matrix multiplication::
* PEP 448 - Additional Unpacking Generalizations::
* PEP 461 - percent formatting support for bytes and bytearray::
* PEP 484 - Type Hints::
* PEP 471 - os.scandir() function – a better and faster directory iterator: PEP 471 - os scandir function – a better and faster directory iterator.
* PEP 475; Retry system calls failing with EINTR: PEP 475 Retry system calls failing with EINTR.
* PEP 479; Change StopIteration handling inside generators: PEP 479 Change StopIteration handling inside generators.
* PEP 485; A function for testing approximate equality: PEP 485 A function for testing approximate equality.
* PEP 486; Make the Python Launcher aware of virtual environments: PEP 486 Make the Python Launcher aware of virtual environments.
* PEP 488; Elimination of PYO files: PEP 488 Elimination of PYO files.
* PEP 489; Multi-phase extension module initialization: PEP 489 Multi-phase extension module initialization.
New Modules
* typing: typing<10>.
* zipapp: zipapp<2>.
Improved Modules
* argparse: argparse<4>.
* asyncio: asyncio<10>.
* bz2::
* cgi::
* cmath: cmath<2>.
* code::
* collections: collections<5>.
* collections.abc: collections abc<3>.
* compileall: compileall<4>.
* concurrent.futures: concurrent futures<5>.
* configparser: configparser<4>.
* contextlib: contextlib<5>.
* csv: csv<3>.
* curses: curses<4>.
* dbm: dbm<5>.
* difflib::
* distutils: distutils<7>.
* doctest: doctest<3>.
* email: email<3>.
* enum: enum<8>.
* faulthandler: faulthandler<3>.
* functools: functools<4>.
* glob: glob<3>.
* gzip: gzip<4>.
* heapq::
* http: http<2>.
* http.client: http client<3>.
* idlelib and IDLE: idlelib and IDLE<3>.
* imaplib: imaplib<2>.
* imghdr::
* importlib: importlib<8>.
* inspect: inspect<7>.
* io: io<5>.
* ipaddress: ipaddress<4>.
* json: json<2>.
* linecache: linecache<2>.
* locale: locale<6>.
* logging: logging<5>.
* lzma::
* math: math<8>.
* multiprocessing: multiprocessing<6>.
* operator: operator<2>.
* os: os<10>.
* pathlib: pathlib<10>.
* pickle: pickle<3>.
* poplib: poplib<2>.
* re: re<7>.
* readline: readline<2>.
* selectors::
* shutil: shutil<5>.
* signal: signal<3>.
* smtpd: smtpd<2>.
* smtplib: smtplib<2>.
* sndhdr::
* socket: socket<8>.
* ssl: ssl<9>.
* sqlite3: sqlite3<8>.
* subprocess: subprocess<4>.
* sys: sys<11>.
* sysconfig: sysconfig<2>.
* tarfile: tarfile<7>.
* threading: threading<6>.
* time: time<7>.
* timeit: timeit<2>.
* tkinter: tkinter<8>.
* traceback: traceback<5>.
* types: types<5>.
* unicodedata: unicodedata<8>.
* unittest: unittest<8>.
* unittest.mock: unittest mock<3>.
* urllib: urllib<2>.
* wsgiref::
* xmlrpc: xmlrpc<2>.
* xml.sax: xml sax.
* zipfile: zipfile<4>.
ssl
* Memory BIO Support::
* Application-Layer Protocol Negotiation Support::
* Other Changes::
Deprecated
* New Keywords: New Keywords<2>.
* Deprecated Python Behavior: Deprecated Python Behavior<2>.
* Unsupported Operating Systems::
* Deprecated Python modules, functions and methods: Deprecated Python modules functions and methods<3>.
Removed
* API and Feature Removals: API and Feature Removals<4>.
Porting to Python 3.5
* Changes in Python behavior: Changes in Python behavior<2>.
* Changes in the Python API: Changes in the Python API<9>.
* Changes in the C API: Changes in the C API<7>.
Notable changes in Python 3.5.4
* New make regen-all build target: New make regen-all build target<2>.
* Removal of make touch build target: Removal of make touch build target<2>.
What’s New In Python 3.4
* Summary – Release Highlights: Summary – Release Highlights<3>.
* New Features: New Features<15>.
* New Modules: New Modules<10>.
* Improved Modules: Improved Modules<10>.
* CPython Implementation Changes::
* Deprecated: Deprecated<11>.
* Removed: Removed<11>.
* Porting to Python 3.4: Porting to Python 3 4.
* Changed in 3.4.3: Changed in 3 4 3.
New Features
* PEP 453; Explicit Bootstrapping of PIP in Python Installations: PEP 453 Explicit Bootstrapping of PIP in Python Installations.
* PEP 446; Newly Created File Descriptors Are Non-Inheritable: PEP 446 Newly Created File Descriptors Are Non-Inheritable.
* Improvements to Codec Handling::
* PEP 451; A ModuleSpec Type for the Import System: PEP 451 A ModuleSpec Type for the Import System.
* Other Language Changes: Other Language Changes<10>.
PEP 453: Explicit Bootstrapping of PIP in Python Installations
* Bootstrapping pip By Default::
* Documentation Changes::
New Modules
* asyncio: asyncio<11>.
* ensurepip: ensurepip<2>.
* enum: enum<9>.
* pathlib: pathlib<11>.
* selectors: selectors<2>.
* statistics: statistics<6>.
* tracemalloc: tracemalloc<4>.
Improved Modules
* abc::
* aifc: aifc<2>.
* argparse: argparse<5>.
* audioop::
* base64: base64<3>.
* collections: collections<6>.
* colorsys::
* contextlib: contextlib<6>.
* dbm: dbm<6>.
* dis: dis<4>.
* doctest: doctest<4>.
* email: email<4>.
* filecmp::
* functools: functools<5>.
* gc: gc<5>.
* glob: glob<4>.
* hashlib: hashlib<6>.
* hmac: hmac<3>.
* html::
* http: http<3>.
* idlelib and IDLE: idlelib and IDLE<4>.
* importlib: importlib<9>.
* inspect: inspect<8>.
* ipaddress: ipaddress<5>.
* logging: logging<6>.
* marshal: marshal<2>.
* mmap: mmap<3>.
* multiprocessing: multiprocessing<7>.
* operator: operator<3>.
* os: os<11>.
* pdb: pdb<6>.
* pickle: pickle<4>.
* plistlib: plistlib<2>.
* poplib: poplib<3>.
* pprint: pprint<4>.
* pty::
* pydoc: pydoc<4>.
* re: re<8>.
* resource::
* select::
* shelve: shelve<2>.
* shutil: shutil<6>.
* smtpd: smtpd<3>.
* smtplib: smtplib<3>.
* socket: socket<9>.
* sqlite3: sqlite3<9>.
* ssl: ssl<10>.
* stat::
* struct: struct<2>.
* subprocess: subprocess<5>.
* sunau: sunau<2>.
* sys: sys<12>.
* tarfile: tarfile<8>.
* textwrap::
* threading: threading<7>.
* traceback: traceback<6>.
* types: types<6>.
* urllib: urllib<3>.
* unittest: unittest<9>.
* venv: venv<7>.
* wave: wave<2>.
* weakref: weakref<2>.
* xml.etree: xml etree<2>.
* zipfile: zipfile<5>.
CPython Implementation Changes
* PEP 445; Customization of CPython Memory Allocators: PEP 445 Customization of CPython Memory Allocators.
* PEP 442; Safe Object Finalization: PEP 442 Safe Object Finalization.
* PEP 456; Secure and Interchangeable Hash Algorithm: PEP 456 Secure and Interchangeable Hash Algorithm.
* PEP 436; Argument Clinic: PEP 436 Argument Clinic.
* Other Build and C API Changes::
* Other Improvements: Other Improvements<2>.
* Significant Optimizations::
Deprecated
* Deprecations in the Python API::
* Deprecated Features::
Removed
* Operating Systems No Longer Supported::
* API and Feature Removals: API and Feature Removals<5>.
* Code Cleanups::
Porting to Python 3.4
* Changes in ‘python’ Command Behavior: Changes in ‘python’ Command Behavior<2>.
* Changes in the Python API: Changes in the Python API<10>.
* Changes in the C API: Changes in the C API<8>.
Changed in 3.4.3
* PEP 476; Enabling certificate verification by default for stdlib http clients: PEP 476 Enabling certificate verification by default for stdlib http clients.
What’s New In Python 3.3
* Summary – Release highlights: Summary – Release highlights<8>.
* PEP 405; Virtual Environments: PEP 405 Virtual Environments.
* PEP 420; Implicit Namespace Packages: PEP 420 Implicit Namespace Packages.
* PEP 3118; New memoryview implementation and buffer protocol documentation: PEP 3118 New memoryview implementation and buffer protocol documentation.
* PEP 393; Flexible String Representation: PEP 393 Flexible String Representation.
* PEP 397; Python Launcher for Windows: PEP 397 Python Launcher for Windows.
* PEP 3151; Reworking the OS and IO exception hierarchy: PEP 3151 Reworking the OS and IO exception hierarchy.
* PEP 380; Syntax for Delegating to a Subgenerator: PEP 380 Syntax for Delegating to a Subgenerator.
* PEP 409; Suppressing exception context: PEP 409 Suppressing exception context.
* PEP 414; Explicit Unicode literals: PEP 414 Explicit Unicode literals.
* PEP 3155; Qualified name for classes and functions: PEP 3155 Qualified name for classes and functions.
* PEP 412; Key-Sharing Dictionary: PEP 412 Key-Sharing Dictionary.
* PEP 362; Function Signature Object: PEP 362 Function Signature Object.
* PEP 421; Adding sys.implementation: PEP 421 Adding sys implementation.
* Using importlib as the Implementation of Import::
* Other Language Changes: Other Language Changes<11>.
* A Finer-Grained Import Lock::
* Builtin functions and types::
* New Modules: New Modules<11>.
* Improved Modules: Improved Modules<11>.
* Optimizations: Optimizations<10>.
* Build and C API Changes: Build and C API Changes<4>.
* Deprecated: Deprecated<12>.
* Porting to Python 3.3: Porting to Python 3 3.
PEP 3118: New memoryview implementation and buffer protocol documentation
* Features::
* API changes::
PEP 393: Flexible String Representation
* Functionality::
* Performance and resource usage::
PEP 421: Adding sys.implementation
* SimpleNamespace::
Using importlib as the Implementation of Import
* New APIs::
* Visible Changes::
New Modules
* faulthandler: faulthandler<4>.
* ipaddress: ipaddress<6>.
* lzma: lzma<2>.
Improved Modules
* abc: abc<2>.
* array: array<5>.
* base64: base64<4>.
* binascii: binascii<3>.
* bz2: bz2<2>.
* codecs: codecs<2>.
* collections: collections<7>.
* contextlib: contextlib<7>.
* crypt: crypt<2>.
* curses: curses<5>.
* datetime: datetime<6>.
* decimal: decimal<3>.
* email: email<5>.
* ftplib: ftplib<3>.
* functools: functools<6>.
* gc: gc<6>.
* hmac: hmac<4>.
* http: http<4>.
* html: html<2>.
* imaplib: imaplib<3>.
* inspect: inspect<9>.
* io: io<6>.
* itertools: itertools<6>.
* logging: logging<7>.
* math: math<9>.
* mmap: mmap<4>.
* multiprocessing: multiprocessing<8>.
* nntplib: nntplib<2>.
* os: os<12>.
* pdb: pdb<7>.
* pickle: pickle<5>.
* pydoc: pydoc<5>.
* re: re<9>.
* sched::
* select: select<2>.
* shlex: shlex<3>.
* shutil: shutil<7>.
* signal: signal<4>.
* smtpd: smtpd<4>.
* smtplib: smtplib<4>.
* socket: socket<10>.
* socketserver: socketserver<3>.
* sqlite3: sqlite3<10>.
* ssl: ssl<11>.
* stat: stat<2>.
* struct: struct<3>.
* subprocess: subprocess<6>.
* sys: sys<13>.
* tarfile: tarfile<9>.
* tempfile: tempfile<4>.
* textwrap: textwrap<2>.
* threading: threading<8>.
* time: time<8>.
* types: types<7>.
* unittest: unittest<10>.
* urllib: urllib<4>.
* webbrowser: webbrowser<3>.
* xml.etree.ElementTree: xml etree ElementTree<2>.
* zlib: zlib<2>.
decimal
* Features: Features<2>.
* API changes: API changes<2>.
email
* Policy Framework::
* Provisional Policy with New Header API::
* Other API Changes::
Deprecated
* Unsupported Operating Systems: Unsupported Operating Systems<2>.
* Deprecated Python modules, functions and methods: Deprecated Python modules functions and methods<4>.
* Deprecated functions and types of the C API: Deprecated functions and types of the C API<3>.
* Deprecated features::
Porting to Python 3.3
* Porting Python code::
* Porting C code::
* Building C extensions::
* Command Line Switch Changes::
What’s New In Python 3.2
* PEP 384; Defining a Stable ABI: PEP 384 Defining a Stable ABI.
* PEP 389; Argparse Command Line Parsing Module: PEP 389 Argparse Command Line Parsing Module.
* PEP 391; Dictionary Based Configuration for Logging: PEP 391 Dictionary Based Configuration for Logging.
* PEP 3148; The concurrent.futures module: PEP 3148 The concurrent futures module.
* PEP 3147; PYC Repository Directories: PEP 3147 PYC Repository Directories.
* PEP 3149; ABI Version Tagged .so Files: PEP 3149 ABI Version Tagged so Files.
* PEP 3333; Python Web Server Gateway Interface v1.0.1: PEP 3333 Python Web Server Gateway Interface v1 0 1.
* Other Language Changes: Other Language Changes<12>.
* New, Improved, and Deprecated Modules: New Improved and Deprecated Modules.
* Multi-threading::
* Optimizations: Optimizations<11>.
* Unicode::
* Codecs::
* Documentation::
* IDLE::
* Code Repository::
* Build and C API Changes: Build and C API Changes<5>.
* Porting to Python 3.2: Porting to Python 3 2.
New, Improved, and Deprecated Modules
* email: email<6>.
* elementtree::
* functools: functools<7>.
* itertools: itertools<7>.
* collections: collections<8>.
* threading: threading<9>.
* datetime and time::
* math: math<10>.
* abc: abc<3>.
* io: io<7>.
* reprlib::
* logging: logging<8>.
* csv: csv<4>.
* contextlib: contextlib<8>.
* decimal and fractions::
* ftp::
* popen::
* select: select<3>.
* gzip and zipfile::
* tarfile: tarfile<10>.
* hashlib: hashlib<7>.
* ast: ast<5>.
* os: os<13>.
* shutil: shutil<8>.
* sqlite3: sqlite3<11>.
* html: html<3>.
* socket: socket<11>.
* ssl: ssl<12>.
* nntp::
* certificates::
* imaplib: imaplib<4>.
* http.client: http client<4>.
* unittest: unittest<11>.
* random: random<5>.
* poplib: poplib<4>.
* asyncore: asyncore<2>.
* tempfile: tempfile<5>.
* inspect: inspect<10>.
* pydoc: pydoc<6>.
* dis: dis<5>.
* dbm: dbm<7>.
* ctypes: ctypes<3>.
* site: site<4>.
* sysconfig: sysconfig<3>.
* pdb: pdb<8>.
* configparser: configparser<5>.
* urllib.parse: urllib parse<6>.
* mailbox::
* turtledemo::
What’s New In Python 3.1
* PEP 372; Ordered Dictionaries: PEP 372 Ordered Dictionaries.
* PEP 378; Format Specifier for Thousands Separator: PEP 378 Format Specifier for Thousands Separator.
* Other Language Changes: Other Language Changes<13>.
* New, Improved, and Deprecated Modules: New Improved and Deprecated Modules<2>.
* Optimizations: Optimizations<12>.
* IDLE: IDLE<2>.
* Build and C API Changes: Build and C API Changes<6>.
* Porting to Python 3.1: Porting to Python 3 1.
What’s New In Python 3.0
* Common Stumbling Blocks::
* Overview Of Syntax Changes::
* Changes Already Present In Python 2.6: Changes Already Present In Python 2 6.
* Library Changes::
* PEP 3101; A New Approach To String Formatting: PEP 3101 A New Approach To String Formatting.
* Changes To Exceptions::
* Miscellaneous Other Changes::
* Build and C API Changes: Build and C API Changes<7>.
* Performance::
* Porting To Python 3.0: Porting To Python 3 0.
Common Stumbling Blocks
* Print Is A Function::
* Views And Iterators Instead Of Lists::
* Ordering Comparisons::
* Integers::
* Text Vs. Data Instead Of Unicode Vs. 8-bit: Text Vs Data Instead Of Unicode Vs 8-bit.
Overview Of Syntax Changes
* New Syntax::
* Changed Syntax::
* Removed Syntax::
Miscellaneous Other Changes
* Operators And Special Methods::
* Builtins::
What’s New in Python 2.7
* The Future for Python 2.x: The Future for Python 2 x.
* Changes to the Handling of Deprecation Warnings::
* Python 3.1 Features: Python 3 1 Features.
* PEP 372; Adding an Ordered Dictionary to collections: PEP 372 Adding an Ordered Dictionary to collections.
* PEP 378; Format Specifier for Thousands Separator: PEP 378 Format Specifier for Thousands Separator<2>.
* PEP 389; The argparse Module for Parsing Command Lines: PEP 389 The argparse Module for Parsing Command Lines.
* PEP 391; Dictionary-Based Configuration For Logging: PEP 391 Dictionary-Based Configuration For Logging.
* PEP 3106; Dictionary Views: PEP 3106 Dictionary Views.
* PEP 3137; The memoryview Object: PEP 3137 The memoryview Object.
* Other Language Changes: Other Language Changes<14>.
* New and Improved Modules::
* Build and C API Changes: Build and C API Changes<8>.
* Other Changes and Fixes::
* Porting to Python 2.7: Porting to Python 2 7.
* New Features Added to Python 2.7 Maintenance Releases: New Features Added to Python 2 7 Maintenance Releases.
* Acknowledgements::
Other Language Changes
* Interpreter Changes::
* Optimizations: Optimizations<13>.
New and Improved Modules
* New module; importlib: New module importlib.
* New module; sysconfig: New module sysconfig.
* ttk; Themed Widgets for Tk: ttk Themed Widgets for Tk.
* Updated module; unittest: Updated module unittest.
* Updated module; ElementTree 1.3: Updated module ElementTree 1 3.
Build and C API Changes
* Capsules::
* Port-Specific Changes; Windows: Port-Specific Changes Windows.
* Port-Specific Changes; Mac OS X: Port-Specific Changes Mac OS X.
* Port-Specific Changes; FreeBSD: Port-Specific Changes FreeBSD.
New Features Added to Python 2.7 Maintenance Releases
* Two new environment variables for debug mode::
* PEP 434; IDLE Enhancement Exception for All Branches: PEP 434 IDLE Enhancement Exception for All Branches.
* PEP 466; Network Security Enhancements for Python 2.7: PEP 466 Network Security Enhancements for Python 2 7.
* PEP 477; Backport ensurepip (PEP 453) to Python 2.7: PEP 477 Backport ensurepip PEP 453 to Python 2 7.
* PEP 476; Enabling certificate verification by default for stdlib http clients: PEP 476 Enabling certificate verification by default for stdlib http clients<2>.
* PEP 493; HTTPS verification migration tools for Python 2.7: PEP 493 HTTPS verification migration tools for Python 2 7.
* New make regen-all build target: New make regen-all build target<3>.
* Removal of make touch build target: Removal of make touch build target<3>.
PEP 477: Backport ensurepip (PEP 453) to Python 2.7
* Bootstrapping pip By Default: Bootstrapping pip By Default<2>.
* Documentation Changes: Documentation Changes<2>.
What’s New in Python 2.6
* Python 3.0: Python 3 0.
* Changes to the Development Process::
* PEP 343; The ‘with’ statement: PEP 343 The ‘with’ statement.
* PEP 366; Explicit Relative Imports From a Main Module: PEP 366 Explicit Relative Imports From a Main Module.
* PEP 370; Per-user site-packages Directory: PEP 370 Per-user site-packages Directory.
* PEP 371; The multiprocessing Package: PEP 371 The multiprocessing Package.
* PEP 3101; Advanced String Formatting: PEP 3101 Advanced String Formatting.
* PEP 3105; print As a Function: PEP 3105 print As a Function.
* PEP 3110; Exception-Handling Changes: PEP 3110 Exception-Handling Changes.
* PEP 3112; Byte Literals: PEP 3112 Byte Literals.
* PEP 3116; New I/O Library: PEP 3116 New I/O Library.
* PEP 3118; Revised Buffer Protocol: PEP 3118 Revised Buffer Protocol.
* PEP 3119; Abstract Base Classes: PEP 3119 Abstract Base Classes.
* PEP 3127; Integer Literal Support and Syntax: PEP 3127 Integer Literal Support and Syntax.
* PEP 3129; Class Decorators: PEP 3129 Class Decorators.
* PEP 3141; A Type Hierarchy for Numbers: PEP 3141 A Type Hierarchy for Numbers.
* Other Language Changes: Other Language Changes<15>.
* New and Improved Modules: New and Improved Modules<2>.
* Deprecations and Removals::
* Build and C API Changes: Build and C API Changes<9>.
* Porting to Python 2.6: Porting to Python 2 6.
* Acknowledgements: Acknowledgements<2>.
Changes to the Development Process
* New Issue Tracker; Roundup: New Issue Tracker Roundup.
* New Documentation Format; reStructuredText Using Sphinx: New Documentation Format reStructuredText Using Sphinx.
PEP 343: The ‘with’ statement
* Writing Context Managers::
* The contextlib module::
PEP 3141: A Type Hierarchy for Numbers
* The fractions Module::
Other Language Changes
* Optimizations: Optimizations<14>.
* Interpreter Changes: Interpreter Changes<2>.
New and Improved Modules
* The ast module::
* The future_builtins module::
* The json module; JavaScript Object Notation: The json module JavaScript Object Notation.
* The plistlib module; A Property-List Parser: The plistlib module A Property-List Parser.
* ctypes Enhancements::
* Improved SSL Support::
Build and C API Changes
* Port-Specific Changes; Windows: Port-Specific Changes Windows<2>.
* Port-Specific Changes; Mac OS X: Port-Specific Changes Mac OS X<2>.
* Port-Specific Changes; IRIX: Port-Specific Changes IRIX.
What’s New in Python 2.5
* PEP 308; Conditional Expressions: PEP 308 Conditional Expressions.
* PEP 309; Partial Function Application: PEP 309 Partial Function Application.
* PEP 314; Metadata for Python Software Packages v1.1: PEP 314 Metadata for Python Software Packages v1 1.
* PEP 328; Absolute and Relative Imports: PEP 328 Absolute and Relative Imports.
* PEP 338; Executing Modules as Scripts: PEP 338 Executing Modules as Scripts.
* PEP 341; Unified try/except/finally: PEP 341 Unified try/except/finally.
* PEP 342; New Generator Features: PEP 342 New Generator Features.
* PEP 343; The ‘with’ statement: PEP 343 The ‘with’ statement<2>.
* PEP 352; Exceptions as New-Style Classes: PEP 352 Exceptions as New-Style Classes.
* PEP 353; Using ssize_t as the index type: PEP 353 Using ssize_t as the index type.
* PEP 357; The ‘__index__’ method: PEP 357 The ‘__index__’ method.
* Other Language Changes: Other Language Changes<16>.
* New, Improved, and Removed Modules: New Improved and Removed Modules.
* Build and C API Changes: Build and C API Changes<10>.
* Porting to Python 2.5: Porting to Python 2 5.
* Acknowledgements: Acknowledgements<3>.
PEP 343: The ‘with’ statement
* Writing Context Managers: Writing Context Managers<2>.
* The contextlib module: The contextlib module<2>.
Other Language Changes
* Interactive Interpreter Changes::
* Optimizations: Optimizations<15>.
New, Improved, and Removed Modules
* The ctypes package::
* The ElementTree package::
* The hashlib package::
* The sqlite3 package::
* The wsgiref package::
Build and C API Changes
* Port-Specific Changes::
What’s New in Python 2.4
* PEP 218; Built-In Set Objects: PEP 218 Built-In Set Objects.
* PEP 237; Unifying Long Integers and Integers: PEP 237 Unifying Long Integers and Integers.
* PEP 289; Generator Expressions: PEP 289 Generator Expressions.
* PEP 292; Simpler String Substitutions: PEP 292 Simpler String Substitutions.
* PEP 318; Decorators for Functions and Methods: PEP 318 Decorators for Functions and Methods.
* PEP 322; Reverse Iteration: PEP 322 Reverse Iteration.
* PEP 324; New subprocess Module: PEP 324 New subprocess Module.
* PEP 327; Decimal Data Type: PEP 327 Decimal Data Type.
* PEP 328; Multi-line Imports: PEP 328 Multi-line Imports.
* PEP 331; Locale-Independent Float/String Conversions: PEP 331 Locale-Independent Float/String Conversions.
* Other Language Changes: Other Language Changes<17>.
* New, Improved, and Deprecated Modules: New Improved and Deprecated Modules<3>.
* Build and C API Changes: Build and C API Changes<11>.
* Porting to Python 2.4: Porting to Python 2 4.
* Acknowledgements: Acknowledgements<4>.
PEP 327: Decimal Data Type
* Why is Decimal needed?::
* The Decimal type::
* The Context type::
Other Language Changes
* Optimizations: Optimizations<16>.
New, Improved, and Deprecated Modules
* cookielib::
* doctest: doctest<5>.
Build and C API Changes
* Port-Specific Changes: Port-Specific Changes<2>.
What’s New in Python 2.3
* PEP 218; A Standard Set Datatype: PEP 218 A Standard Set Datatype.
* PEP 255; Simple Generators: PEP 255 Simple Generators.
* PEP 263; Source Code Encodings: PEP 263 Source Code Encodings.
* PEP 273; Importing Modules from ZIP Archives: PEP 273 Importing Modules from ZIP Archives.
* PEP 277; Unicode file name support for Windows NT: PEP 277 Unicode file name support for Windows NT.
* PEP 278; Universal Newline Support: PEP 278 Universal Newline Support.
* PEP 279; enumerate(): PEP 279 enumerate.
* PEP 282; The logging Package: PEP 282 The logging Package.
* PEP 285; A Boolean Type: PEP 285 A Boolean Type.
* PEP 293; Codec Error Handling Callbacks: PEP 293 Codec Error Handling Callbacks.
* PEP 301; Package Index and Metadata for Distutils: PEP 301 Package Index and Metadata for Distutils.
* PEP 302; New Import Hooks: PEP 302 New Import Hooks.
* PEP 305; Comma-separated Files: PEP 305 Comma-separated Files.
* PEP 307; Pickle Enhancements: PEP 307 Pickle Enhancements.
* Extended Slices::
* Other Language Changes: Other Language Changes<18>.
* New, Improved, and Deprecated Modules: New Improved and Deprecated Modules<4>.
* Pymalloc; A Specialized Object Allocator: Pymalloc A Specialized Object Allocator.
* Build and C API Changes: Build and C API Changes<12>.
* Other Changes and Fixes: Other Changes and Fixes<2>.
* Porting to Python 2.3: Porting to Python 2 3.
* Acknowledgements: Acknowledgements<5>.
Other Language Changes
* String Changes::
* Optimizations: Optimizations<17>.
New, Improved, and Deprecated Modules
* Date/Time Type::
* The optparse Module::
Build and C API Changes
* Port-Specific Changes: Port-Specific Changes<3>.
What’s New in Python 2.2
* Introduction::
* PEPs 252 and 253; Type and Class Changes: PEPs 252 and 253 Type and Class Changes.
* PEP 234; Iterators: PEP 234 Iterators.
* PEP 255; Simple Generators: PEP 255 Simple Generators<2>.
* PEP 237; Unifying Long Integers and Integers: PEP 237 Unifying Long Integers and Integers<2>.
* PEP 238; Changing the Division Operator: PEP 238 Changing the Division Operator.
* Unicode Changes::
* PEP 227; Nested Scopes: PEP 227 Nested Scopes.
* New and Improved Modules: New and Improved Modules<3>.
* Interpreter Changes and Fixes::
* Other Changes and Fixes: Other Changes and Fixes<3>.
* Acknowledgements: Acknowledgements<6>.
PEPs 252 and 253: Type and Class Changes
* Old and New Classes::
* Descriptors::
* Multiple Inheritance; The Diamond Rule: Multiple Inheritance The Diamond Rule.
* Attribute Access::
* Related Links::
What’s New in Python 2.1
* Introduction: Introduction<2>.
* PEP 227; Nested Scopes: PEP 227 Nested Scopes<2>.
* PEP 236; __future__ Directives: PEP 236 __future__ Directives.
* PEP 207; Rich Comparisons: PEP 207 Rich Comparisons.
* PEP 230; Warning Framework: PEP 230 Warning Framework.
* PEP 229; New Build System: PEP 229 New Build System.
* PEP 205; Weak References: PEP 205 Weak References.
* PEP 232; Function Attributes: PEP 232 Function Attributes.
* PEP 235; Importing Modules on Case-Insensitive Platforms: PEP 235 Importing Modules on Case-Insensitive Platforms.
* PEP 217; Interactive Display Hook: PEP 217 Interactive Display Hook.
* PEP 208; New Coercion Model: PEP 208 New Coercion Model.
* PEP 241; Metadata in Python Packages: PEP 241 Metadata in Python Packages.
* New and Improved Modules: New and Improved Modules<4>.
* Other Changes and Fixes: Other Changes and Fixes<4>.
* Acknowledgements: Acknowledgements<7>.
What’s New in Python 2.0
* Introduction: Introduction<3>.
* What About Python 1.6?: What About Python 1 6?.
* New Development Process::
* Unicode: Unicode<2>.
* List Comprehensions::
* Augmented Assignment::
* String Methods::
* Garbage Collection of Cycles::
* Other Core Changes::
* Porting to 2.0: Porting to 2 0.
* Extending/Embedding Changes::
* Distutils; Making Modules Easy to Install: Distutils Making Modules Easy to Install.
* XML Modules::
* Module changes::
* New modules::
* IDLE Improvements::
* Deleted and Deprecated Modules::
* Acknowledgements: Acknowledgements<8>.
Other Core Changes
* Minor Language Changes::
* Changes to Built-in Functions::
XML Modules
* SAX2 Support::
* DOM Support::
* Relationship to PyXML::
Changelog
* Python 3.13.7 final: Python 3 13 7 final.
* Python 3.13.6 final: Python 3 13 6 final.
* Python 3.13.5 final: Python 3 13 5 final.
* Python 3.13.4 final: Python 3 13 4 final.
* Python 3.13.3 final: Python 3 13 3 final.
* Python 3.13.2 final: Python 3 13 2 final.
* Python 3.13.1 final: Python 3 13 1 final.
* Python 3.13.0 final: Python 3 13 0 final.
* Python 3.13.0 release candidate 3: Python 3 13 0 release candidate 3.
* Python 3.13.0 release candidate 2: Python 3 13 0 release candidate 2.
* Python 3.13.0 release candidate 1: Python 3 13 0 release candidate 1.
* Python 3.13.0 beta 4: Python 3 13 0 beta 4.
* Python 3.13.0 beta 3: Python 3 13 0 beta 3.
* Python 3.13.0 beta 2: Python 3 13 0 beta 2.
* Python 3.13.0 beta 1: Python 3 13 0 beta 1.
* Python 3.13.0 alpha 6: Python 3 13 0 alpha 6.
* Python 3.13.0 alpha 5: Python 3 13 0 alpha 5.
* Python 3.13.0 alpha 4: Python 3 13 0 alpha 4.
* Python 3.13.0 alpha 3: Python 3 13 0 alpha 3.
* Python 3.13.0 alpha 2: Python 3 13 0 alpha 2.
* Python 3.13.0 alpha 1: Python 3 13 0 alpha 1.
* Python 3.12.0 beta 1: Python 3 12 0 beta 1.
* Python 3.12.0 alpha 7: Python 3 12 0 alpha 7.
* Python 3.12.0 alpha 6: Python 3 12 0 alpha 6.
* Python 3.12.0 alpha 5: Python 3 12 0 alpha 5.
* Python 3.12.0 alpha 4: Python 3 12 0 alpha 4.
* Python 3.12.0 alpha 3: Python 3 12 0 alpha 3.
* Python 3.12.0 alpha 2: Python 3 12 0 alpha 2.
* Python 3.12.0 alpha 1: Python 3 12 0 alpha 1.
* Python 3.11.0 beta 1: Python 3 11 0 beta 1.
* Python 3.11.0 alpha 7: Python 3 11 0 alpha 7.
* Python 3.11.0 alpha 6: Python 3 11 0 alpha 6.
* Python 3.11.0 alpha 5: Python 3 11 0 alpha 5.
* Python 3.11.0 alpha 4: Python 3 11 0 alpha 4.
* Python 3.11.0 alpha 3: Python 3 11 0 alpha 3.
* Python 3.11.0 alpha 2: Python 3 11 0 alpha 2.
* Python 3.11.0 alpha 1: Python 3 11 0 alpha 1.
* Python 3.10.0 beta 1: Python 3 10 0 beta 1.
* Python 3.10.0 alpha 7: Python 3 10 0 alpha 7.
* Python 3.10.0 alpha 6: Python 3 10 0 alpha 6.
* Python 3.10.0 alpha 5: Python 3 10 0 alpha 5.
* Python 3.10.0 alpha 4: Python 3 10 0 alpha 4.
* Python 3.10.0 alpha 3: Python 3 10 0 alpha 3.
* Python 3.10.0 alpha 2: Python 3 10 0 alpha 2.
* Python 3.10.0 alpha 1: Python 3 10 0 alpha 1.
* Python 3.9.0 beta 1: Python 3 9 0 beta 1.
* Python 3.9.0 alpha 6: Python 3 9 0 alpha 6.
* Python 3.9.0 alpha 5: Python 3 9 0 alpha 5.
* Python 3.9.0 alpha 4: Python 3 9 0 alpha 4.
* Python 3.9.0 alpha 3: Python 3 9 0 alpha 3.
* Python 3.9.0 alpha 2: Python 3 9 0 alpha 2.
* Python 3.9.0 alpha 1: Python 3 9 0 alpha 1.
* Python 3.8.0 beta 1: Python 3 8 0 beta 1.
* Python 3.8.0 alpha 4: Python 3 8 0 alpha 4.
* Python 3.8.0 alpha 3: Python 3 8 0 alpha 3.
* Python 3.8.0 alpha 2: Python 3 8 0 alpha 2.
* Python 3.8.0 alpha 1: Python 3 8 0 alpha 1.
* Python 3.7.0 final: Python 3 7 0 final.
* Python 3.7.0 release candidate 1: Python 3 7 0 release candidate 1.
* Python 3.7.0 beta 5: Python 3 7 0 beta 5.
* Python 3.7.0 beta 4: Python 3 7 0 beta 4.
* Python 3.7.0 beta 3: Python 3 7 0 beta 3.
* Python 3.7.0 beta 2: Python 3 7 0 beta 2.
* Python 3.7.0 beta 1: Python 3 7 0 beta 1.
* Python 3.7.0 alpha 4: Python 3 7 0 alpha 4.
* Python 3.7.0 alpha 3: Python 3 7 0 alpha 3.
* Python 3.7.0 alpha 2: Python 3 7 0 alpha 2.
* Python 3.7.0 alpha 1: Python 3 7 0 alpha 1.
* Python 3.6.6 final: Python 3 6 6 final.
* Python 3.6.6 release candidate 1: Python 3 6 6 release candidate 1.
* Python 3.6.5 final: Python 3 6 5 final.
* Python 3.6.5 release candidate 1: Python 3 6 5 release candidate 1.
* Python 3.6.4 final: Python 3 6 4 final.
* Python 3.6.4 release candidate 1: Python 3 6 4 release candidate 1.
* Python 3.6.3 final: Python 3 6 3 final.
* Python 3.6.3 release candidate 1: Python 3 6 3 release candidate 1.
* Python 3.6.2 final: Python 3 6 2 final.
* Python 3.6.2 release candidate 2: Python 3 6 2 release candidate 2.
* Python 3.6.2 release candidate 1: Python 3 6 2 release candidate 1.
* Python 3.6.1 final: Python 3 6 1 final.
* Python 3.6.1 release candidate 1: Python 3 6 1 release candidate 1.
* Python 3.6.0 final: Python 3 6 0 final.
* Python 3.6.0 release candidate 2: Python 3 6 0 release candidate 2.
* Python 3.6.0 release candidate 1: Python 3 6 0 release candidate 1.
* Python 3.6.0 beta 4: Python 3 6 0 beta 4.
* Python 3.6.0 beta 3: Python 3 6 0 beta 3.
* Python 3.6.0 beta 2: Python 3 6 0 beta 2.
* Python 3.6.0 beta 1: Python 3 6 0 beta 1.
* Python 3.6.0 alpha 4: Python 3 6 0 alpha 4.
* Python 3.6.0 alpha 3: Python 3 6 0 alpha 3.
* Python 3.6.0 alpha 2: Python 3 6 0 alpha 2.
* Python 3.6.0 alpha 1: Python 3 6 0 alpha 1.
* Python 3.5.5 final: Python 3 5 5 final.
* Python 3.5.5 release candidate 1: Python 3 5 5 release candidate 1.
* Python 3.5.4 final: Python 3 5 4 final.
* Python 3.5.4 release candidate 1: Python 3 5 4 release candidate 1.
* Python 3.5.3 final: Python 3 5 3 final.
* Python 3.5.3 release candidate 1: Python 3 5 3 release candidate 1.
* Python 3.5.2 final: Python 3 5 2 final.
* Python 3.5.2 release candidate 1: Python 3 5 2 release candidate 1.
* Python 3.5.1 final: Python 3 5 1 final.
* Python 3.5.1 release candidate 1: Python 3 5 1 release candidate 1.
* Python 3.5.0 final: Python 3 5 0 final.
* Python 3.5.0 release candidate 4: Python 3 5 0 release candidate 4.
* Python 3.5.0 release candidate 3: Python 3 5 0 release candidate 3.
* Python 3.5.0 release candidate 2: Python 3 5 0 release candidate 2.
* Python 3.5.0 release candidate 1: Python 3 5 0 release candidate 1.
* Python 3.5.0 beta 4: Python 3 5 0 beta 4.
* Python 3.5.0 beta 3: Python 3 5 0 beta 3.
* Python 3.5.0 beta 2: Python 3 5 0 beta 2.
* Python 3.5.0 beta 1: Python 3 5 0 beta 1.
* Python 3.5.0 alpha 4: Python 3 5 0 alpha 4.
* Python 3.5.0 alpha 3: Python 3 5 0 alpha 3.
* Python 3.5.0 alpha 2: Python 3 5 0 alpha 2.
* Python 3.5.0 alpha 1: Python 3 5 0 alpha 1.
Python 3.13.7 final
* Library::
* Documentation: Documentation<2>.
* Core and Builtins::
Python 3.13.6 final
* macOS::
* Windows::
* Tools/Demos::
* Tests::
* Security::
* Library: Library<2>.
* Documentation: Documentation<3>.
* Core and Builtins: Core and Builtins<2>.
* Build::
Python 3.13.5 final
* Windows: Windows<2>.
* Tests: Tests<2>.
* Library: Library<3>.
* Core and Builtins: Core and Builtins<3>.
* C API::
Python 3.13.4 final
* Windows: Windows<3>.
* Tests: Tests<3>.
* Security: Security<2>.
* Library: Library<4>.
* IDLE: IDLE<3>.
* Documentation: Documentation<4>.
* Core and Builtins: Core and Builtins<4>.
* C API: C API<2>.
* Build: Build<2>.
Python 3.13.3 final
* macOS: macOS<2>.
* Windows: Windows<4>.
* Tools/Demos: Tools/Demos<2>.
* Tests: Tests<4>.
* Security: Security<3>.
* Library: Library<5>.
* IDLE: IDLE<4>.
* Documentation: Documentation<5>.
* Core and Builtins: Core and Builtins<5>.
* C API: C API<3>.
* Build: Build<3>.
Python 3.13.2 final
* macOS: macOS<3>.
* Windows: Windows<5>.
* Tools/Demos: Tools/Demos<3>.
* Tests: Tests<5>.
* Security: Security<4>.
* Library: Library<6>.
* Documentation: Documentation<6>.
* Core and Builtins: Core and Builtins<6>.
* C API: C API<4>.
* Build: Build<4>.
Python 3.13.1 final
* macOS: macOS<4>.
* Windows: Windows<6>.
* Tools/Demos: Tools/Demos<4>.
* Tests: Tests<6>.
* Security: Security<5>.
* Library: Library<7>.
* IDLE: IDLE<5>.
* Documentation: Documentation<7>.
* Core and Builtins: Core and Builtins<7>.
* C API: C API<5>.
* Build: Build<5>.
Python 3.13.0 final
* Core and Builtins: Core and Builtins<8>.
Python 3.13.0 release candidate 3
* macOS: macOS<5>.
* Windows: Windows<7>.
* Tests: Tests<7>.
* Library: Library<8>.
* IDLE: IDLE<6>.
* Documentation: Documentation<8>.
* Core and Builtins: Core and Builtins<9>.
* C API: C API<6>.
* Build: Build<6>.
Python 3.13.0 release candidate 2
* macOS: macOS<6>.
* Windows: Windows<8>.
* Tools/Demos: Tools/Demos<5>.
* Tests: Tests<8>.
* Security: Security<6>.
* Library: Library<9>.
* IDLE: IDLE<7>.
* Core and Builtins: Core and Builtins<10>.
* C API: C API<7>.
* Build: Build<7>.
Python 3.13.0 release candidate 1
* Tests: Tests<9>.
* Security: Security<7>.
* Library: Library<10>.
* IDLE: IDLE<8>.
* Core and Builtins: Core and Builtins<11>.
* C API: C API<8>.
* Build: Build<8>.
Python 3.13.0 beta 4
* Tests: Tests<10>.
* Library: Library<11>.
* IDLE: IDLE<9>.
* Documentation: Documentation<9>.
* Core and Builtins: Core and Builtins<12>.
* C API: C API<9>.
* Build: Build<9>.
Python 3.13.0 beta 3
* Core and Builtins: Core and Builtins<13>.
* Library: Library<12>.
* Build: Build<10>.
* C API: C API<10>.
Python 3.13.0 beta 2
* Security: Security<8>.
* Core and Builtins: Core and Builtins<14>.
* Library: Library<13>.
* Tests: Tests<11>.
* Build: Build<11>.
* Windows: Windows<9>.
* C API: C API<11>.
Python 3.13.0 beta 1
* Security: Security<9>.
* Core and Builtins: Core and Builtins<15>.
* Library: Library<14>.
* Documentation: Documentation<10>.
* Build: Build<12>.
* Windows: Windows<10>.
* macOS: macOS<7>.
* IDLE: IDLE<10>.
* C API: C API<12>.
Python 3.13.0 alpha 6
* Core and Builtins: Core and Builtins<16>.
* Library: Library<15>.
* Documentation: Documentation<11>.
* Tests: Tests<12>.
* Build: Build<13>.
* Windows: Windows<11>.
* C API: C API<13>.
Python 3.13.0 alpha 5
* Security: Security<10>.
* Core and Builtins: Core and Builtins<17>.
* Library: Library<16>.
* Documentation: Documentation<12>.
* Tests: Tests<13>.
* Build: Build<14>.
* Windows: Windows<12>.
* macOS: macOS<8>.
* IDLE: IDLE<11>.
* Tools/Demos: Tools/Demos<6>.
* C API: C API<14>.
Python 3.13.0 alpha 4
* Security: Security<11>.
* Core and Builtins: Core and Builtins<18>.
* Library: Library<17>.
* Documentation: Documentation<13>.
* Tests: Tests<14>.
* Build: Build<15>.
* Windows: Windows<13>.
* macOS: macOS<9>.
* IDLE: IDLE<12>.
* Tools/Demos: Tools/Demos<7>.
* C API: C API<15>.
Python 3.13.0 alpha 3
* Security: Security<12>.
* Core and Builtins: Core and Builtins<19>.
* Library: Library<18>.
* Documentation: Documentation<14>.
* Tests: Tests<15>.
* Build: Build<16>.
* Windows: Windows<14>.
* macOS: macOS<10>.
* IDLE: IDLE<13>.
* C API: C API<16>.
Python 3.13.0 alpha 2
* Core and Builtins: Core and Builtins<20>.
* Library: Library<19>.
* Tests: Tests<16>.
* Build: Build<17>.
* Windows: Windows<15>.
* macOS: macOS<11>.
* IDLE: IDLE<14>.
* Tools/Demos: Tools/Demos<8>.
* C API: C API<17>.
Python 3.13.0 alpha 1
* Security: Security<13>.
* Core and Builtins: Core and Builtins<21>.
* Library: Library<20>.
* Documentation: Documentation<15>.
* Tests: Tests<17>.
* Build: Build<18>.
* Windows: Windows<16>.
* macOS: macOS<12>.
* IDLE: IDLE<15>.
* Tools/Demos: Tools/Demos<9>.
* C API: C API<18>.
Python 3.12.0 beta 1
* Security: Security<14>.
* Core and Builtins: Core and Builtins<22>.
* Library: Library<21>.
* Documentation: Documentation<16>.
* Tests: Tests<18>.
* Build: Build<19>.
* Windows: Windows<17>.
* macOS: macOS<13>.
* IDLE: IDLE<16>.
* Tools/Demos: Tools/Demos<10>.
* C API: C API<19>.
Python 3.12.0 alpha 7
* Core and Builtins: Core and Builtins<23>.
* Library: Library<22>.
* Documentation: Documentation<17>.
* Tests: Tests<19>.
* Build: Build<20>.
* Windows: Windows<18>.
* Tools/Demos: Tools/Demos<11>.
* C API: C API<20>.
Python 3.12.0 alpha 6
* Security: Security<15>.
* Core and Builtins: Core and Builtins<24>.
* Library: Library<23>.
* Documentation: Documentation<18>.
* Tests: Tests<20>.
* Build: Build<21>.
* Windows: Windows<19>.
* macOS: macOS<14>.
* C API: C API<21>.
Python 3.12.0 alpha 5
* Security: Security<16>.
* Core and Builtins: Core and Builtins<25>.
* Library: Library<24>.
* Documentation: Documentation<19>.
* Tests: Tests<21>.
* Build: Build<22>.
* Windows: Windows<20>.
Python 3.12.0 alpha 4
* Core and Builtins: Core and Builtins<26>.
* Library: Library<25>.
* Documentation: Documentation<20>.
* Tests: Tests<22>.
* Build: Build<23>.
* Windows: Windows<21>.
* macOS: macOS<15>.
* Tools/Demos: Tools/Demos<12>.
* C API: C API<22>.
Python 3.12.0 alpha 3
* Security: Security<17>.
* Core and Builtins: Core and Builtins<27>.
* Library: Library<26>.
* Documentation: Documentation<21>.
* Tests: Tests<23>.
* Build: Build<24>.
* Windows: Windows<22>.
* macOS: macOS<16>.
* Tools/Demos: Tools/Demos<13>.
* C API: C API<23>.
Python 3.12.0 alpha 2
* Security: Security<18>.
* Core and Builtins: Core and Builtins<28>.
* Library: Library<27>.
* Documentation: Documentation<22>.
* Tests: Tests<24>.
* Build: Build<25>.
* Windows: Windows<23>.
* macOS: macOS<17>.
* C API: C API<24>.
Python 3.12.0 alpha 1
* Security: Security<19>.
* Core and Builtins: Core and Builtins<29>.
* Library: Library<28>.
* Documentation: Documentation<23>.
* Tests: Tests<25>.
* Build: Build<26>.
* Windows: Windows<24>.
* macOS: macOS<18>.
* IDLE: IDLE<17>.
* Tools/Demos: Tools/Demos<14>.
* C API: C API<25>.
Python 3.11.0 beta 1
* Security: Security<20>.
* Core and Builtins: Core and Builtins<30>.
* Library: Library<29>.
* Documentation: Documentation<24>.
* Tests: Tests<26>.
* Build: Build<27>.
* Windows: Windows<25>.
* macOS: macOS<19>.
* Tools/Demos: Tools/Demos<15>.
* C API: C API<26>.
Python 3.11.0 alpha 7
* Core and Builtins: Core and Builtins<31>.
* Library: Library<30>.
* Documentation: Documentation<25>.
* Tests: Tests<27>.
* Build: Build<28>.
* Windows: Windows<26>.
* macOS: macOS<20>.
* Tools/Demos: Tools/Demos<16>.
* C API: C API<27>.
Python 3.11.0 alpha 6
* Core and Builtins: Core and Builtins<32>.
* Library: Library<31>.
* Documentation: Documentation<26>.
* Tests: Tests<28>.
* Build: Build<29>.
* Windows: Windows<27>.
* IDLE: IDLE<18>.
* C API: C API<28>.
Python 3.11.0 alpha 5
* Core and Builtins: Core and Builtins<33>.
* Library: Library<32>.
* Documentation: Documentation<27>.
* Tests: Tests<29>.
* Build: Build<30>.
* Windows: Windows<28>.
* macOS: macOS<21>.
* IDLE: IDLE<19>.
* C API: C API<29>.
Python 3.11.0 alpha 4
* Core and Builtins: Core and Builtins<34>.
* Library: Library<33>.
* Documentation: Documentation<28>.
* Tests: Tests<30>.
* Build: Build<31>.
* Windows: Windows<29>.
* macOS: macOS<22>.
* C API: C API<30>.
Python 3.11.0 alpha 3
* Core and Builtins: Core and Builtins<35>.
* Library: Library<34>.
* Documentation: Documentation<29>.
* Tests: Tests<31>.
* Build: Build<32>.
* Windows: Windows<30>.
* macOS: macOS<23>.
* C API: C API<31>.
Python 3.11.0 alpha 2
* Core and Builtins: Core and Builtins<36>.
* Library: Library<35>.
* Documentation: Documentation<30>.
* Tests: Tests<32>.
* Build: Build<33>.
* Windows: Windows<31>.
* macOS: macOS<24>.
* IDLE: IDLE<20>.
* C API: C API<32>.
Python 3.11.0 alpha 1
* Security: Security<21>.
* Core and Builtins: Core and Builtins<37>.
* Library: Library<36>.
* Documentation: Documentation<31>.
* Tests: Tests<33>.
* Build: Build<34>.
* Windows: Windows<32>.
* macOS: macOS<25>.
* IDLE: IDLE<21>.
* Tools/Demos: Tools/Demos<17>.
* C API: C API<33>.
Python 3.10.0 beta 1
* Security: Security<22>.
* Core and Builtins: Core and Builtins<38>.
* Library: Library<37>.
* Documentation: Documentation<32>.
* Tests: Tests<34>.
* Build: Build<35>.
* Windows: Windows<33>.
* macOS: macOS<26>.
* IDLE: IDLE<22>.
* C API: C API<34>.
Python 3.10.0 alpha 7
* Security: Security<23>.
* Core and Builtins: Core and Builtins<39>.
* Library: Library<38>.
* Documentation: Documentation<33>.
* Tests: Tests<35>.
* Build: Build<36>.
* Windows: Windows<34>.
* IDLE: IDLE<23>.
* C API: C API<35>.
Python 3.10.0 alpha 6
* Security: Security<24>.
* Core and Builtins: Core and Builtins<40>.
* Library: Library<39>.
* Documentation: Documentation<34>.
* Tests: Tests<36>.
* Build: Build<37>.
* Windows: Windows<35>.
* macOS: macOS<27>.
* IDLE: IDLE<24>.
* C API: C API<36>.
Python 3.10.0 alpha 5
* Security: Security<25>.
* Core and Builtins: Core and Builtins<41>.
* Library: Library<40>.
* Documentation: Documentation<35>.
* Tests: Tests<37>.
* Build: Build<38>.
* Windows: Windows<36>.
* macOS: macOS<28>.
* IDLE: IDLE<25>.
* C API: C API<37>.
Python 3.10.0 alpha 4
* Core and Builtins: Core and Builtins<42>.
* Library: Library<41>.
* Documentation: Documentation<36>.
* Tests: Tests<38>.
* Build: Build<39>.
* macOS: macOS<29>.
* Tools/Demos: Tools/Demos<18>.
* C API: C API<38>.
Python 3.10.0 alpha 3
* Security: Security<26>.
* Core and Builtins: Core and Builtins<43>.
* Library: Library<42>.
* Documentation: Documentation<37>.
* Tests: Tests<39>.
* Build: Build<40>.
* Windows: Windows<37>.
* macOS: macOS<30>.
* IDLE: IDLE<26>.
* Tools/Demos: Tools/Demos<19>.
* C API: C API<39>.
Python 3.10.0 alpha 2
* Security: Security<27>.
* Core and Builtins: Core and Builtins<44>.
* Library: Library<43>.
* Documentation: Documentation<38>.
* Tests: Tests<40>.
* Build: Build<41>.
* Windows: Windows<38>.
* macOS: macOS<31>.
* IDLE: IDLE<27>.
* C API: C API<40>.
Python 3.10.0 alpha 1
* Security: Security<28>.
* Core and Builtins: Core and Builtins<45>.
* Library: Library<44>.
* Documentation: Documentation<39>.
* Tests: Tests<41>.
* Build: Build<42>.
* Windows: Windows<39>.
* macOS: macOS<32>.
* IDLE: IDLE<28>.
* C API: C API<41>.
Python 3.9.0 beta 1
* Security: Security<29>.
* Core and Builtins: Core and Builtins<46>.
* Library: Library<45>.
* Documentation: Documentation<40>.
* Tests: Tests<42>.
* Build: Build<43>.
* Windows: Windows<40>.
* macOS: macOS<33>.
* Tools/Demos: Tools/Demos<20>.
* C API: C API<42>.
Python 3.9.0 alpha 6
* Security: Security<30>.
* Core and Builtins: Core and Builtins<47>.
* Library: Library<46>.
* Documentation: Documentation<41>.
* Tests: Tests<43>.
* Build: Build<44>.
* Windows: Windows<41>.
* macOS: macOS<34>.
* IDLE: IDLE<29>.
* Tools/Demos: Tools/Demos<21>.
* C API: C API<43>.
Python 3.9.0 alpha 5
* Security: Security<31>.
* Core and Builtins: Core and Builtins<48>.
* Library: Library<47>.
* Documentation: Documentation<42>.
* Tests: Tests<44>.
* Build: Build<45>.
* Windows: Windows<42>.
* macOS: macOS<35>.
* IDLE: IDLE<30>.
* Tools/Demos: Tools/Demos<22>.
* C API: C API<44>.
Python 3.9.0 alpha 4
* Security: Security<32>.
* Core and Builtins: Core and Builtins<49>.
* Library: Library<48>.
* Documentation: Documentation<43>.
* Tests: Tests<45>.
* Build: Build<46>.
* Windows: Windows<43>.
* IDLE: IDLE<31>.
* C API: C API<45>.
Python 3.9.0 alpha 3
* Core and Builtins: Core and Builtins<50>.
* Library: Library<49>.
* Documentation: Documentation<44>.
* Build: Build<47>.
* IDLE: IDLE<32>.
* C API: C API<46>.
Python 3.9.0 alpha 2
* Security: Security<33>.
* Core and Builtins: Core and Builtins<51>.
* Library: Library<50>.
* Documentation: Documentation<45>.
* Tests: Tests<46>.
* Build: Build<48>.
* Windows: Windows<44>.
* macOS: macOS<36>.
* IDLE: IDLE<33>.
* C API: C API<47>.
Python 3.9.0 alpha 1
* Security: Security<34>.
* Core and Builtins: Core and Builtins<52>.
* Library: Library<51>.
* Documentation: Documentation<46>.
* Tests: Tests<47>.
* Build: Build<49>.
* Windows: Windows<45>.
* macOS: macOS<37>.
* IDLE: IDLE<34>.
* Tools/Demos: Tools/Demos<23>.
* C API: C API<48>.
Python 3.8.0 beta 1
* Security: Security<35>.
* Core and Builtins: Core and Builtins<53>.
* Library: Library<52>.
* Documentation: Documentation<47>.
* Tests: Tests<48>.
* Build: Build<50>.
* Windows: Windows<46>.
* macOS: macOS<38>.
* IDLE: IDLE<35>.
* Tools/Demos: Tools/Demos<24>.
* C API: C API<49>.
Python 3.8.0 alpha 4
* Security: Security<36>.
* Core and Builtins: Core and Builtins<54>.
* Library: Library<53>.
* Documentation: Documentation<48>.
* Tests: Tests<49>.
* Build: Build<51>.
* Windows: Windows<47>.
* macOS: macOS<39>.
* IDLE: IDLE<36>.
* Tools/Demos: Tools/Demos<25>.
* C API: C API<50>.
Python 3.8.0 alpha 3
* Security: Security<37>.
* Core and Builtins: Core and Builtins<55>.
* Library: Library<54>.
* Documentation: Documentation<49>.
* Tests: Tests<50>.
* Build: Build<52>.
* Windows: Windows<48>.
* IDLE: IDLE<37>.
* Tools/Demos: Tools/Demos<26>.
* C API: C API<51>.
Python 3.8.0 alpha 2
* Core and Builtins: Core and Builtins<56>.
* Library: Library<55>.
* Documentation: Documentation<50>.
* Tests: Tests<51>.
* Windows: Windows<49>.
* IDLE: IDLE<38>.
Python 3.8.0 alpha 1
* Security: Security<38>.
* Core and Builtins: Core and Builtins<57>.
* Library: Library<56>.
* Documentation: Documentation<51>.
* Tests: Tests<52>.
* Build: Build<53>.
* Windows: Windows<50>.
* macOS: macOS<40>.
* IDLE: IDLE<39>.
* Tools/Demos: Tools/Demos<27>.
* C API: C API<52>.
Python 3.7.0 final
* Library: Library<57>.
* C API: C API<53>.
Python 3.7.0 release candidate 1
* Core and Builtins: Core and Builtins<58>.
* Library: Library<58>.
* Documentation: Documentation<52>.
* Build: Build<54>.
* Windows: Windows<51>.
* IDLE: IDLE<40>.
Python 3.7.0 beta 5
* Core and Builtins: Core and Builtins<59>.
* Library: Library<59>.
* Documentation: Documentation<53>.
* Tests: Tests<53>.
* Build: Build<55>.
* macOS: macOS<41>.
* IDLE: IDLE<41>.
Python 3.7.0 beta 4
* Core and Builtins: Core and Builtins<60>.
* Library: Library<60>.
* Documentation: Documentation<54>.
* Tests: Tests<54>.
* Build: Build<56>.
* Windows: Windows<52>.
* macOS: macOS<42>.
* IDLE: IDLE<42>.
* Tools/Demos: Tools/Demos<28>.
Python 3.7.0 beta 3
* Security: Security<39>.
* Core and Builtins: Core and Builtins<61>.
* Library: Library<61>.
* Documentation: Documentation<55>.
* Tests: Tests<55>.
* Build: Build<57>.
* Windows: Windows<53>.
* macOS: macOS<43>.
* IDLE: IDLE<43>.
* Tools/Demos: Tools/Demos<29>.
* C API: C API<54>.
Python 3.7.0 beta 2
* Security: Security<40>.
* Core and Builtins: Core and Builtins<62>.
* Library: Library<62>.
* Documentation: Documentation<56>.
* Tests: Tests<56>.
* Build: Build<58>.
* Windows: Windows<54>.
* macOS: macOS<44>.
* IDLE: IDLE<44>.
* Tools/Demos: Tools/Demos<30>.
Python 3.7.0 beta 1
* Core and Builtins: Core and Builtins<63>.
* Library: Library<63>.
* Documentation: Documentation<57>.
* Tests: Tests<57>.
* Build: Build<59>.
* Windows: Windows<55>.
* macOS: macOS<45>.
* C API: C API<55>.
Python 3.7.0 alpha 4
* Core and Builtins: Core and Builtins<64>.
* Library: Library<64>.
* Documentation: Documentation<58>.
* Tests: Tests<58>.
* Windows: Windows<56>.
* Tools/Demos: Tools/Demos<31>.
* C API: C API<56>.
Python 3.7.0 alpha 3
* Core and Builtins: Core and Builtins<65>.
* Library: Library<65>.
* Documentation: Documentation<59>.
* Tests: Tests<59>.
* Build: Build<60>.
* Windows: Windows<57>.
* macOS: macOS<46>.
* IDLE: IDLE<45>.
* Tools/Demos: Tools/Demos<32>.
* C API: C API<57>.
Python 3.7.0 alpha 2
* Core and Builtins: Core and Builtins<66>.
* Library: Library<66>.
* Documentation: Documentation<60>.
* Build: Build<61>.
* IDLE: IDLE<46>.
* C API: C API<58>.
Python 3.7.0 alpha 1
* Security: Security<41>.
* Core and Builtins: Core and Builtins<67>.
* Library: Library<67>.
* Documentation: Documentation<61>.
* Tests: Tests<60>.
* Build: Build<62>.
* Windows: Windows<58>.
* IDLE: IDLE<47>.
* Tools/Demos: Tools/Demos<33>.
* C API: C API<59>.
Python 3.6.6 release candidate 1
* Core and Builtins: Core and Builtins<68>.
* Library: Library<68>.
* Documentation: Documentation<62>.
* Tests: Tests<61>.
* Build: Build<63>.
* Windows: Windows<59>.
* macOS: macOS<47>.
* IDLE: IDLE<48>.
* Tools/Demos: Tools/Demos<34>.
* C API: C API<60>.
Python 3.6.5 final
* Tests: Tests<62>.
* Build: Build<64>.
Python 3.6.5 release candidate 1
* Security: Security<42>.
* Core and Builtins: Core and Builtins<69>.
* Library: Library<69>.
* Documentation: Documentation<63>.
* Tests: Tests<63>.
* Build: Build<65>.
* Windows: Windows<60>.
* macOS: macOS<48>.
* IDLE: IDLE<49>.
* Tools/Demos: Tools/Demos<35>.
* C API: C API<61>.
Python 3.6.4 release candidate 1
* Core and Builtins: Core and Builtins<70>.
* Library: Library<70>.
* Documentation: Documentation<64>.
* Tests: Tests<64>.
* Build: Build<66>.
* Windows: Windows<61>.
* macOS: macOS<49>.
* IDLE: IDLE<50>.
* Tools/Demos: Tools/Demos<36>.
* C API: C API<62>.
Python 3.6.3 final
* Library: Library<71>.
* Build: Build<67>.
Python 3.6.3 release candidate 1
* Security: Security<43>.
* Core and Builtins: Core and Builtins<71>.
* Library: Library<72>.
* Documentation: Documentation<65>.
* Tests: Tests<65>.
* Build: Build<68>.
* Windows: Windows<62>.
* IDLE: IDLE<51>.
* Tools/Demos: Tools/Demos<37>.
Python 3.6.2 release candidate 2
* Security: Security<44>.
Python 3.6.2 release candidate 1
* Security: Security<45>.
* Core and Builtins: Core and Builtins<72>.
* Library: Library<73>.
* IDLE: IDLE<52>.
* C API: C API<63>.
* Build: Build<69>.
* Documentation: Documentation<66>.
* Tools/Demos: Tools/Demos<38>.
* Tests: Tests<66>.
* Windows: Windows<63>.
Python 3.6.1 final
* Core and Builtins: Core and Builtins<73>.
* Build: Build<70>.
Python 3.6.1 release candidate 1
* Core and Builtins: Core and Builtins<74>.
* Library: Library<74>.
* IDLE: IDLE<53>.
* Windows: Windows<64>.
* C API: C API<64>.
* Documentation: Documentation<67>.
* Tests: Tests<67>.
* Build: Build<71>.
Python 3.6.0 release candidate 2
* Core and Builtins: Core and Builtins<75>.
* Tools/Demos: Tools/Demos<39>.
* Windows: Windows<65>.
* Build: Build<72>.
Python 3.6.0 release candidate 1
* Core and Builtins: Core and Builtins<76>.
* Library: Library<75>.
* C API: C API<65>.
* Documentation: Documentation<68>.
* Tools/Demos: Tools/Demos<40>.
Python 3.6.0 beta 4
* Core and Builtins: Core and Builtins<77>.
* Library: Library<76>.
* Documentation: Documentation<69>.
* Tests: Tests<68>.
* Build: Build<73>.
Python 3.6.0 beta 3
* Core and Builtins: Core and Builtins<78>.
* Library: Library<77>.
* Windows: Windows<66>.
* Build: Build<74>.
* Tests: Tests<69>.
Python 3.6.0 beta 2
* Core and Builtins: Core and Builtins<79>.
* Library: Library<78>.
* Windows: Windows<67>.
* C API: C API<66>.
* Build: Build<75>.
* Tests: Tests<70>.
Python 3.6.0 beta 1
* Core and Builtins: Core and Builtins<80>.
* Library: Library<79>.
* IDLE: IDLE<54>.
* C API: C API<67>.
* Tests: Tests<71>.
* Build: Build<76>.
* Tools/Demos: Tools/Demos<41>.
* Windows: Windows<68>.
Python 3.6.0 alpha 4
* Core and Builtins: Core and Builtins<81>.
* Library: Library<80>.
* IDLE: IDLE<55>.
* Tests: Tests<72>.
* Windows: Windows<69>.
* Build: Build<77>.
Python 3.6.0 alpha 3
* Security: Security<46>.
* Core and Builtins: Core and Builtins<82>.
* Library: Library<81>.
* IDLE: IDLE<56>.
* C API: C API<68>.
* Build: Build<78>.
* Tools/Demos: Tools/Demos<42>.
* Documentation: Documentation<70>.
* Tests: Tests<73>.
Python 3.6.0 alpha 2
* Security: Security<47>.
* Core and Builtins: Core and Builtins<83>.
* Library: Library<82>.
* IDLE: IDLE<57>.
* Documentation: Documentation<71>.
* Tests: Tests<74>.
* Windows: Windows<70>.
* Build: Build<79>.
* C API: C API<69>.
* Tools/Demos: Tools/Demos<43>.
Python 3.6.0 alpha 1
* Security: Security<48>.
* Core and Builtins: Core and Builtins<84>.
* Library: Library<83>.
* IDLE: IDLE<58>.
* Documentation: Documentation<72>.
* Tests: Tests<75>.
* Build: Build<80>.
* Windows: Windows<71>.
* Tools/Demos: Tools/Demos<44>.
* C API: C API<70>.
Python 3.5.5 release candidate 1
* Security: Security<49>.
* Core and Builtins: Core and Builtins<85>.
* Library: Library<84>.
Python 3.5.4 final
* Library: Library<85>.
Python 3.5.4 release candidate 1
* Security: Security<50>.
* Core and Builtins: Core and Builtins<86>.
* Library: Library<86>.
* Documentation: Documentation<73>.
* Tests: Tests<76>.
* Build: Build<81>.
* Windows: Windows<72>.
* C API: C API<71>.
Python 3.5.3 release candidate 1
* Security: Security<51>.
* Core and Builtins: Core and Builtins<87>.
* Library: Library<87>.
* IDLE: IDLE<59>.
* C API: C API<72>.
* Documentation: Documentation<74>.
* Tests: Tests<77>.
* Tools/Demos: Tools/Demos<45>.
* Windows: Windows<73>.
* Build: Build<82>.
Python 3.5.2 final
* Core and Builtins: Core and Builtins<88>.
* Tests: Tests<78>.
* IDLE: IDLE<60>.
Python 3.5.2 release candidate 1
* Security: Security<52>.
* Core and Builtins: Core and Builtins<89>.
* Library: Library<88>.
* IDLE: IDLE<61>.
* Documentation: Documentation<75>.
* Tests: Tests<79>.
* Build: Build<83>.
* Windows: Windows<74>.
* Tools/Demos: Tools/Demos<46>.
Python 3.5.1 final
* Core and Builtins: Core and Builtins<90>.
* Windows: Windows<75>.
Python 3.5.1 release candidate 1
* Core and Builtins: Core and Builtins<91>.
* Library: Library<89>.
* IDLE: IDLE<62>.
* Documentation: Documentation<76>.
* Tests: Tests<80>.
* Build: Build<84>.
* Windows: Windows<76>.
* Tools/Demos: Tools/Demos<47>.
Python 3.5.0 final
* Build: Build<85>.
Python 3.5.0 release candidate 4
* Library: Library<90>.
* Build: Build<86>.
Python 3.5.0 release candidate 3
* Core and Builtins: Core and Builtins<92>.
* Library: Library<91>.
Python 3.5.0 release candidate 2
* Core and Builtins: Core and Builtins<93>.
* Library: Library<92>.
Python 3.5.0 release candidate 1
* Core and Builtins: Core and Builtins<94>.
* Library: Library<93>.
* IDLE: IDLE<63>.
* Documentation: Documentation<77>.
* Tests: Tests<81>.
Python 3.5.0 beta 4
* Core and Builtins: Core and Builtins<95>.
* Library: Library<94>.
* Build: Build<87>.
Python 3.5.0 beta 3
* Core and Builtins: Core and Builtins<96>.
* Library: Library<95>.
* Tests: Tests<82>.
* Documentation: Documentation<78>.
* Build: Build<88>.
Python 3.5.0 beta 2
* Core and Builtins: Core and Builtins<97>.
* Library: Library<96>.
Python 3.5.0 beta 1
* Core and Builtins: Core and Builtins<98>.
* Library: Library<97>.
* IDLE: IDLE<64>.
* Tests: Tests<83>.
* Documentation: Documentation<79>.
* Tools/Demos: Tools/Demos<48>.
Python 3.5.0 alpha 4
* Core and Builtins: Core and Builtins<99>.
* Library: Library<98>.
* Build: Build<89>.
* Tests: Tests<84>.
* Tools/Demos: Tools/Demos<49>.
* C API: C API<73>.
Python 3.5.0 alpha 3
* Core and Builtins: Core and Builtins<100>.
* Library: Library<99>.
* Build: Build<90>.
* Tests: Tests<85>.
* Tools/Demos: Tools/Demos<50>.
Python 3.5.0 alpha 2
* Core and Builtins: Core and Builtins<101>.
* Library: Library<100>.
* Build: Build<91>.
* C API: C API<74>.
* Windows: Windows<77>.
Python 3.5.0 alpha 1
* Core and Builtins: Core and Builtins<102>.
* Library: Library<101>.
* IDLE: IDLE<65>.
* Build: Build<92>.
* C API: C API<75>.
* Documentation: Documentation<80>.
* Tests: Tests<86>.
* Tools/Demos: Tools/Demos<51>.
* Windows: Windows<78>.
The Python Tutorial
* Whetting Your Appetite::
* Using the Python Interpreter::
* An Informal Introduction to Python::
* More Control Flow Tools::
* Data Structures::
* Modules: Modules<2>.
* Input and Output::
* Errors and Exceptions::
* Classes::
* Brief Tour of the Standard Library::
* Brief Tour of the Standard Library — Part II::
* Virtual Environments and Packages::
* What Now?::
* Interactive Input Editing and History Substitution::
* Floating-Point Arithmetic; Issues and Limitations: Floating-Point Arithmetic Issues and Limitations.
* Appendix::
Using the Python Interpreter
* Invoking the Interpreter::
* The Interpreter and Its Environment::
Invoking the Interpreter
* Argument Passing::
* Interactive Mode::
The Interpreter and Its Environment
* Source Code Encoding::
An Informal Introduction to Python
* Using Python as a Calculator::
* First Steps Towards Programming::
Using Python as a Calculator
* Numbers::
* Text::
* Lists::
More Control Flow Tools
* if Statements::
* for Statements::
* The range() Function: The range Function.
* break and continue Statements::
* else Clauses on Loops::
* pass Statements::
* match Statements::
* Defining Functions::
* More on Defining Functions::
* Intermezzo; Coding Style: Intermezzo Coding Style.
More on Defining Functions
* Default Argument Values::
* Keyword Arguments::
* Special parameters::
* Arbitrary Argument Lists::
* Unpacking Argument Lists::
* Lambda Expressions::
* Documentation Strings::
* Function Annotations::
Special parameters
* Positional-or-Keyword Arguments::
* Positional-Only Parameters::
* Keyword-Only Arguments::
* Function Examples::
* Recap::
Data Structures
* More on Lists::
* The del statement::
* Tuples and Sequences::
* Sets::
* Dictionaries::
* Looping Techniques::
* More on Conditions::
* Comparing Sequences and Other Types::
More on Lists
* Using Lists as Stacks::
* Using Lists as Queues::
* List Comprehensions: List Comprehensions<2>.
* Nested List Comprehensions::
Modules
* More on Modules::
* Standard Modules::
* The dir() Function: The dir Function.
* Packages::
More on Modules
* Executing modules as scripts::
* The Module Search Path::
* “Compiled” Python files::
Packages
* Importing * From a Package::
* Intra-package References::
* Packages in Multiple Directories::
Input and Output
* Fancier Output Formatting::
* Reading and Writing Files::
Fancier Output Formatting
* Formatted String Literals::
* The String format() Method: The String format Method.
* Manual String Formatting::
* Old string formatting::
Reading and Writing Files
* Methods of File Objects::
* Saving structured data with json::
Errors and Exceptions
* Syntax Errors::
* Exceptions::
* Handling Exceptions::
* Raising Exceptions::
* Exception Chaining::
* User-defined Exceptions::
* Defining Clean-up Actions::
* Predefined Clean-up Actions::
* Raising and Handling Multiple Unrelated Exceptions::
* Enriching Exceptions with Notes::
Classes
* A Word About Names and Objects::
* Python Scopes and Namespaces::
* A First Look at Classes::
* Random Remarks::
* Inheritance::
* Private Variables::
* Odds and Ends::
* Iterators::
* Generators::
* Generator Expressions::
Python Scopes and Namespaces
* Scopes and Namespaces Example::
A First Look at Classes
* Class Definition Syntax::
* Class Objects::
* Instance Objects::
* Method Objects::
* Class and Instance Variables::
Inheritance
* Multiple Inheritance::
Brief Tour of the Standard Library
* Operating System Interface::
* File Wildcards::
* Command Line Arguments::
* Error Output Redirection and Program Termination::
* String Pattern Matching::
* Mathematics::
* Internet Access::
* Dates and Times::
* Data Compression::
* Performance Measurement::
* Quality Control::
* Batteries Included::
Brief Tour of the Standard Library — Part II
* Output Formatting::
* Templating::
* Working with Binary Data Record Layouts::
* Multi-threading: Multi-threading<2>.
* Logging::
* Weak References::
* Tools for Working with Lists::
* Decimal Floating-Point Arithmetic::
Virtual Environments and Packages
* Introduction: Introduction<4>.
* Creating Virtual Environments::
* Managing Packages with pip::
Interactive Input Editing and History Substitution
* Tab Completion and History Editing::
* Alternatives to the Interactive Interpreter::
Floating-Point Arithmetic: Issues and Limitations
* Representation Error::
Appendix
* Interactive Mode: Interactive Mode<2>.
Interactive Mode
* Error Handling::
* Executable Python Scripts::
* The Interactive Startup File::
* The Customization Modules::
Python Setup and Usage
* Command line and environment::
* Using Python on Unix platforms::
* Configure Python::
* Using Python on Windows::
* Using Python on macOS::
* Using Python on Android::
* Using Python on iOS::
* Editors and IDEs::
Command line and environment
* Command line::
* Environment variables::
Command line
* Interface options::
* Generic options::
* Miscellaneous options::
* Controlling color::
* Options you shouldn’t use::
Environment variables
* Debug-mode variables::
Using Python on Unix platforms
* Getting and installing the latest version of Python::
* Building Python::
* Python-related paths and files::
* Miscellaneous::
* Custom OpenSSL::
Getting and installing the latest version of Python
* On Linux::
* On FreeBSD and OpenBSD::
On Linux
* Installing IDLE::
Configure Python
* Build Requirements::
* Generated files::
* Configure Options::
* Python Build System::
* Compiler and linker flags::
Generated files
* configure script::
Configure Options
* General Options::
* C compiler options::
* Linker options::
* Options for third-party dependencies::
* WebAssembly Options::
* Install Options::
* Performance options::
* Python Debug Build::
* Debug options::
* Linker options: Linker options<2>.
* Libraries options::
* Security Options::
* macOS Options::
* iOS Options::
* Cross Compiling Options::
Python Build System
* Main files of the build system::
* Main build steps::
* Main Makefile targets::
* C extensions::
Main Makefile targets
* make::
* make platform::
* make profile-opt::
* make clean::
* make distclean::
* make install::
* make test::
* make buildbottest::
* make regen-all::
Compiler and linker flags
* Preprocessor flags::
* Compiler flags::
* Linker flags::
Using Python on Windows
* The full installer::
* The Microsoft Store package::
* The nuget.org packages: The nuget org packages.
* The embeddable package::
* Alternative bundles::
* Configuring Python::
* UTF-8 mode::
* Python Launcher for Windows::
* Finding modules::
* Additional modules::
* Compiling Python on Windows::
* Other Platforms::
The full installer
* Installation steps::
* Removing the MAX_PATH Limitation::
* Installing Without UI::
* Installing Without Downloading::
* Modifying an install::
* Installing Free-threaded Binaries::
The Microsoft Store package
* Known issues::
Known issues
* Redirection of local data, registry, and temporary paths: Redirection of local data registry and temporary paths.
The nuget.org packages
* Free-threaded packages::
The embeddable package
* Python Application::
* Embedding Python::
Configuring Python
* Excursus; Setting environment variables: Excursus Setting environment variables.
* Finding the Python executable::
Python Launcher for Windows
* Getting started::
* Shebang Lines::
* Arguments in shebang lines::
* Customization::
* Diagnostics::
* Dry Run::
* Install on demand::
* Return codes::
Getting started
* From the command-line::
* Virtual environments::
* From a script::
* From file associations::
Customization
* Customization via INI files::
* Customizing default Python versions::
Additional modules
* PyWin32::
* cx_Freeze::
Using Python on macOS
* Using Python for macOS from python.org: Using Python for macOS from python org.
* Alternative Distributions::
* Installing Additional Python Packages::
* GUI Programming::
* Advanced Topics::
* Other Resources::
Using Python for macOS from python.org
* Installation steps: Installation steps<2>.
* How to run a Python script::
Advanced Topics
* Installing Free-threaded Binaries: Installing Free-threaded Binaries<2>.
* Installing using the command line::
* Distributing Python Applications::
* App Store Compliance::
Using Python on Android
* Adding Python to an Android app::
* Building a Python package for Android::
Using Python on iOS
* Python at runtime on iOS::
* Installing Python on iOS::
* App Store Compliance: App Store Compliance<2>.
Python at runtime on iOS
* iOS version compatibility::
* Platform identification::
* Standard library availability::
* Binary extension modules::
* Compiler stub binaries::
Installing Python on iOS
* Tools for building iOS apps::
* Adding Python to an iOS project::
* Testing a Python package::
Editors and IDEs
* IDLE — Python editor and shell::
* Other Editors and IDEs::
The Python Language Reference
* Introduction: Introduction<5>.
* Lexical analysis::
* Data model::
* Execution model::
* The import system::
* Expressions::
* Simple statements::
* Compound statements::
* Top-level components::
* Full Grammar specification::
Introduction
* Alternate Implementations::
* Notation::
Lexical analysis
* Line structure::
* Other tokens::
* Identifiers and keywords::
* Literals::
* Operators::
* Delimiters::
Line structure
* Logical lines::
* Physical lines::
* Comments::
* Encoding declarations::
* Explicit line joining::
* Implicit line joining::
* Blank lines::
* Indentation::
* Whitespace between tokens::
Identifiers and keywords
* Keywords::
* Soft Keywords::
* Reserved classes of identifiers::
Literals
* String and Bytes literals::
* String literal concatenation::
* f-strings::
* Numeric literals::
* Integer literals::
* Floating-point literals::
* Imaginary literals::
String and Bytes literals
* Escape sequences::
Data model
* Objects, values and types: Objects values and types.
* The standard type hierarchy::
* Special method names::
* Coroutines::
The standard type hierarchy
* None::
* NotImplemented::
* Ellipsis::
* numbers.Number: numbers Number.
* Sequences::
* Set types::
* Mappings::
* Callable types::
* Modules: Modules<3>.
* Custom classes::
* Class instances::
* I/O objects (also known as file objects): I/O objects also known as file objects.
* Internal types::
numbers.Number
* numbers.Integral: numbers Integral.
* numbers.Real (float): numbers Real float.
* numbers.Complex (complex): numbers Complex complex.
Sequences
* Immutable sequences::
* Mutable sequences::
Mappings
* Dictionaries: Dictionaries<2>.
Callable types
* User-defined functions::
* Instance methods::
* Generator functions::
* Coroutine functions::
* Asynchronous generator functions::
* Built-in functions::
* Built-in methods::
* Classes: Classes<2>.
* Class Instances::
User-defined functions
* Special read-only attributes::
* Special writable attributes::
Modules
* Import-related attributes on module objects::
* Other writable attributes on module objects::
* Module dictionaries::
Custom classes
* Special attributes::
* Special methods::
Class instances
* Special attributes: Special attributes<2>.
Internal types
* Code objects::
* Frame objects::
* Traceback objects::
* Slice objects::
* Static method objects::
* Class method objects::
Code objects
* Special read-only attributes: Special read-only attributes<2>.
* Methods on code objects::
Frame objects
* Special read-only attributes: Special read-only attributes<3>.
* Special writable attributes: Special writable attributes<2>.
* Frame object methods::
Special method names
* Basic customization::
* Customizing attribute access::
* Customizing class creation::
* Customizing instance and subclass checks::
* Emulating generic types::
* Emulating callable objects::
* Emulating container types::
* Emulating numeric types::
* With Statement Context Managers::
* Customizing positional arguments in class pattern matching::
* Emulating buffer types::
* Special method lookup::
Customizing attribute access
* Customizing module attribute access::
* Implementing Descriptors::
* Invoking Descriptors::
* __slots__: __slots__<2>.
Customizing class creation
* Metaclasses::
* Resolving MRO entries::
* Determining the appropriate metaclass::
* Preparing the class namespace::
* Executing the class body::
* Creating the class object::
* Uses for metaclasses::
Emulating generic types
* The purpose of __class_getitem__::
* __class_getitem__ versus __getitem__::
Coroutines
* Awaitable Objects::
* Coroutine Objects::
* Asynchronous Iterators::
* Asynchronous Context Managers::
Execution model
* Structure of a program::
* Naming and binding::
* Exceptions: Exceptions<2>.
Naming and binding
* Binding of names::
* Resolution of names::
* Annotation scopes::
* Lazy evaluation::
* Builtins and restricted execution::
* Interaction with dynamic features::
The import system
* importlib: importlib<10>.
* Packages: Packages<2>.
* Searching::
* Loading::
* The Path Based Finder::
* Replacing the standard import system::
* Package Relative Imports::
* Special considerations for __main__::
* References::
Packages
* Regular packages::
* Namespace packages::
Searching
* The module cache::
* Finders and loaders::
* Import hooks::
* The meta path::
Loading
* Loaders::
* Submodules::
* Module specs::
* __path__ attributes on modules::
* Module reprs::
* Cached bytecode invalidation::
The Path Based Finder
* Path entry finders::
* Path entry finder protocol::
Special considerations for __main__
* __main__.__spec__: __main__ __spec__.
Expressions
* Arithmetic conversions::
* Atoms::
* Primaries::
* Await expression::
* The power operator::
* Unary arithmetic and bitwise operations::
* Binary arithmetic operations::
* Shifting operations::
* Binary bitwise operations::
* Comparisons::
* Boolean operations::
* Assignment expressions: Assignment expressions<2>.
* Conditional expressions::
* Lambdas::
* Expression lists::
* Evaluation order::
* Operator precedence::
Atoms
* Identifiers (Names): Identifiers Names.
* Literals: Literals<2>.
* Parenthesized forms::
* Displays for lists, sets and dictionaries: Displays for lists sets and dictionaries.
* List displays::
* Set displays::
* Dictionary displays::
* Generator expressions::
* Yield expressions::
Identifiers (Names)
* Private name mangling::
Yield expressions
* Generator-iterator methods::
* Examples::
* Asynchronous generator functions: Asynchronous generator functions<2>.
* Asynchronous generator-iterator methods::
Primaries
* Attribute references::
* Subscriptions::
* Slicings::
* Calls::
Comparisons
* Value comparisons::
* Membership test operations::
* Identity comparisons::
Simple statements
* Expression statements::
* Assignment statements::
* The assert statement::
* The pass statement::
* The del statement: The del statement<2>.
* The return statement::
* The yield statement::
* The raise statement::
* The break statement::
* The continue statement::
* The import statement::
* The global statement::
* The nonlocal statement::
* The type statement::
Assignment statements
* Augmented assignment statements::
* Annotated assignment statements::
The import statement
* Future statements::
Compound statements
* The if statement::
* The while statement::
* The for statement::
* The try statement::
* The with statement::
* The match statement::
* Function definitions::
* Class definitions::
* Coroutines: Coroutines<2>.
* Type parameter lists::
The try statement
* except clause::
* except* clause::
* else clause::
* finally clause::
The match statement
* Overview::
* Guards::
* Irrefutable Case Blocks::
* Patterns::
Patterns
* OR Patterns::
* AS Patterns::
* Literal Patterns::
* Capture Patterns::
* Wildcard Patterns::
* Value Patterns::
* Group Patterns::
* Sequence Patterns::
* Mapping Patterns::
* Class Patterns::
Coroutines
* Coroutine function definition::
* The async for statement::
* The async with statement::
Type parameter lists
* Generic functions::
* Generic classes::
* Generic type aliases::
Top-level components
* Complete Python programs::
* File input::
* Interactive input::
* Expression input::
The Python Standard Library
* Introduction: Introduction<6>.
* Built-in Functions::
* Built-in Constants::
* Built-in Types::
* Built-in Exceptions::
* Text Processing Services::
* Binary Data Services::
* Data Types::
* Numeric and Mathematical Modules::
* Functional Programming Modules::
* File and Directory Access::
* Data Persistence::
* Data Compression and Archiving::
* File Formats::
* Cryptographic Services::
* Generic Operating System Services::
* Command Line Interface Libraries::
* Concurrent Execution::
* Networking and Interprocess Communication::
* Internet Data Handling::
* Structured Markup Processing Tools::
* Internet Protocols and Support::
* Multimedia Services::
* Internationalization::
* Program Frameworks::
* Graphical User Interfaces with Tk::
* Development Tools::
* Debugging and Profiling::
* Software Packaging and Distribution::
* Python Runtime Services::
* Custom Python Interpreters::
* Importing Modules::
* Python Language Services::
* MS Windows Specific Services::
* Unix Specific Services::
* Modules command-line interface (CLI): Modules command-line interface CLI.
* Superseded Modules::
* Removed Modules::
* Security Considerations: Security Considerations<3>.
Introduction
* Notes on availability::
Notes on availability
* WebAssembly platforms::
* Mobile platforms::
Built-in Constants
* Constants added by the site module::
Built-in Types
* Truth Value Testing::
* Boolean Operations — and, or, not: Boolean Operations — and or not.
* Comparisons: Comparisons<2>.
* Numeric Types — int, float, complex: Numeric Types — int float complex.
* Boolean Type - bool::
* Iterator Types::
* Sequence Types — list, tuple, range: Sequence Types — list tuple range.
* Text Sequence Type — str::
* Binary Sequence Types — bytes, bytearray, memoryview: Binary Sequence Types — bytes bytearray memoryview.
* Set Types — set, frozenset: Set Types — set frozenset.
* Mapping Types — dict::
* Context Manager Types::
* Type Annotation Types — Generic Alias, Union: Type Annotation Types — Generic Alias Union.
* Other Built-in Types::
* Special Attributes::
* Integer string conversion length limitation::
Numeric Types — int, float, complex
* Bitwise Operations on Integer Types::
* Additional Methods on Integer Types::
* Additional Methods on Float::
* Hashing of numeric types::
Iterator Types
* Generator Types::
Sequence Types — list, tuple, range
* Common Sequence Operations::
* Immutable Sequence Types::
* Mutable Sequence Types::
* Lists: Lists<2>.
* Tuples::
* Ranges::
Text Sequence Type — str
* String Methods: String Methods<2>.
* Formatted String Literals (f-strings): Formatted String Literals f-strings.
* printf-style String Formatting::
Binary Sequence Types — bytes, bytearray, memoryview
* Bytes Objects::
* Bytearray Objects::
* Bytes and Bytearray Operations::
* printf-style Bytes Formatting::
* Memory Views::
Mapping Types — dict
* Dictionary view objects::
Type Annotation Types — Generic Alias, Union
* Generic Alias Type::
* Union Type::
Generic Alias Type
* Standard Generic Classes::
* Special Attributes of GenericAlias objects::
Other Built-in Types
* Modules: Modules<4>.
* Classes and Class Instances::
* Functions::
* Methods::
* Code Objects::
* Type Objects::
* The Null Object::
* The Ellipsis Object::
* The NotImplemented Object::
* Internal Objects::
Integer string conversion length limitation
* Affected APIs::
* Configuring the limit::
* Recommended configuration::
Built-in Exceptions
* Exception context::
* Inheriting from built-in exceptions::
* Base classes::
* Concrete exceptions::
* Warnings::
* Exception groups::
* Exception hierarchy::
Concrete exceptions
* OS exceptions::
Text Processing Services
* string — Common string operations::
* re — Regular expression operations::
* difflib — Helpers for computing deltas::
* textwrap — Text wrapping and filling::
* unicodedata — Unicode Database::
* stringprep — Internet String Preparation::
* readline — GNU readline interface::
* rlcompleter — Completion function for GNU readline::
string — Common string operations
* String constants::
* Custom String Formatting::
* Format String Syntax::
* Template strings::
* Helper functions::
Format String Syntax
* Format Specification Mini-Language::
* Format examples::
re — Regular expression operations
* Regular Expression Syntax::
* Module Contents::
* Regular Expression Objects::
* Match Objects::
* Regular Expression Examples::
Module Contents
* Flags::
* Functions: Functions<2>.
* Exceptions: Exceptions<3>.
Regular Expression Examples
* Checking for a Pair::
* Simulating scanf(): Simulating scanf.
* search() vs. match(): search vs match.
* Making a Phonebook::
* Text Munging::
* Finding all Adverbs::
* Finding all Adverbs and their Positions::
* Raw String Notation::
* Writing a Tokenizer::
difflib — Helpers for computing deltas
* SequenceMatcher Objects::
* SequenceMatcher Examples::
* Differ Objects::
* Differ Example::
* A command-line interface to difflib::
* ndiff example::
readline — GNU readline interface
* Init file::
* Line buffer::
* History file::
* History list::
* Startup hooks::
* Completion::
* Example::
Binary Data Services
* struct — Interpret bytes as packed binary data::
* codecs — Codec registry and base classes::
struct — Interpret bytes as packed binary data
* Functions and Exceptions::
* Format Strings::
* Applications::
* Classes: Classes<3>.
Format Strings
* Byte Order, Size, and Alignment: Byte Order Size and Alignment.
* Format Characters::
* Examples: Examples<2>.
Applications
* Native Formats::
* Standard Formats::
codecs — Codec registry and base classes
* Codec Base Classes::
* Encodings and Unicode::
* Standard Encodings::
* Python Specific Encodings::
* encodings — Encodings package::
* encodings.idna — Internationalized Domain Names in Applications: encodings idna — Internationalized Domain Names in Applications.
* encodings.mbcs — Windows ANSI codepage: encodings mbcs — Windows ANSI codepage.
* encodings.utf_8_sig — UTF-8 codec with BOM signature: encodings utf_8_sig — UTF-8 codec with BOM signature.
Codec Base Classes
* Error Handlers::
* Stateless Encoding and Decoding::
* Incremental Encoding and Decoding::
* Stream Encoding and Decoding::
Incremental Encoding and Decoding
* IncrementalEncoder Objects::
* IncrementalDecoder Objects::
Stream Encoding and Decoding
* StreamWriter Objects::
* StreamReader Objects::
* StreamReaderWriter Objects::
* StreamRecoder Objects::
Python Specific Encodings
* Text Encodings::
* Binary Transforms::
* Standalone Codec Functions::
* Text Transforms::
Data Types
* datetime — Basic date and time types::
* zoneinfo — IANA time zone support::
* calendar — General calendar-related functions::
* collections — Container datatypes::
* collections.abc — Abstract Base Classes for Containers: collections abc — Abstract Base Classes for Containers.
* heapq — Heap queue algorithm::
* bisect — Array bisection algorithm::
* array — Efficient arrays of numeric values::
* weakref — Weak references::
* types — Dynamic type creation and names for built-in types::
* copy — Shallow and deep copy operations::
* pprint — Data pretty printer::
* reprlib — Alternate repr() implementation: reprlib — Alternate repr implementation.
* enum — Support for enumerations::
* graphlib — Functionality to operate with graph-like structures::
datetime — Basic date and time types
* Aware and Naive Objects::
* Constants::
* Available Types::
* timedelta Objects::
* date Objects::
* datetime Objects::
* time Objects::
* tzinfo Objects::
* timezone Objects::
* strftime() and strptime() Behavior: strftime and strptime Behavior.
Available Types
* Common Properties::
* Determining if an Object is Aware or Naive::
timedelta Objects
* Examples of usage; timedelta: Examples of usage timedelta.
date Objects
* Examples of Usage; date: Examples of Usage date.
datetime Objects
* Examples of Usage; datetime: Examples of Usage datetime.
time Objects
* Examples of Usage; time: Examples of Usage time.
strftime() and strptime() Behavior
* strftime() and strptime() Format Codes: strftime and strptime Format Codes.
* Technical Detail::
zoneinfo — IANA time zone support
* Using ZoneInfo::
* Data sources::
* The ZoneInfo class::
* Functions: Functions<3>.
* Globals::
* Exceptions and warnings::
Data sources
* Configuring the data sources::
Configuring the data sources
* Compile-time configuration::
* Environment configuration::
* Runtime configuration::
The ZoneInfo class
* String representations::
* Pickle serialization::
calendar — General calendar-related functions
* Command-Line Usage::
collections — Container datatypes
* ChainMap objects::
* Counter objects::
* deque objects::
* defaultdict objects::
* namedtuple() Factory Function for Tuples with Named Fields: namedtuple Factory Function for Tuples with Named Fields.
* OrderedDict objects::
* UserDict objects::
* UserList objects::
* UserString objects::
ChainMap objects
* ChainMap Examples and Recipes::
deque objects
* deque Recipes::
defaultdict objects
* defaultdict Examples::
OrderedDict objects
* OrderedDict Examples and Recipes::
collections.abc — Abstract Base Classes for Containers
* Collections Abstract Base Classes::
* Collections Abstract Base Classes – Detailed Descriptions::
* Examples and Recipes::
heapq — Heap queue algorithm
* Basic Examples::
* Priority Queue Implementation Notes::
* Theory::
bisect — Array bisection algorithm
* Performance Notes::
* Searching Sorted Lists::
* Examples: Examples<3>.
weakref — Weak references
* Weak Reference Objects::
* Example: Example<2>.
* Finalizer Objects::
* Comparing finalizers with __del__() methods: Comparing finalizers with __del__ methods.
types — Dynamic type creation and names for built-in types
* Dynamic Type Creation::
* Standard Interpreter Types::
* Additional Utility Classes and Functions::
* Coroutine Utility Functions::
pprint — Data pretty printer
* Functions: Functions<4>.
* PrettyPrinter Objects::
* Example: Example<3>.
reprlib — Alternate repr() implementation
* Repr Objects::
* Subclassing Repr Objects::
enum — Support for enumerations
* Module Contents: Module Contents<2>.
* Data Types: Data Types<2>.
* Utilities and Decorators::
* Notes::
Data Types
* Supported __dunder__ names::
* Supported _sunder_ names::
graphlib — Functionality to operate with graph-like structures
* Exceptions: Exceptions<4>.
Numeric and Mathematical Modules
* numbers — Numeric abstract base classes::
* math — Mathematical functions::
* cmath — Mathematical functions for complex numbers::
* decimal — Decimal fixed-point and floating-point arithmetic::
* fractions — Rational numbers::
* random — Generate pseudo-random numbers::
* statistics — Mathematical statistics functions::
numbers — Numeric abstract base classes
* The numeric tower::
* Notes for type implementers::
Notes for type implementers
* Adding More Numeric ABCs::
* Implementing the arithmetic operations::
math — Mathematical functions
* Number-theoretic functions::
* Floating point arithmetic::
* Floating point manipulation functions::
* Power, exponential and logarithmic functions: Power exponential and logarithmic functions.
* Summation and product functions::
* Angular conversion::
* Trigonometric functions::
* Hyperbolic functions::
* Special functions::
* Constants: Constants<2>.
cmath — Mathematical functions for complex numbers
* Conversions to and from polar coordinates::
* Power and logarithmic functions::
* Trigonometric functions: Trigonometric functions<2>.
* Hyperbolic functions: Hyperbolic functions<2>.
* Classification functions::
* Constants: Constants<3>.
decimal — Decimal fixed-point and floating-point arithmetic
* Quick-start tutorial::
* Decimal objects::
* Context objects::
* Constants: Constants<4>.
* Rounding modes::
* Signals::
* Floating-point notes::
* Working with threads::
* Recipes::
* Decimal FAQ::
Decimal objects
* Logical operands::
Floating-point notes
* Mitigating round-off error with increased precision::
* Special values::
random — Generate pseudo-random numbers
* Bookkeeping functions::
* Functions for bytes::
* Functions for integers::
* Functions for sequences::
* Discrete distributions::
* Real-valued distributions::
* Alternative Generator::
* Notes on Reproducibility::
* Examples: Examples<4>.
* Recipes: Recipes<2>.
* Command-line usage::
* Command-line example::
statistics — Mathematical statistics functions
* Averages and measures of central location::
* Measures of spread::
* Statistics for relations between two inputs::
* Function details::
* Exceptions: Exceptions<5>.
* NormalDist objects::
* Examples and Recipes: Examples and Recipes<2>.
Examples and Recipes
* Classic probability problems::
* Monte Carlo inputs for simulations::
* Approximating binomial distributions::
* Naive bayesian classifier::
Functional Programming Modules
* itertools — Functions creating iterators for efficient looping::
* functools — Higher-order functions and operations on callable objects::
* operator — Standard operators as functions::
itertools — Functions creating iterators for efficient looping
* Itertool Functions::
* Itertools Recipes::
functools — Higher-order functions and operations on callable objects
* partial Objects::
operator — Standard operators as functions
* Mapping Operators to Functions::
* In-place Operators::
File and Directory Access
* pathlib — Object-oriented filesystem paths::
* os.path — Common pathname manipulations: os path — Common pathname manipulations.
* stat — Interpreting stat() results: stat — Interpreting stat results.
* filecmp — File and Directory Comparisons::
* tempfile — Generate temporary files and directories::
* glob — Unix style pathname pattern expansion::
* fnmatch — Unix filename pattern matching::
* linecache — Random access to text lines::
* shutil — High-level file operations::
pathlib — Object-oriented filesystem paths
* Basic use::
* Exceptions: Exceptions<6>.
* Pure paths::
* Concrete paths::
* Pattern language::
* Comparison to the glob module::
* Comparison to the os and os.path modules: Comparison to the os and os path modules.
Pure paths
* General properties::
* Operators: Operators<2>.
* Accessing individual parts::
* Methods and properties::
Concrete paths
* Parsing and generating URIs::
* Expanding and resolving paths::
* Querying file type and status::
* Reading and writing files::
* Reading directories::
* Creating files and directories::
* Renaming and deleting::
* Permissions and ownership::
Comparison to the os and os.path modules
* Corresponding tools::
filecmp — File and Directory Comparisons
* The dircmp class::
tempfile — Generate temporary files and directories
* Examples: Examples<5>.
* Deprecated functions and variables::
glob — Unix style pathname pattern expansion
* Examples: Examples<6>.
shutil — High-level file operations
* Directory and files operations::
* Archiving operations::
* Querying the size of the output terminal::
Directory and files operations
* Platform-dependent efficient copy operations::
* copytree example::
* rmtree example::
Archiving operations
* Archiving example::
* Archiving example with base_dir::
Data Persistence
* pickle — Python object serialization::
* copyreg — Register pickle support functions::
* shelve — Python object persistence::
* marshal — Internal Python object serialization::
* dbm — Interfaces to Unix “databases”::
* sqlite3 — DB-API 2.0 interface for SQLite databases: sqlite3 — DB-API 2 0 interface for SQLite databases.
pickle — Python object serialization
* Relationship to other Python modules::
* Data stream format::
* Module Interface::
* What can be pickled and unpickled?::
* Pickling Class Instances::
* Custom Reduction for Types, Functions, and Other Objects: Custom Reduction for Types Functions and Other Objects.
* Out-of-band Buffers::
* Restricting Globals::
* Performance: Performance<2>.
* Examples: Examples<7>.
Relationship to other Python modules
* Comparison with marshal::
* Comparison with json::
Pickling Class Instances
* Persistence of External Objects::
* Dispatch Tables::
* Handling Stateful Objects::
Out-of-band Buffers
* Provider API::
* Consumer API::
* Example: Example<4>.
copyreg — Register pickle support functions
* Example: Example<5>.
shelve — Python object persistence
* Restrictions::
* Example: Example<6>.
dbm — Interfaces to Unix “databases”
* dbm.sqlite3 — SQLite backend for dbm: dbm sqlite3 — SQLite backend for dbm.
* dbm.gnu — GNU database manager: dbm gnu — GNU database manager.
* dbm.ndbm — New Database Manager: dbm ndbm — New Database Manager.
* dbm.dumb — Portable DBM implementation: dbm dumb — Portable DBM implementation.
sqlite3 — DB-API 2.0 interface for SQLite databases
* Tutorial::
* Reference::
* How-to guides::
* Explanation::
Reference
* Module functions::
* Module constants::
* Connection objects::
* Cursor objects::
* Row objects::
* Blob objects::
* PrepareProtocol objects::
* Exceptions: Exceptions<7>.
* SQLite and Python types::
* Default adapters and converters (deprecated): Default adapters and converters deprecated.
* Command-line interface::
How-to guides
* How to use placeholders to bind values in SQL queries::
* How to adapt custom Python types to SQLite values::
* How to convert SQLite values to custom Python types::
* Adapter and converter recipes::
* How to use connection shortcut methods::
* How to use the connection context manager::
* How to work with SQLite URIs::
* How to create and use row factories::
* How to handle non-UTF-8 text encodings::
How to adapt custom Python types to SQLite values
* How to write adaptable objects::
* How to register adapter callables::
Explanation
* Transaction control::
Transaction control
* Transaction control via the autocommit attribute::
* Transaction control via the isolation_level attribute::
Data Compression and Archiving
* zlib — Compression compatible with gzip::
* gzip — Support for gzip files::
* bz2 — Support for bzip2 compression::
* lzma — Compression using the LZMA algorithm::
* zipfile — Work with ZIP archives::
* tarfile — Read and write tar archive files::
gzip — Support for gzip files
* Examples of usage::
* Command Line Interface::
Command Line Interface
* Command line options::
bz2 — Support for bzip2 compression
* (De)compression of files: De compression of files.
* Incremental (de)compression: Incremental de compression.
* One-shot (de)compression: One-shot de compression.
* Examples of usage: Examples of usage<2>.
lzma — Compression using the LZMA algorithm
* Reading and writing compressed files::
* Compressing and decompressing data in memory::
* Miscellaneous: Miscellaneous<2>.
* Specifying custom filter chains::
* Examples: Examples<8>.
zipfile — Work with ZIP archives
* ZipFile Objects::
* Path Objects::
* PyZipFile Objects::
* ZipInfo Objects::
* Command-Line Interface::
* Decompression pitfalls::
Command-Line Interface
* Command-line options::
Decompression pitfalls
* From file itself::
* File System limitations::
* Resources limitations::
* Interruption::
* Default behaviors of extraction::
tarfile — Read and write tar archive files
* TarFile Objects::
* TarInfo Objects::
* Extraction filters::
* Command-Line Interface: Command-Line Interface<2>.
* Examples: Examples<9>.
* Supported tar formats::
* Unicode issues::
Extraction filters
* Default named filters::
* Filter errors::
* Hints for further verification::
* Supporting older Python versions::
* Stateful extraction filter example::
Command-Line Interface
* Command-line options: Command-line options<2>.
Examples
* Reading examples::
* Writing examples::
File Formats
* csv — CSV File Reading and Writing::
* configparser — Configuration file parser::
* tomllib — Parse TOML files::
* netrc — netrc file processing::
* plistlib — Generate and parse Apple .plist files: plistlib — Generate and parse Apple plist files.
csv — CSV File Reading and Writing
* Module Contents: Module Contents<3>.
* Dialects and Formatting Parameters::
* Reader Objects::
* Writer Objects::
* Examples: Examples<10>.
configparser — Configuration file parser
* Quick Start::
* Supported Datatypes::
* Fallback Values::
* Supported INI File Structure::
* Unnamed Sections::
* Interpolation of values::
* Mapping Protocol Access::
* Customizing Parser Behaviour::
* Legacy API Examples::
* ConfigParser Objects::
* RawConfigParser Objects::
* Exceptions: Exceptions<8>.
tomllib — Parse TOML files
* Examples: Examples<11>.
* Conversion Table::
netrc — netrc file processing
* netrc Objects::
plistlib — Generate and parse Apple .plist files
* Examples: Examples<12>.
Cryptographic Services
* hashlib — Secure hashes and message digests::
* hmac — Keyed-Hashing for Message Authentication::
* secrets — Generate secure random numbers for managing secrets::
hashlib — Secure hashes and message digests
* Hash algorithms::
* Usage::
* Constructors::
* Attributes::
* Hash Objects::
* SHAKE variable length digests::
* File hashing::
* Key derivation::
* BLAKE2::
BLAKE2
* Creating hash objects::
* Constants: Constants<5>.
* Examples: Examples<13>.
* Credits::
Examples
* Simple hashing::
* Using different digest sizes::
* Keyed hashing::
* Randomized hashing::
* Personalization::
* Tree mode::
secrets — Generate secure random numbers for managing secrets
* Random numbers::
* Generating tokens::
* Other functions::
* Recipes and best practices::
Generating tokens
* How many bytes should tokens use?::
Generic Operating System Services
* os — Miscellaneous operating system interfaces::
* io — Core tools for working with streams::
* time — Time access and conversions::
* logging — Logging facility for Python::
* logging.config — Logging configuration: logging config — Logging configuration.
* logging.handlers — Logging handlers: logging handlers — Logging handlers.
* platform — Access to underlying platform’s identifying data::
* errno — Standard errno system symbols::
* ctypes — A foreign function library for Python::
os — Miscellaneous operating system interfaces
* File Names, Command Line Arguments, and Environment Variables: File Names Command Line Arguments and Environment Variables.
* Python UTF-8 Mode::
* Process Parameters::
* File Object Creation::
* File Descriptor Operations::
* Files and Directories::
* Process Management::
* Interface to the scheduler::
* Miscellaneous System Information::
* Random numbers: Random numbers<2>.
File Descriptor Operations
* Querying the size of a terminal::
* Inheritance of File Descriptors::
Files and Directories
* Timer File Descriptors::
* Linux extended attributes::
io — Core tools for working with streams
* Overview: Overview<2>.
* Text Encoding::
* High-level Module Interface::
* Class hierarchy::
* Performance: Performance<3>.
Overview
* Text I/O::
* Binary I/O::
* Raw I/O::
Text Encoding
* Opt-in EncodingWarning::
Class hierarchy
* I/O Base Classes::
* Raw File I/O::
* Buffered Streams::
* Text I/O: Text I/O<2>.
Performance
* Binary I/O: Binary I/O<2>.
* Text I/O: Text I/O<3>.
* Multi-threading: Multi-threading<3>.
* Reentrancy::
time — Time access and conversions
* Functions: Functions<5>.
* Clock ID Constants::
* Timezone Constants::
logging — Logging facility for Python
* Logger Objects::
* Logging Levels::
* Handler Objects::
* Formatter Objects::
* Filter Objects::
* LogRecord Objects::
* LogRecord attributes::
* LoggerAdapter Objects::
* Thread Safety::
* Module-Level Functions::
* Module-Level Attributes::
* Integration with the warnings module::
logging.config — Logging configuration
* Configuration functions::
* Security considerations::
* Configuration dictionary schema::
* Configuration file format::
Configuration dictionary schema
* Dictionary Schema Details::
* Incremental Configuration::
* Object connections::
* User-defined objects::
* Handler configuration order::
* Access to external objects::
* Access to internal objects::
* Import resolution and custom importers::
* Configuring QueueHandler and QueueListener::
logging.handlers — Logging handlers
* StreamHandler::
* FileHandler::
* NullHandler::
* WatchedFileHandler::
* BaseRotatingHandler::
* RotatingFileHandler::
* TimedRotatingFileHandler::
* SocketHandler::
* DatagramHandler::
* SysLogHandler::
* NTEventLogHandler::
* SMTPHandler::
* MemoryHandler::
* HTTPHandler::
* QueueHandler::
* QueueListener::
platform — Access to underlying platform’s identifying data
* Cross platform::
* Java platform::
* Windows platform::
* macOS platform::
* iOS platform::
* Unix platforms::
* Linux platforms::
* Android platform::
* Command-line usage: Command-line usage<2>.
ctypes — A foreign function library for Python
* ctypes tutorial::
* ctypes reference::
ctypes tutorial
* Loading dynamic link libraries::
* Accessing functions from loaded dlls::
* Calling functions::
* Fundamental data types::
* Calling functions, continued: Calling functions continued.
* Calling variadic functions::
* Calling functions with your own custom data types::
* Specifying the required argument types (function prototypes): Specifying the required argument types function prototypes.
* Return types::
* Passing pointers (or; passing parameters by reference): Passing pointers or passing parameters by reference.
* Structures and unions::
* Structure/union alignment and byte order::
* Bit fields in structures and unions::
* Arrays::
* Pointers::
* Type conversions::
* Incomplete Types::
* Callback functions::
* Accessing values exported from dlls::
* Surprises::
* Variable-sized data types::
ctypes reference
* Finding shared libraries::
* Loading shared libraries::
* Foreign functions::
* Function prototypes::
* Utility functions::
* Data types::
* Fundamental data types: Fundamental data types<2>.
* Structured data types::
* Arrays and pointers::
Command Line Interface Libraries
* argparse — Parser for command-line options, arguments and subcommands: argparse — Parser for command-line options arguments and subcommands.
* optparse — Parser for command line options::
* getpass — Portable password input::
* fileinput — Iterate over lines from multiple input streams::
* curses — Terminal handling for character-cell displays::
* curses.textpad — Text input widget for curses programs: curses textpad — Text input widget for curses programs.
* curses.ascii — Utilities for ASCII characters: curses ascii — Utilities for ASCII characters.
* curses.panel — A panel stack extension for curses: curses panel — A panel stack extension for curses.
argparse — Parser for command-line options, arguments and subcommands
* ArgumentParser objects::
* The add_argument() method: The add_argument method.
* The parse_args() method: The parse_args method.
* Other utilities::
* Exceptions: Exceptions<9>.
ArgumentParser objects
* prog::
* usage::
* description::
* epilog::
* parents::
* formatter_class::
* prefix_chars::
* fromfile_prefix_chars::
* argument_default::
* allow_abbrev::
* conflict_handler::
* add_help::
* exit_on_error::
The add_argument() method
* name or flags::
* action::
* nargs::
* const::
* default::
* type::
* choices::
* required::
* help::
* metavar::
* dest::
* deprecated::
* Action classes::
The parse_args() method
* Option value syntax::
* Invalid arguments::
* Arguments containing -::
* Argument abbreviations (prefix matching): Argument abbreviations prefix matching.
* Beyond sys.argv: Beyond sys argv.
* The Namespace object::
Other utilities
* Sub-commands::
* FileType objects::
* Argument groups::
* Mutual exclusion::
* Parser defaults::
* Printing help::
* Partial parsing::
* Customizing file parsing::
* Exiting methods::
* Intermixed parsing::
* Registering custom types or actions::
Exceptions
* Argparse Tutorial::
* Migrating optparse code to argparse::
Argparse Tutorial
* Concepts::
* The basics::
* Introducing Positional arguments::
* Introducing Optional arguments::
* Combining Positional and Optional arguments::
* Getting a little more advanced::
* How to translate the argparse output::
* Custom type converters::
* Conclusion::
Introducing Optional arguments
* Short options::
Getting a little more advanced
* Specifying ambiguous arguments::
* Conflicting options::
optparse — Parser for command line options
* Choosing an argument parsing library::
* Introduction: Introduction<7>.
* Background::
* Tutorial: Tutorial<2>.
* Reference Guide::
* Option Callbacks::
* Extending optparse::
* Exceptions: Exceptions<10>.
Background
* Terminology::
* What are options for?::
* What are positional arguments for?::
Tutorial
* Understanding option actions::
* The store action::
* Handling boolean (flag) options: Handling boolean flag options.
* Other actions::
* Default values::
* Generating help::
* Printing a version string::
* How optparse handles errors::
* Putting it all together::
Generating help
* Grouping Options::
Reference Guide
* Creating the parser::
* Populating the parser::
* Defining options::
* Option attributes::
* Standard option actions::
* Standard option types::
* Parsing arguments::
* Querying and manipulating your option parser::
* Conflicts between options::
* Cleanup::
* Other methods::
Option Callbacks
* Defining a callback option::
* How callbacks are called::
* Raising errors in a callback::
* Callback example 1; trivial callback: Callback example 1 trivial callback.
* Callback example 2; check option order: Callback example 2 check option order.
* Callback example 3; check option order (generalized): Callback example 3 check option order generalized.
* Callback example 4; check arbitrary condition: Callback example 4 check arbitrary condition.
* Callback example 5; fixed arguments: Callback example 5 fixed arguments.
* Callback example 6; variable arguments: Callback example 6 variable arguments.
Extending optparse
* Adding new types::
* Adding new actions::
curses — Terminal handling for character-cell displays
* Functions: Functions<6>.
* Window Objects::
* Constants: Constants<6>.
curses.textpad — Text input widget for curses programs
* Textbox objects::
curses.panel — A panel stack extension for curses
* Functions: Functions<7>.
* Panel Objects::
Concurrent Execution
* threading — Thread-based parallelism::
* multiprocessing — Process-based parallelism::
* multiprocessing.shared_memory — Shared memory for direct access across processes: multiprocessing shared_memory — Shared memory for direct access across processes.
* The concurrent package::
* concurrent.futures — Launching parallel tasks: concurrent futures — Launching parallel tasks.
* subprocess — Subprocess management::
* sched — Event scheduler::
* queue — A synchronized queue class::
* contextvars — Context Variables::
* _thread — Low-level threading API::
threading — Thread-based parallelism
* Introduction: Introduction<8>.
* GIL and performance considerations::
* Reference: Reference<2>.
* Using locks, conditions, and semaphores in the with statement: Using locks conditions and semaphores in the with statement.
Reference
* Thread-local data::
* Thread objects::
* Lock objects::
* RLock objects::
* Condition objects::
* Semaphore objects::
* Semaphore example::
* Event objects::
* Timer objects::
* Barrier objects::
multiprocessing — Process-based parallelism
* Introduction: Introduction<9>.
* Reference: Reference<3>.
* Programming guidelines::
* Examples: Examples<14>.
Introduction
* The Process class::
* Contexts and start methods::
* Exchanging objects between processes::
* Synchronization between processes::
* Sharing state between processes::
* Using a pool of workers::
Reference
* Process and exceptions::
* Pipes and Queues::
* Miscellaneous: Miscellaneous<3>.
* Connection Objects::
* Synchronization primitives::
* Shared ctypes Objects::
* Managers::
* Proxy Objects::
* Process Pools::
* Listeners and Clients::
* Authentication keys::
* Logging: Logging<2>.
* The multiprocessing.dummy module: The multiprocessing dummy module.
Shared ctypes Objects
* The multiprocessing.sharedctypes module: The multiprocessing sharedctypes module.
Managers
* Customized managers::
* Using a remote manager::
Proxy Objects
* Cleanup: Cleanup<2>.
Listeners and Clients
* Address Formats::
Programming guidelines
* All start methods::
* The spawn and forkserver start methods::
concurrent.futures — Launching parallel tasks
* Executor Objects::
* ThreadPoolExecutor::
* ProcessPoolExecutor::
* Future Objects::
* Module Functions::
* Exception classes::
ThreadPoolExecutor
* ThreadPoolExecutor Example::
ProcessPoolExecutor
* ProcessPoolExecutor Example::
subprocess — Subprocess management
* Using the subprocess Module::
* Security Considerations::
* Popen Objects::
* Windows Popen Helpers::
* Older high-level API::
* Replacing Older Functions with the subprocess Module::
* Legacy Shell Invocation Functions::
* Notes: Notes<2>.
Using the subprocess Module
* Frequently Used Arguments::
* Popen Constructor::
* Exceptions: Exceptions<11>.
Windows Popen Helpers
* Windows Constants::
Replacing Older Functions with the subprocess Module
* Replacing /bin/sh shell command substitution::
* Replacing shell pipeline::
* Replacing os.system(): Replacing os system.
* Replacing the os.spawn family: Replacing the os spawn family.
* Replacing os.popen(), os.popen2(), os.popen3(): Replacing os popen os popen2 os popen3.
* Replacing functions from the popen2 module::
Notes
* Timeout Behavior::
* Converting an argument sequence to a string on Windows::
* Disabling use of vfork() or posix_spawn(): Disabling use of vfork or posix_spawn.
sched — Event scheduler
* Scheduler Objects::
queue — A synchronized queue class
* Queue Objects::
* SimpleQueue Objects::
Queue Objects
* Waiting for task completion::
* Terminating queues::
contextvars — Context Variables
* Context Variables::
* Manual Context Management::
* asyncio support::
Networking and Interprocess Communication
* asyncio — Asynchronous I/O::
* socket — Low-level networking interface::
* ssl — TLS/SSL wrapper for socket objects::
* select — Waiting for I/O completion::
* selectors — High-level I/O multiplexing::
* signal — Set handlers for asynchronous events::
* mmap — Memory-mapped file support::
asyncio — Asynchronous I/O
* Runners::
* Coroutines and Tasks::
* Streams::
* Synchronization Primitives::
* Subprocesses::
* Queues::
* Exceptions: Exceptions<13>.
* Event Loop::
* Futures::
* Transports and Protocols::
* Policies::
* Platform Support::
* Extending::
* High-level API Index::
* Low-level API Index::
* Developing with asyncio::
Runners
* Running an asyncio Program::
* Runner context manager::
* Handling Keyboard Interruption::
Coroutines and Tasks
* Coroutines: Coroutines<3>.
* Awaitables::
* Creating Tasks::
* Task Cancellation::
* Task Groups::
* Sleeping::
* Running Tasks Concurrently::
* Eager Task Factory::
* Shielding From Cancellation::
* Timeouts::
* Waiting Primitives::
* Running in Threads::
* Scheduling From Other Threads::
* Introspection::
* Task Object::
Task Groups
* Terminating a Task Group::
Streams
* StreamReader::
* StreamWriter::
* Examples: Examples<15>.
Examples
* TCP echo client using streams::
* TCP echo server using streams::
* Get HTTP headers::
* Register an open socket to wait for data using streams::
Synchronization Primitives
* Lock::
* Event::
* Condition::
* Semaphore::
* BoundedSemaphore::
* Barrier::
Subprocesses
* Creating Subprocesses::
* Constants: Constants<7>.
* Interacting with Subprocesses::
Interacting with Subprocesses
* Subprocess and Threads::
* Examples: Examples<16>.
Queues
* Queue::
* Priority Queue::
* LIFO Queue::
* Exceptions: Exceptions<12>.
* Examples: Examples<17>.
Event Loop
* Event Loop Methods::
* Callback Handles::
* Server Objects::
* Event Loop Implementations::
* Examples: Examples<18>.
Event Loop Methods
* Running and stopping the loop::
* Scheduling callbacks::
* Scheduling delayed callbacks::
* Creating Futures and Tasks::
* Opening network connections::
* Creating network servers::
* Transferring files::
* TLS Upgrade::
* Watching file descriptors::
* Working with socket objects directly::
* DNS::
* Working with pipes::
* Unix signals::
* Executing code in thread or process pools::
* Error Handling API::
* Enabling debug mode::
* Running Subprocesses::
Examples
* Hello World with call_soon(): Hello World with call_soon.
* Display the current date with call_later(): Display the current date with call_later.
* Watch a file descriptor for read events::
* Set signal handlers for SIGINT and SIGTERM::
Futures
* Future Functions::
* Future Object::
Transports and Protocols
* Transports::
* Protocols::
* Examples: Examples<19>.
Transports
* Transports Hierarchy::
* Base Transport::
* Read-only Transports::
* Write-only Transports::
* Datagram Transports::
* Subprocess Transports::
Protocols
* Base Protocols::
* Base Protocol::
* Streaming Protocols::
* Buffered Streaming Protocols::
* Datagram Protocols::
* Subprocess Protocols::
Examples
* TCP Echo Server::
* TCP Echo Client::
* UDP Echo Server::
* UDP Echo Client::
* Connecting Existing Sockets::
* loop.subprocess_exec() and SubprocessProtocol: loop subprocess_exec and SubprocessProtocol.
Policies
* Getting and Setting the Policy::
* Policy Objects::
* Process Watchers::
* Custom Policies::
Platform Support
* All Platforms::
* Windows: Windows<79>.
* macOS: macOS<50>.
Windows
* Subprocess Support on Windows::
Extending
* Writing a Custom Event Loop::
* Future and Task private constructors::
* Task lifetime support::
High-level API Index
* Tasks::
* Queues: Queues<2>.
* Subprocesses: Subprocesses<2>.
* Streams: Streams<2>.
* Synchronization::
* Exceptions: Exceptions<14>.
Low-level API Index
* Obtaining the Event Loop::
* Event Loop Methods: Event Loop Methods<2>.
* Transports: Transports<2>.
* Protocols: Protocols<2>.
* Event Loop Policies::
Developing with asyncio
* Debug Mode::
* Concurrency and Multithreading::
* Running Blocking Code::
* Logging: Logging<3>.
* Detect never-awaited coroutines::
* Detect never-retrieved exceptions::
socket — Low-level networking interface
* Socket families::
* Module contents::
* Socket Objects::
* Notes on socket timeouts::
* Example: Example<7>.
Module contents
* Exceptions: Exceptions<15>.
* Constants: Constants<8>.
* Functions: Functions<8>.
Functions
* Creating sockets::
* Other functions: Other functions<2>.
Notes on socket timeouts
* Timeouts and the connect method::
* Timeouts and the accept method::
ssl — TLS/SSL wrapper for socket objects
* Functions, Constants, and Exceptions: Functions Constants and Exceptions.
* SSL Sockets::
* SSL Contexts::
* Certificates::
* Examples: Examples<20>.
* Notes on non-blocking sockets::
* Memory BIO Support: Memory BIO Support<2>.
* SSL session::
* Security considerations: Security considerations<2>.
* TLS 1.3: TLS 1 3.
Functions, Constants, and Exceptions
* Socket creation::
* Context creation::
* Exceptions: Exceptions<16>.
* Random generation::
* Certificate handling::
* Constants: Constants<9>.
Certificates
* Certificate chains::
* CA certificates::
* Combined key and certificate::
* Self-signed certificates::
Examples
* Testing for SSL support::
* Client-side operation::
* Server-side operation::
Security considerations
* Best defaults::
* Manual settings::
* Multi-processing::
Manual settings
* Verifying certificates::
* Protocol versions::
* Cipher selection::
select — Waiting for I/O completion
* /dev/poll Polling Objects::
* Edge and Level Trigger Polling (epoll) Objects: Edge and Level Trigger Polling epoll Objects.
* Polling Objects::
* Kqueue Objects::
* Kevent Objects::
selectors — High-level I/O multiplexing
* Introduction: Introduction<10>.
* Classes: Classes<4>.
* Examples: Examples<21>.
signal — Set handlers for asynchronous events
* General rules::
* Module contents: Module contents<2>.
* Examples: Examples<22>.
* Note on SIGPIPE::
* Note on Signal Handlers and Exceptions::
General rules
* Execution of Python signal handlers::
* Signals and threads::
mmap — Memory-mapped file support
* MADV_* Constants::
* MAP_* Constants::
Internet Data Handling
* email — An email and MIME handling package::
* json — JSON encoder and decoder::
* mailbox — Manipulate mailboxes in various formats::
* mimetypes — Map filenames to MIME types::
* base64 — Base16, Base32, Base64, Base85 Data Encodings: base64 — Base16 Base32 Base64 Base85 Data Encodings.
* binascii — Convert between binary and ASCII::
* quopri — Encode and decode MIME quoted-printable data::
email — An email and MIME handling package
* email.message; Representing an email message: email message Representing an email message.
* email.parser; Parsing email messages: email parser Parsing email messages.
* email.generator; Generating MIME documents: email generator Generating MIME documents.
* email.policy; Policy Objects: email policy Policy Objects.
* email.errors; Exception and Defect classes: email errors Exception and Defect classes.
* email.headerregistry; Custom Header Objects: email headerregistry Custom Header Objects.
* email.contentmanager; Managing MIME Content: email contentmanager Managing MIME Content.
* email; Examples: email Examples.
* email.message.Message; Representing an email message using the compat32 API: email message Message Representing an email message using the compat32 API.
* email.mime; Creating email and MIME objects from scratch: email mime Creating email and MIME objects from scratch.
* email.header; Internationalized headers: email header Internationalized headers.
* email.charset; Representing character sets: email charset Representing character sets.
* email.encoders; Encoders: email encoders Encoders.
* email.utils; Miscellaneous utilities: email utils Miscellaneous utilities.
* email.iterators; Iterators: email iterators Iterators.
email.parser: Parsing email messages
* FeedParser API::
* Parser API::
* Additional notes::
email.contentmanager: Managing MIME Content
* Content Manager Instances::
json — JSON encoder and decoder
* Basic Usage::
* Encoders and Decoders::
* Exceptions: Exceptions<17>.
* Standard Compliance and Interoperability::
* Command Line Interface: Command Line Interface<2>.
Standard Compliance and Interoperability
* Character Encodings::
* Infinite and NaN Number Values::
* Repeated Names Within an Object::
* Top-level Non-Object, Non-Array Values: Top-level Non-Object Non-Array Values.
* Implementation Limitations::
Command Line Interface
* Command line options: Command line options<2>.
mailbox — Manipulate mailboxes in various formats
* Mailbox objects::
* Message objects::
* Exceptions: Exceptions<18>.
* Examples: Examples<23>.
Mailbox objects
* Maildir objects::
* mbox objects::
* MH objects::
* Babyl objects::
* MMDF objects::
Message objects
* MaildirMessage objects::
* mboxMessage objects::
* MHMessage objects::
* BabylMessage objects::
* MMDFMessage objects::
mimetypes — Map filenames to MIME types
* MimeTypes Objects::
base64 — Base16, Base32, Base64, Base85 Data Encodings
* RFC 4648 Encodings::
* Base85 Encodings::
* Legacy Interface::
* Security Considerations: Security Considerations<2>.
Structured Markup Processing Tools
* html — HyperText Markup Language support::
* html.parser — Simple HTML and XHTML parser: html parser — Simple HTML and XHTML parser.
* html.entities — Definitions of HTML general entities: html entities — Definitions of HTML general entities.
* XML Processing Modules::
* xml.etree.ElementTree — The ElementTree XML API: xml etree ElementTree — The ElementTree XML API.
* xml.dom — The Document Object Model API: xml dom — The Document Object Model API.
* xml.dom.minidom — Minimal DOM implementation: xml dom minidom — Minimal DOM implementation.
* xml.dom.pulldom — Support for building partial DOM trees: xml dom pulldom — Support for building partial DOM trees.
* xml.sax — Support for SAX2 parsers: xml sax — Support for SAX2 parsers.
* xml.sax.handler — Base classes for SAX handlers: xml sax handler — Base classes for SAX handlers.
* xml.sax.saxutils — SAX Utilities: xml sax saxutils — SAX Utilities.
* xml.sax.xmlreader — Interface for XML parsers: xml sax xmlreader — Interface for XML parsers.
* xml.parsers.expat — Fast XML parsing using Expat: xml parsers expat — Fast XML parsing using Expat.
html.parser — Simple HTML and XHTML parser
* Example HTML Parser Application::
* HTMLParser Methods::
* Examples: Examples<24>.
XML Processing Modules
* XML security::
xml.etree.ElementTree — The ElementTree XML API
* Tutorial: Tutorial<3>.
* XPath support::
* Reference: Reference<4>.
* XInclude support::
* Reference: Reference<5>.
Tutorial
* XML tree and elements::
* Parsing XML::
* Pull API for non-blocking parsing::
* Finding interesting elements::
* Modifying an XML File::
* Building XML documents::
* Parsing XML with Namespaces::
XPath support
* Example: Example<8>.
* Supported XPath syntax::
Reference
* Functions: Functions<9>.
XInclude support
* Example: Example<9>.
Reference
* Functions: Functions<10>.
* Element Objects::
* ElementTree Objects::
* QName Objects::
* TreeBuilder Objects::
* XMLParser Objects::
* XMLPullParser Objects::
* Exceptions: Exceptions<19>.
xml.dom — The Document Object Model API
* Module Contents: Module Contents<4>.
* Objects in the DOM::
* Conformance::
Objects in the DOM
* DOMImplementation Objects::
* Node Objects::
* NodeList Objects::
* DocumentType Objects::
* Document Objects::
* Element Objects: Element Objects<2>.
* Attr Objects::
* NamedNodeMap Objects::
* Comment Objects::
* Text and CDATASection Objects::
* ProcessingInstruction Objects::
* Exceptions: Exceptions<20>.
Conformance
* Type Mapping::
* Accessor Methods::
xml.dom.minidom — Minimal DOM implementation
* DOM Objects::
* DOM Example::
* minidom and the DOM standard::
xml.dom.pulldom — Support for building partial DOM trees
* DOMEventStream Objects::
xml.sax — Support for SAX2 parsers
* SAXException Objects::
xml.sax.handler — Base classes for SAX handlers
* ContentHandler Objects::
* DTDHandler Objects::
* EntityResolver Objects::
* ErrorHandler Objects::
* LexicalHandler Objects::
xml.sax.xmlreader — Interface for XML parsers
* XMLReader Objects::
* IncrementalParser Objects::
* Locator Objects::
* InputSource Objects::
* The Attributes Interface::
* The AttributesNS Interface::
xml.parsers.expat — Fast XML parsing using Expat
* XMLParser Objects: XMLParser Objects<2>.
* ExpatError Exceptions::
* Example: Example<10>.
* Content Model Descriptions::
* Expat error constants::
Internet Protocols and Support
* webbrowser — Convenient web-browser controller::
* wsgiref — WSGI Utilities and Reference Implementation::
* urllib — URL handling modules::
* urllib.request — Extensible library for opening URLs: urllib request — Extensible library for opening URLs.
* urllib.response — Response classes used by urllib: urllib response — Response classes used by urllib.
* urllib.parse — Parse URLs into components: urllib parse — Parse URLs into components.
* urllib.error — Exception classes raised by urllib.request: urllib error — Exception classes raised by urllib request.
* urllib.robotparser — Parser for robots.txt: urllib robotparser — Parser for robots txt.
* http — HTTP modules::
* http.client — HTTP protocol client: http client — HTTP protocol client.
* ftplib — FTP protocol client::
* poplib — POP3 protocol client::
* imaplib — IMAP4 protocol client::
* smtplib — SMTP protocol client::
* uuid — UUID objects according to RFC 4122::
* socketserver — A framework for network servers::
* http.server — HTTP servers: http server — HTTP servers.
* http.cookies — HTTP state management: http cookies — HTTP state management.
* http.cookiejar — Cookie handling for HTTP clients: http cookiejar — Cookie handling for HTTP clients.
* xmlrpc — XMLRPC server and client modules::
* xmlrpc.client — XML-RPC client access: xmlrpc client — XML-RPC client access.
* xmlrpc.server — Basic XML-RPC servers: xmlrpc server — Basic XML-RPC servers.
* ipaddress — IPv4/IPv6 manipulation library::
webbrowser — Convenient web-browser controller
* Browser Controller Objects::
wsgiref — WSGI Utilities and Reference Implementation
* wsgiref.util – WSGI environment utilities: wsgiref util – WSGI environment utilities.
* wsgiref.headers – WSGI response header tools: wsgiref headers – WSGI response header tools.
* wsgiref.simple_server – a simple WSGI HTTP server: wsgiref simple_server – a simple WSGI HTTP server.
* wsgiref.validate — WSGI conformance checker: wsgiref validate — WSGI conformance checker.
* wsgiref.handlers – server/gateway base classes: wsgiref handlers – server/gateway base classes.
* wsgiref.types – WSGI types for static type checking: wsgiref types – WSGI types for static type checking.
* Examples: Examples<25>.
urllib.request — Extensible library for opening URLs
* Request Objects::
* OpenerDirector Objects::
* BaseHandler Objects::
* HTTPRedirectHandler Objects::
* HTTPCookieProcessor Objects::
* ProxyHandler Objects::
* HTTPPasswordMgr Objects::
* HTTPPasswordMgrWithPriorAuth Objects::
* AbstractBasicAuthHandler Objects::
* HTTPBasicAuthHandler Objects::
* ProxyBasicAuthHandler Objects::
* AbstractDigestAuthHandler Objects::
* HTTPDigestAuthHandler Objects::
* ProxyDigestAuthHandler Objects::
* HTTPHandler Objects::
* HTTPSHandler Objects::
* FileHandler Objects::
* DataHandler Objects::
* FTPHandler Objects::
* CacheFTPHandler Objects::
* UnknownHandler Objects::
* HTTPErrorProcessor Objects::
* Examples: Examples<26>.
* Legacy interface::
* urllib.request Restrictions: urllib request Restrictions.
urllib.parse — Parse URLs into components
* URL Parsing::
* URL parsing security::
* Parsing ASCII Encoded Bytes::
* Structured Parse Results::
* URL Quoting::
http — HTTP modules
* HTTP status codes::
* HTTP status category::
* HTTP methods::
http.client — HTTP protocol client
* HTTPConnection Objects::
* HTTPResponse Objects::
* Examples: Examples<27>.
* HTTPMessage Objects::
ftplib — FTP protocol client
* Reference: Reference<6>.
Reference
* FTP objects::
* FTP_TLS objects::
* Module variables::
poplib — POP3 protocol client
* POP3 Objects::
* POP3 Example::
imaplib — IMAP4 protocol client
* IMAP4 Objects::
* IMAP4 Example::
smtplib — SMTP protocol client
* SMTP Objects::
* SMTP Example::
uuid — UUID objects according to RFC 4122
* Command-Line Usage: Command-Line Usage<2>.
* Example: Example<11>.
* Command-Line Example::
socketserver — A framework for network servers
* Server Creation Notes::
* Server Objects: Server Objects<2>.
* Request Handler Objects::
* Examples: Examples<28>.
Examples
* socketserver.TCPServer Example: socketserver TCPServer Example.
* socketserver.UDPServer Example: socketserver UDPServer Example.
* Asynchronous Mixins::
http.server — HTTP servers
* Command-line interface: Command-line interface<2>.
* Security considerations: Security considerations<3>.
http.cookies — HTTP state management
* Cookie Objects::
* Morsel Objects::
* Example: Example<12>.
http.cookiejar — Cookie handling for HTTP clients
* CookieJar and FileCookieJar Objects::
* FileCookieJar subclasses and co-operation with web browsers::
* CookiePolicy Objects::
* DefaultCookiePolicy Objects::
* Cookie Objects: Cookie Objects<2>.
* Examples: Examples<29>.
xmlrpc.client — XML-RPC client access
* ServerProxy Objects::
* DateTime Objects::
* Binary Objects::
* Fault Objects::
* ProtocolError Objects::
* MultiCall Objects::
* Convenience Functions::
* Example of Client Usage::
* Example of Client and Server Usage::
xmlrpc.server — Basic XML-RPC servers
* SimpleXMLRPCServer Objects::
* CGIXMLRPCRequestHandler::
* Documenting XMLRPC server::
* DocXMLRPCServer Objects::
* DocCGIXMLRPCRequestHandler::
SimpleXMLRPCServer Objects
* SimpleXMLRPCServer Example::
ipaddress — IPv4/IPv6 manipulation library
* Convenience factory functions::
* IP Addresses::
* IP Network definitions::
* Interface objects::
* Other Module Level Functions::
* Custom Exceptions::
IP Addresses
* Address objects::
* Conversion to Strings and Integers::
* Operators: Operators<3>.
Operators
* Comparison operators::
* Arithmetic operators::
IP Network definitions
* Prefix, net mask and host mask: Prefix net mask and host mask.
* Network objects::
* Operators: Operators<4>.
Operators
* Logical operators::
* Iteration::
* Networks as containers of addresses::
Interface objects
* Operators: Operators<5>.
Operators
* Logical operators: Logical operators<2>.
Multimedia Services
* wave — Read and write WAV files::
* colorsys — Conversions between color systems::
wave — Read and write WAV files
* Wave_read Objects::
* Wave_write Objects::
Internationalization
* gettext — Multilingual internationalization services::
* locale — Internationalization services::
gettext — Multilingual internationalization services
* GNU gettext API::
* Class-based API::
* Internationalizing your programs and modules::
* Acknowledgements: Acknowledgements<9>.
Class-based API
* The NullTranslations class::
* The GNUTranslations class::
* Solaris message catalog support::
* The Catalog constructor::
Internationalizing your programs and modules
* Localizing your module::
* Localizing your application::
* Changing languages on the fly::
* Deferred translations::
locale — Internationalization services
* Background, details, hints, tips and caveats: Background details hints tips and caveats.
* Locale names::
* For extension writers and programs that embed Python::
* Access to message catalogs::
Program Frameworks
* turtle — Turtle graphics::
* cmd — Support for line-oriented command interpreters::
* shlex — Simple lexical analysis::
turtle — Turtle graphics
* Introduction: Introduction<11>.
* Get started::
* Tutorial: Tutorial<4>.
* How to…::
* Turtle graphics reference::
* Methods of RawTurtle/Turtle and corresponding functions::
* Methods of TurtleScreen/Screen and corresponding functions::
* Public classes::
* Explanation: Explanation<2>.
* Help and configuration::
* turtledemo — Demo scripts::
* Changes since Python 2.6: Changes since Python 2 6.
* Changes since Python 3.0: Changes since Python 3 0.
Tutorial
* Starting a turtle environment::
* Basic drawing::
* Making algorithmic patterns::
Basic drawing
* Pen control::
* The turtle’s position::
How to…
* Get started as quickly as possible::
* Use the turtle module namespace::
* Use turtle graphics in a script::
* Use object-oriented turtle graphics::
Turtle graphics reference
* Turtle methods::
* Methods of TurtleScreen/Screen::
Methods of RawTurtle/Turtle and corresponding functions
* Turtle motion::
* Tell Turtle’s state::
* Settings for measurement::
* Pen control: Pen control<2>.
* Turtle state::
* Using events::
* Special Turtle methods::
* Compound shapes::
Pen control
* Drawing state::
* Color control::
* Filling::
* More drawing control::
Turtle state
* Visibility::
* Appearance::
Methods of TurtleScreen/Screen and corresponding functions
* Window control::
* Animation control::
* Using screen events::
* Input methods::
* Settings and special methods::
* Methods specific to Screen, not inherited from TurtleScreen: Methods specific to Screen not inherited from TurtleScreen.
Help and configuration
* How to use help::
* Translation of docstrings into different languages::
* How to configure Screen and Turtles::
cmd — Support for line-oriented command interpreters
* Cmd Objects::
* Cmd Example::
shlex — Simple lexical analysis
* shlex Objects::
* Parsing Rules::
* Improved Compatibility with Shells::
Graphical User Interfaces with Tk
* tkinter — Python interface to Tcl/Tk::
* tkinter.colorchooser — Color choosing dialog: tkinter colorchooser — Color choosing dialog.
* tkinter.font — Tkinter font wrapper: tkinter font — Tkinter font wrapper.
* Tkinter Dialogs::
* tkinter.messagebox — Tkinter message prompts: tkinter messagebox — Tkinter message prompts.
* tkinter.scrolledtext — Scrolled Text Widget: tkinter scrolledtext — Scrolled Text Widget.
* tkinter.dnd — Drag and drop support: tkinter dnd — Drag and drop support.
* tkinter.ttk — Tk themed widgets: tkinter ttk — Tk themed widgets.
* IDLE — Python editor and shell: IDLE — Python editor and shell<2>.
tkinter — Python interface to Tcl/Tk
* Architecture::
* Tkinter Modules::
* Tkinter Life Preserver::
* Threading model::
* Handy Reference::
* File Handlers::
Tkinter Life Preserver
* A Hello World Program::
* Important Tk Concepts::
* Understanding How Tkinter Wraps Tcl/Tk::
* How do I…? What option does…?::
* Navigating the Tcl/Tk Reference Manual::
Handy Reference
* Setting Options::
* The Packer::
* Packer Options::
* Coupling Widget Variables::
* The Window Manager::
* Tk Option Data Types::
* Bindings and Events::
* The index Parameter::
* Images::
Tkinter Dialogs
* tkinter.simpledialog — Standard Tkinter input dialogs: tkinter simpledialog — Standard Tkinter input dialogs.
* tkinter.filedialog — File selection dialogs: tkinter filedialog — File selection dialogs.
* tkinter.commondialog — Dialog window templates: tkinter commondialog — Dialog window templates.
tkinter.filedialog — File selection dialogs
* Native Load/Save Dialogs::
tkinter.ttk — Tk themed widgets
* Using Ttk::
* Ttk Widgets::
* Widget::
* Combobox::
* Spinbox::
* Notebook::
* Progressbar::
* Separator::
* Sizegrip::
* Treeview::
* Ttk Styling::
Widget
* Standard Options::
* Scrollable Widget Options::
* Label Options::
* Compatibility Options::
* Widget States::
* ttk.Widget: ttk Widget.
Combobox
* Options::
* Virtual events::
* ttk.Combobox: ttk Combobox.
Spinbox
* Options: Options<2>.
* Virtual events: Virtual events<2>.
* ttk.Spinbox: ttk Spinbox.
Notebook
* Options: Options<3>.
* Tab Options::
* Tab Identifiers::
* Virtual Events::
* ttk.Notebook: ttk Notebook.
Progressbar
* Options: Options<4>.
* ttk.Progressbar: ttk Progressbar.
Separator
* Options: Options<5>.
Sizegrip
* Platform-specific notes::
* Bugs::
Treeview
* Options: Options<6>.
* Item Options::
* Tag Options::
* Column Identifiers::
* Virtual Events: Virtual Events<2>.
* ttk.Treeview: ttk Treeview.
Ttk Styling
* Layouts::
IDLE — Python editor and shell
* Menus::
* Editing and Navigation::
* Startup and Code Execution::
* Help and Preferences::
* idlelib — implementation of IDLE application::
Menus
* File menu (Shell and Editor): File menu Shell and Editor.
* Edit menu (Shell and Editor): Edit menu Shell and Editor.
* Format menu (Editor window only): Format menu Editor window only.
* Run menu (Editor window only): Run menu Editor window only.
* Shell menu (Shell window only): Shell menu Shell window only.
* Debug menu (Shell window only): Debug menu Shell window only.
* Options menu (Shell and Editor): Options menu Shell and Editor.
* Window menu (Shell and Editor): Window menu Shell and Editor.
* Help menu (Shell and Editor): Help menu Shell and Editor.
* Context menus::
Editing and Navigation
* Editor windows::
* Key bindings::
* Automatic indentation::
* Search and Replace::
* Completions::
* Calltips::
* Code Context::
* Shell window::
* Text colors::
Startup and Code Execution
* Command line usage::
* Startup failure::
* Running user code::
* User output in Shell::
* Developing tkinter applications::
* Running without a subprocess::
Help and Preferences
* Help sources::
* Setting preferences::
* IDLE on macOS::
* Extensions::
Development Tools
* typing — Support for type hints::
* pydoc — Documentation generator and online help system::
* Python Development Mode::
* doctest — Test interactive Python examples::
* unittest — Unit testing framework::
* unittest.mock — mock object library: unittest mock — mock object library.
* unittest.mock — getting started: unittest mock — getting started.
* test — Regression tests package for Python::
* test.support — Utilities for the Python test suite: test support — Utilities for the Python test suite.
* test.support.socket_helper — Utilities for socket tests: test support socket_helper — Utilities for socket tests.
* test.support.script_helper — Utilities for the Python execution tests: test support script_helper — Utilities for the Python execution tests.
* test.support.bytecode_helper — Support tools for testing correct bytecode generation: test support bytecode_helper — Support tools for testing correct bytecode generation.
* test.support.threading_helper — Utilities for threading tests: test support threading_helper — Utilities for threading tests.
* test.support.os_helper — Utilities for os tests: test support os_helper — Utilities for os tests.
* test.support.import_helper — Utilities for import tests: test support import_helper — Utilities for import tests.
* test.support.warnings_helper — Utilities for warnings tests: test support warnings_helper — Utilities for warnings tests.
typing — Support for type hints
* Specification for the Python Type System::
* Type aliases::
* NewType::
* Annotating callable objects::
* Generics::
* Annotating tuples::
* The type of class objects::
* Annotating generators and coroutines::
* User-defined generic types::
* The Any type::
* Nominal vs structural subtyping::
* Module contents: Module contents<3>.
* Deprecation Timeline of Major Features::
Module contents
* Special typing primitives::
* Protocols: Protocols<3>.
* ABCs for working with IO::
* Functions and decorators::
* Introspection helpers::
* Constant::
* Deprecated aliases::
Special typing primitives
* Special types::
* Special forms::
* Building generic types and type aliases::
* Other special directives::
Deprecated aliases
* Aliases to built-in types::
* Aliases to types in collections::
* Aliases to other concrete types::
* Aliases to container ABCs in collections.abc: Aliases to container ABCs in collections abc.
* Aliases to asynchronous ABCs in collections.abc: Aliases to asynchronous ABCs in collections abc.
* Aliases to other ABCs in collections.abc: Aliases to other ABCs in collections abc.
* Aliases to contextlib ABCs::
Python Development Mode
* Effects of the Python Development Mode::
* ResourceWarning Example::
* Bad file descriptor error example::
doctest — Test interactive Python examples
* Simple Usage; Checking Examples in Docstrings: Simple Usage Checking Examples in Docstrings.
* Simple Usage; Checking Examples in a Text File: Simple Usage Checking Examples in a Text File.
* Command-line Usage::
* How It Works::
* Basic API::
* Unittest API::
* Advanced API::
* Debugging::
* Soapbox::
How It Works
* Which Docstrings Are Examined?::
* How are Docstring Examples Recognized?::
* What’s the Execution Context?::
* What About Exceptions?::
* Option Flags::
* Directives::
* Warnings: Warnings<2>.
Advanced API
* DocTest Objects::
* Example Objects::
* DocTestFinder objects::
* DocTestParser objects::
* TestResults objects::
* DocTestRunner objects::
* OutputChecker objects::
unittest — Unit testing framework
* Basic example::
* Command-Line Interface: Command-Line Interface<3>.
* Test Discovery::
* Organizing test code::
* Re-using old test code::
* Skipping tests and expected failures::
* Distinguishing test iterations using subtests::
* Classes and functions::
* Class and Module Fixtures::
* Signal Handling::
Command-Line Interface
* Command-line options: Command-line options<3>.
Classes and functions
* Test cases::
* Grouping tests::
* Loading and running tests::
Loading and running tests
* load_tests Protocol::
Class and Module Fixtures
* setUpClass and tearDownClass::
* setUpModule and tearDownModule::
unittest.mock — mock object library
* Quick Guide::
* The Mock Class::
* The patchers::
* MagicMock and magic method support::
* Helpers::
* Order of precedence of side_effect, return_value and wraps: Order of precedence of side_effect return_value and wraps.
The Mock Class
* Calling::
* Deleting Attributes::
* Mock names and the name attribute::
* Attaching Mocks as Attributes::
The patchers
* patch::
* patch.object: patch object.
* patch.dict: patch dict.
* patch.multiple: patch multiple.
* patch methods; start and stop: patch methods start and stop.
* patch builtins::
* TEST_PREFIX::
* Nesting Patch Decorators::
* Where to patch::
* Patching Descriptors and Proxy Objects::
MagicMock and magic method support
* Mocking Magic Methods::
* Magic Mock::
Helpers
* sentinel::
* DEFAULT::
* call::
* create_autospec::
* ANY::
* FILTER_DIR::
* mock_open::
* Autospeccing::
* Sealing mocks::
unittest.mock — getting started
* Using Mock::
* Patch Decorators::
* Further Examples::
Using Mock
* Mock Patching Methods::
* Mock for Method Calls on an Object::
* Mocking Classes::
* Naming your mocks::
* Tracking all Calls::
* Setting Return Values and Attributes::
* Raising exceptions with mocks::
* Side effect functions and iterables::
* Mocking asynchronous iterators::
* Mocking asynchronous context manager::
* Creating a Mock from an Existing Object::
* Using side_effect to return per file content::
Further Examples
* Mocking chained calls::
* Partial mocking::
* Mocking a Generator Method::
* Applying the same patch to every test method::
* Mocking Unbound Methods::
* Checking multiple calls with mock::
* Coping with mutable arguments::
* Nesting Patches::
* Mocking a dictionary with MagicMock::
* Mock subclasses and their attributes::
* Mocking imports with patch.dict: Mocking imports with patch dict.
* Tracking order of calls and less verbose call assertions::
* More complex argument matching::
test — Regression tests package for Python
* Writing Unit Tests for the test package::
* Running tests using the command-line interface::
Debugging and Profiling
* Audit events table::
* bdb — Debugger framework::
* faulthandler — Dump the Python traceback::
* pdb — The Python Debugger::
* The Python Profilers::
* timeit — Measure execution time of small code snippets::
* trace — Trace or track Python statement execution::
* tracemalloc — Trace memory allocations::
faulthandler — Dump the Python traceback
* Dumping the traceback::
* Fault handler state::
* Dumping the tracebacks after a timeout::
* Dumping the traceback on a user signal::
* Issue with file descriptors::
* Example: Example<13>.
pdb — The Python Debugger
* Debugger Commands::
The Python Profilers
* Introduction to the profilers::
* Instant User’s Manual::
* profile and cProfile Module Reference::
* The Stats Class::
* What Is Deterministic Profiling?::
* Limitations::
* Calibration::
* Using a custom timer::
timeit — Measure execution time of small code snippets
* Basic Examples: Basic Examples<2>.
* Python Interface::
* Command-Line Interface: Command-Line Interface<4>.
* Examples: Examples<30>.
trace — Trace or track Python statement execution
* Command-Line Usage: Command-Line Usage<3>.
* Programmatic Interface::
Command-Line Usage
* Main options::
* Modifiers::
* Filters::
tracemalloc — Trace memory allocations
* Examples: Examples<31>.
* API::
Examples
* Display the top 10::
* Compute differences::
* Get the traceback of a memory block::
* Pretty top::
Pretty top
* Record the current and peak size of all traced memory blocks::
API
* Functions: Functions<11>.
* DomainFilter::
* Filter::
* Frame::
* Snapshot::
* Statistic::
* StatisticDiff::
* Trace::
* Traceback::
Software Packaging and Distribution
* ensurepip — Bootstrapping the pip installer::
* venv — Creation of virtual environments::
* zipapp — Manage executable Python zip archives::
ensurepip — Bootstrapping the pip installer
* Command line interface::
* Module API::
venv — Creation of virtual environments
* Creating virtual environments::
* How venvs work::
* API: API<2>.
* An example of extending EnvBuilder::
zipapp — Manage executable Python zip archives
* Basic Example::
* Command-Line Interface: Command-Line Interface<5>.
* Python API::
* Examples: Examples<32>.
* Specifying the Interpreter::
* Creating Standalone Applications with zipapp::
* The Python Zip Application Archive Format::
Creating Standalone Applications with zipapp
* Caveats::
Python Runtime Services
* sys — System-specific parameters and functions::
* sys.monitoring — Execution event monitoring: sys monitoring — Execution event monitoring.
* sysconfig — Provide access to Python’s configuration information::
* builtins — Built-in objects::
* __main__ — Top-level code environment::
* warnings — Warning control::
* dataclasses — Data Classes::
* contextlib — Utilities for with-statement contexts::
* abc — Abstract Base Classes::
* atexit — Exit handlers::
* traceback — Print or retrieve a stack traceback::
* __future__ — Future statement definitions::
* gc — Garbage Collector interface::
* inspect — Inspect live objects::
* site — Site-specific configuration hook::
sys.monitoring — Execution event monitoring
* Tool identifiers::
* Events::
* Turning events on and off::
* Registering callback functions::
Tool identifiers
* Registering and using tools::
Events
* Local events::
* Ancillary events::
* Other events::
* The STOP_ITERATION event::
Turning events on and off
* Setting events globally::
* Per code object events::
* Disabling events::
Registering callback functions
* Callback function arguments::
sysconfig — Provide access to Python’s configuration information
* Configuration variables::
* Installation paths::
* User scheme::
* Home scheme::
* Prefix scheme::
* Installation path functions::
* Other functions: Other functions<3>.
* Command-line usage: Command-line usage<3>.
User scheme
* posix_user::
* nt_user::
* osx_framework_user::
Home scheme
* posix_home::
Prefix scheme
* posix_prefix::
* nt::
__main__ — Top-level code environment
* __name__ == '__main__'::
* __main__.py in Python Packages: __main__ py in Python Packages.
* import __main__::
__name__ == '__main__'
* What is the “top-level code environment”?::
* Idiomatic Usage::
* Packaging Considerations::
__main__.py in Python Packages
* Idiomatic Usage: Idiomatic Usage<2>.
warnings — Warning control
* Warning Categories::
* The Warnings Filter::
* Temporarily Suppressing Warnings::
* Testing Warnings::
* Updating Code For New Versions of Dependencies::
* Available Functions::
* Available Context Managers::
The Warnings Filter
* Repeated Warning Suppression Criteria::
* Describing Warning Filters::
* Default Warning Filter::
* Overriding the default filter::
dataclasses — Data Classes
* Module contents: Module contents<4>.
* Post-init processing::
* Class variables::
* Init-only variables::
* Frozen instances::
* Inheritance: Inheritance<2>.
* Re-ordering of keyword-only parameters in __init__(): Re-ordering of keyword-only parameters in __init__.
* Default factory functions::
* Mutable default values::
* Descriptor-typed fields::
contextlib — Utilities for with-statement contexts
* Utilities::
* Examples and Recipes: Examples and Recipes<3>.
* Single use, reusable and reentrant context managers: Single use reusable and reentrant context managers.
Examples and Recipes
* Supporting a variable number of context managers::
* Catching exceptions from __enter__ methods::
* Cleaning up in an __enter__ implementation::
* Replacing any use of try-finally and flag variables::
* Using a context manager as a function decorator::
Single use, reusable and reentrant context managers
* Reentrant context managers::
* Reusable context managers::
atexit — Exit handlers
* atexit Example::
traceback — Print or retrieve a stack traceback
* Module-Level Functions: Module-Level Functions<2>.
* TracebackException Objects::
* StackSummary Objects::
* FrameSummary Objects::
* Examples of Using the Module-Level Functions::
* Examples of Using TracebackException::
__future__ — Future statement definitions
* Module Contents: Module Contents<5>.
inspect — Inspect live objects
* Types and members::
* Retrieving source code::
* Introspecting callables with the Signature object::
* Classes and functions: Classes and functions<2>.
* The interpreter stack::
* Fetching attributes statically::
* Current State of Generators, Coroutines, and Asynchronous Generators: Current State of Generators Coroutines and Asynchronous Generators.
* Code Objects Bit Flags::
* Buffer flags::
* Command Line Interface: Command Line Interface<3>.
site — Site-specific configuration hook
* sitecustomize::
* usercustomize::
* Readline configuration::
* Module contents: Module contents<5>.
* Command Line Interface: Command Line Interface<4>.
Custom Python Interpreters
* code — Interpreter base classes::
* codeop — Compile Python code::
code — Interpreter base classes
* Interactive Interpreter Objects::
* Interactive Console Objects::
Importing Modules
* zipimport — Import modules from Zip archives::
* pkgutil — Package extension utility::
* modulefinder — Find modules used by a script::
* runpy — Locating and executing Python modules::
* importlib — The implementation of import::
* importlib.resources – Package resource reading, opening and access: importlib resources – Package resource reading opening and access.
* importlib.resources.abc – Abstract base classes for resources: importlib resources abc – Abstract base classes for resources.
* importlib.metadata – Accessing package metadata: importlib metadata – Accessing package metadata.
* The initialization of the sys.path module search path: The initialization of the sys path module search path.
zipimport — Import modules from Zip archives
* zipimporter Objects::
* Examples: Examples<33>.
modulefinder — Find modules used by a script
* Example usage of ModuleFinder::
importlib — The implementation of import
* Introduction: Introduction<12>.
* Functions: Functions<12>.
* importlib.abc – Abstract base classes related to import: importlib abc – Abstract base classes related to import.
* importlib.machinery – Importers and path hooks: importlib machinery – Importers and path hooks.
* importlib.util – Utility code for importers: importlib util – Utility code for importers.
* Examples: Examples<34>.
Examples
* Importing programmatically::
* Checking if a module can be imported::
* Importing a source file directly::
* Implementing lazy imports::
* Setting up an importer::
* Approximating importlib.import_module(): Approximating importlib import_module.
importlib.resources – Package resource reading, opening and access
* Functional API::
importlib.metadata – Accessing package metadata
* Overview: Overview<3>.
* Functional API: Functional API<2>.
* Distributions::
* Distribution Discovery::
* Extending the search algorithm::
Functional API
* Entry points::
* Distribution metadata::
* Distribution versions::
* Distribution files::
* Distribution requirements::
* Mapping import to distribution packages::
Extending the search algorithm
* Example: Example<14>.
The initialization of the sys.path module search path
* Virtual environments: Virtual environments<2>.
* _pth files::
* Embedded Python::
Python Language Services
* ast — Abstract Syntax Trees::
* symtable — Access to the compiler’s symbol tables::
* token — Constants used with Python parse trees::
* keyword — Testing for Python keywords::
* tokenize — Tokenizer for Python source::
* tabnanny — Detection of ambiguous indentation::
* pyclbr — Python module browser support::
* py_compile — Compile Python source files::
* compileall — Byte-compile Python libraries::
* dis — Disassembler for Python bytecode::
* pickletools — Tools for pickle developers::
ast — Abstract Syntax Trees
* Abstract Grammar::
* Node classes::
* ast Helpers::
* Compiler Flags::
* Command-Line Usage: Command-Line Usage<4>.
Node classes
* Root nodes::
* Literals: Literals<3>.
* Variables::
* Expressions: Expressions<2>.
* Statements::
* Control flow::
* Pattern matching::
* Type annotations::
* Type parameters::
* Function and class definitions::
* Async and await::
Expressions
* Subscripting::
* Comprehensions::
Statements
* Imports::
symtable — Access to the compiler’s symbol tables
* Generating Symbol Tables::
* Examining Symbol Tables::
* Command-Line Usage: Command-Line Usage<5>.
tokenize — Tokenizer for Python source
* Tokenizing Input::
* Command-Line Usage: Command-Line Usage<6>.
* Examples: Examples<35>.
pyclbr — Python module browser support
* Function Objects::
* Class Objects: Class Objects<2>.
py_compile — Compile Python source files
* Command-Line Interface: Command-Line Interface<6>.
compileall — Byte-compile Python libraries
* Command-line use::
* Public functions::
dis — Disassembler for Python bytecode
* Command-line interface: Command-line interface<3>.
* Bytecode analysis::
* Analysis functions::
* Python Bytecode Instructions::
* Opcode collections::
pickletools — Tools for pickle developers
* Command line usage: Command line usage<2>.
* Programmatic Interface: Programmatic Interface<2>.
Command line usage
* Command line options: Command line options<3>.
MS Windows Specific Services
* msvcrt — Useful routines from the MS VC++ runtime::
* winreg — Windows registry access::
* winsound — Sound-playing interface for Windows::
msvcrt — Useful routines from the MS VC++ runtime
* File Operations::
* Console I/O::
* Other Functions::
winreg — Windows registry access
* Functions: Functions<13>.
* Constants: Constants<10>.
* Registry Handle Objects::
Constants
* HKEY_* Constants::
* Access Rights::
* Value Types::
Access Rights
* 64-bit Specific::
Unix Specific Services
* posix — The most common POSIX system calls::
* pwd — The password database::
* grp — The group database::
* termios — POSIX style tty control::
* tty — Terminal control functions::
* pty — Pseudo-terminal utilities::
* fcntl — The fcntl and ioctl system calls::
* resource — Resource usage information::
* syslog — Unix syslog library routines::
posix — The most common POSIX system calls
* Large File Support::
* Notable Module Contents::
termios — POSIX style tty control
* Example: Example<15>.
pty — Pseudo-terminal utilities
* Example: Example<16>.
resource — Resource usage information
* Resource Limits::
* Resource Usage::
syslog — Unix syslog library routines
* Examples: Examples<36>.
Examples
* Simple example::
Superseded Modules
* getopt — C-style parser for command line options::
Removed Modules
* aifc — Read and write AIFF and AIFC files::
* asynchat — Asynchronous socket command/response handler::
* asyncore — Asynchronous socket handler::
* audioop — Manipulate raw audio data::
* cgi — Common Gateway Interface support::
* cgitb — Traceback manager for CGI scripts::
* chunk — Read IFF chunked data::
* crypt — Function to check Unix passwords::
* distutils — Building and installing Python modules::
* imghdr — Determine the type of an image::
* imp — Access the import internals::
* mailcap — Mailcap file handling::
* msilib — Read and write Microsoft Installer files::
* nis — Interface to Sun’s NIS (Yellow Pages): nis — Interface to Sun’s NIS Yellow Pages.
* nntplib — NNTP protocol client::
* ossaudiodev — Access to OSS-compatible audio devices::
* pipes — Interface to shell pipelines::
* smtpd — SMTP Server::
* sndhdr — Determine type of sound file::
* spwd — The shadow password database::
* sunau — Read and write Sun AU files::
* telnetlib — Telnet client::
* uu — Encode and decode uuencode files::
* xdrlib — Encode and decode XDR data::
Extending and Embedding the Python Interpreter
* Recommended third party tools::
* Creating extensions without third party tools::
* Embedding the CPython runtime in a larger application::
Creating extensions without third party tools
* Extending Python with C or C++::
* Defining Extension Types; Tutorial: Defining Extension Types Tutorial.
* Defining Extension Types; Assorted Topics: Defining Extension Types Assorted Topics.
* Building C and C++ Extensions::
* Building C and C++ Extensions on Windows::
Extending Python with C or C++
* A Simple Example::
* Intermezzo; Errors and Exceptions: Intermezzo Errors and Exceptions.
* Back to the Example::
* The Module’s Method Table and Initialization Function::
* Compilation and Linkage::
* Calling Python Functions from C::
* Extracting Parameters in Extension Functions::
* Keyword Parameters for Extension Functions::
* Building Arbitrary Values::
* Reference Counts::
* Writing Extensions in C++::
* Providing a C API for an Extension Module::
Reference Counts
* Reference Counting in Python::
* Ownership Rules::
* Thin Ice::
* NULL Pointers::
Defining Extension Types: Tutorial
* The Basics::
* Adding data and methods to the Basic example::
* Providing finer control over data attributes::
* Supporting cyclic garbage collection::
* Subclassing other types::
Defining Extension Types: Assorted Topics
* Finalization and De-allocation::
* Object Presentation::
* Attribute Management::
* Object Comparison::
* Abstract Protocol Support::
* Weak Reference Support::
* More Suggestions::
Attribute Management
* Generic Attribute Management::
* Type-specific Attribute Management::
Building C and C++ Extensions
* Building C and C++ Extensions with setuptools::
Building C and C++ Extensions on Windows
* A Cookbook Approach::
* Differences Between Unix and Windows::
* Using DLLs in Practice::
Embedding the CPython runtime in a larger application
* Embedding Python in Another Application::
Embedding Python in Another Application
* Very High Level Embedding::
* Beyond Very High Level Embedding; An overview: Beyond Very High Level Embedding An overview.
* Pure Embedding::
* Extending Embedded Python::
* Embedding Python in C++::
* Compiling and Linking under Unix-like systems::
Python/C API Reference Manual
* Introduction: Introduction<13>.
* C API Stability::
* The Very High Level Layer::
* Reference Counting::
* Exception Handling::
* Utilities: Utilities<2>.
* Abstract Objects Layer::
* Concrete Objects Layer::
* Initialization, Finalization, and Threads: Initialization Finalization and Threads.
* Python Initialization Configuration::
* Memory Management::
* Object Implementation Support::
* API and ABI Versioning::
* Monitoring C API::
* Generating Execution Events::
Introduction
* Coding standards::
* Include Files::
* Useful macros::
* Objects, Types and Reference Counts: Objects Types and Reference Counts.
* Exceptions: Exceptions<21>.
* Embedding Python: Embedding Python<2>.
* Debugging Builds::
* Recommended third party tools: Recommended third party tools<2>.
Objects, Types and Reference Counts
* Reference Counts: Reference Counts<2>.
* Types::
Reference Counts
* Reference Count Details::
C API Stability
* Unstable C API::
* Stable Application Binary Interface::
* Platform Considerations::
* Contents of Limited API::
Stable Application Binary Interface
* Limited C API::
* Stable ABI::
* Limited API Scope and Performance::
* Limited API Caveats::
Exception Handling
* Printing and clearing::
* Raising exceptions::
* Issuing warnings::
* Querying the error indicator::
* Signal Handling: Signal Handling<2>.
* Exception Classes::
* Exception Objects::
* Unicode Exception Objects::
* Recursion Control::
* Exception and warning types::
Exception and warning types
* Exception types::
* OSError aliases::
* Warning types::
Utilities
* Operating System Utilities::
* System Functions::
* Process Control::
* Importing Modules: Importing Modules<2>.
* Data marshalling support::
* Parsing arguments and building values::
* String conversion and formatting::
* PyHash API::
* Reflection::
* Codec registry and support functions::
* PyTime C API::
* Support for Perf Maps::
Parsing arguments and building values
* Parsing arguments: Parsing arguments<2>.
* Building values::
Parsing arguments
* Strings and buffers::
* Numbers: Numbers<2>.
* Other objects::
* API Functions::
Codec registry and support functions
* Codec lookup API::
* Registry API for Unicode encoding error handlers::
PyTime C API
* Types: Types<2>.
* Clock Functions::
* Raw Clock Functions::
* Conversion functions::
Abstract Objects Layer
* Object Protocol::
* Call Protocol::
* Number Protocol::
* Sequence Protocol::
* Mapping Protocol::
* Iterator Protocol::
* Buffer Protocol::
Call Protocol
* The tp_call Protocol::
* The Vectorcall Protocol::
* Object Calling API::
* Call Support API::
The Vectorcall Protocol
* Recursion Control: Recursion Control<2>.
* Vectorcall Support API::
Buffer Protocol
* Buffer structure::
* Buffer request types::
* Complex arrays::
* Buffer-related functions::
Buffer request types
* request-independent fields::
* readonly, format: readonly format.
* shape, strides, suboffsets: shape strides suboffsets.
* contiguity requests::
* compound requests::
Complex arrays
* NumPy-style; shape and strides: NumPy-style shape and strides.
* PIL-style; shape, strides and suboffsets: PIL-style shape strides and suboffsets.
Concrete Objects Layer
* Fundamental Objects::
* Numeric Objects::
* Sequence Objects::
* Container Objects::
* Function Objects: Function Objects<2>.
* Other Objects::
Fundamental Objects
* Type Objects: Type Objects<2>.
* The None Object::
Type Objects
* Creating Heap-Allocated Types::
Numeric Objects
* Integer Objects::
* Boolean Objects::
* Floating-Point Objects::
* Complex Number Objects::
Floating-Point Objects
* Pack and Unpack functions::
Pack and Unpack functions
* Pack functions::
* Unpack functions::
Complex Number Objects
* Complex Numbers as C Structures::
* Complex Numbers as Python Objects::
Sequence Objects
* Bytes Objects: Bytes Objects<2>.
* Byte Array Objects::
* Unicode Objects and Codecs::
* Tuple Objects::
* Struct Sequence Objects::
* List Objects::
Byte Array Objects
* Type check macros::
* Direct API functions::
* Macros::
Unicode Objects and Codecs
* Unicode Objects::
* Built-in Codecs::
* Methods and Slot Functions::
Unicode Objects
* Unicode Type::
* Unicode Character Properties::
* Creating and accessing Unicode strings::
* Locale Encoding::
* File System Encoding::
* wchar_t Support::
Built-in Codecs
* Generic Codecs::
* UTF-8 Codecs::
* UTF-32 Codecs::
* UTF-16 Codecs::
* UTF-7 Codecs::
* Unicode-Escape Codecs::
* Raw-Unicode-Escape Codecs::
* Latin-1 Codecs::
* ASCII Codecs::
* Character Map Codecs::
* MBCS codecs for Windows::
Container Objects
* Dictionary Objects::
* Set Objects::
Function Objects
* Function Objects: Function Objects<3>.
* Instance Method Objects::
* Method Objects: Method Objects<2>.
* Cell Objects::
* Code Objects: Code Objects<2>.
* Code Object Flags::
* Extra information::
Other Objects
* File Objects::
* Module Objects::
* Iterator Objects::
* Descriptor Objects::
* Slice Objects::
* MemoryView objects::
* Weak Reference Objects: Weak Reference Objects<2>.
* Capsules: Capsules<2>.
* Frame Objects::
* Generator Objects::
* Coroutine Objects: Coroutine Objects<2>.
* Context Variables Objects::
* DateTime Objects: DateTime Objects<2>.
* Objects for Type Hinting::
Module Objects
* Initializing C modules::
* Module lookup::
Initializing C modules
* Single-phase initialization::
* Multi-phase initialization::
* Low-level module creation functions::
* Support functions::
Slice Objects
* Ellipsis Object::
Frame Objects
* Frame Locals Proxies::
* Internal Frames::
Initialization, Finalization, and Threads
* Before Python Initialization::
* Global configuration variables::
* Initializing and finalizing the interpreter::
* Process-wide parameters::
* Thread State and the Global Interpreter Lock::
* Sub-interpreter support::
* Asynchronous Notifications::
* Profiling and Tracing::
* Reference tracing::
* Advanced Debugger Support::
* Thread Local Storage Support::
* Synchronization Primitives: Synchronization Primitives<2>.
Thread State and the Global Interpreter Lock
* Releasing the GIL from extension code::
* Non-Python created threads::
* Cautions about fork(): Cautions about fork.
* High-level API::
* Low-level API::
Sub-interpreter support
* A Per-Interpreter GIL::
* Bugs and caveats::
Thread Local Storage Support
* Thread Specific Storage (TSS) API: Thread Specific Storage TSS API.
* Thread Local Storage (TLS) API: Thread Local Storage TLS API.
Thread Specific Storage (TSS) API
* Dynamic Allocation::
* Methods: Methods<2>.
Synchronization Primitives
* Python Critical Section API::
Python Initialization Configuration
* Example: Example<17>.
* PyWideStringList::
* PyStatus::
* PyPreConfig::
* Preinitialize Python with PyPreConfig::
* PyConfig::
* Initialization with PyConfig::
* Isolated Configuration::
* Python Configuration::
* Python Path Configuration::
* Py_GetArgcArgv(): Py_GetArgcArgv.
* Multi-Phase Initialization Private Provisional API::
Memory Management
* Overview: Overview<4>.
* Allocator Domains::
* Raw Memory Interface::
* Memory Interface::
* Object allocators::
* Default Memory Allocators::
* Customize Memory Allocators::
* Debug hooks on the Python memory allocators::
* The pymalloc allocator::
* The mimalloc allocator::
* tracemalloc C API::
* Examples: Examples<37>.
The pymalloc allocator
* Customize pymalloc Arena Allocator::
Object Implementation Support
* Allocating Objects on the Heap::
* Common Object Structures::
* Type Object Structures::
* Supporting Cyclic Garbage Collection::
Common Object Structures
* Base object types and macros::
* Implementing functions and methods::
* Accessing attributes of extension types::
Accessing attributes of extension types
* Member flags::
* Member types::
* Defining Getters and Setters::
Type Object Structures
* Quick Reference::
* PyTypeObject Definition::
* PyObject Slots::
* PyVarObject Slots::
* PyTypeObject Slots::
* Static Types::
* Heap Types::
* Number Object Structures::
* Mapping Object Structures::
* Sequence Object Structures::
* Buffer Object Structures::
* Async Object Structures::
* Slot Type typedefs::
* Examples: Examples<38>.
Quick Reference
* “tp slots”::
* sub-slots::
* slot typedefs::
Supporting Cyclic Garbage Collection
* Controlling the Garbage Collector State::
* Querying Garbage Collector State::
Generating Execution Events
* Managing the Monitoring State::
Installing Python Modules
* Key terms::
* Basic usage::
* How do I …?::
* Common installation issues::
How do I …?
* … install pip in versions of Python prior to Python 3.4?: … install pip in versions of Python prior to Python 3 4?.
* … install packages just for the current user?::
* … install scientific Python packages?::
* … work with multiple versions of Python installed in parallel?::
Common installation issues
* Installing into the system Python on Linux::
* Pip not installed::
* Installing binary extensions::
Python HOWTOs
* A Conceptual Overview of asyncio::
* Porting Extension Modules to Python 3::
* Curses Programming with Python::
* Descriptor Guide::
* Debugging C API extensions and CPython Internals with GDB::
* Enum HOWTO::
* Functional Programming HOWTO::
* Logging HOWTO::
* Logging Cookbook::
* Regular Expression HOWTO::
* Socket Programming HOWTO::
* Sorting Techniques::
* Unicode HOWTO::
* HOWTO Fetch Internet Resources Using The urllib Package::
* An introduction to the ipaddress module::
* Instrumenting CPython with DTrace and SystemTap::
* Python support for the Linux perf profiler::
* Annotations Best Practices::
* Isolating Extension Modules::
* timer file descriptor HOWTO::
* The Python 2.3 Method Resolution Order: The Python 2 3 Method Resolution Order.
* Python experimental support for free threading::
* C API Extension Support for Free Threading::
A Conceptual Overview of asyncio
* A conceptual overview part 1; the high-level: A conceptual overview part 1 the high-level.
* A conceptual overview part 2; the nuts and bolts: A conceptual overview part 2 the nuts and bolts.
A conceptual overview part 1: the high-level
* Event Loop: Event Loop<2>.
* Asynchronous functions and coroutines::
* Tasks: Tasks<2>.
* await::
A conceptual overview part 2: the nuts and bolts
* The inner workings of coroutines::
* Futures: Futures<2>.
* A homemade asyncio.sleep: A homemade asyncio sleep.
Curses Programming with Python
* What is curses?::
* Starting and ending a curses application::
* Windows and Pads::
* Displaying Text::
* User Input::
* For More Information::
What is curses?
* The Python curses module::
Displaying Text
* Attributes and Color::
Descriptor Guide
* Primer::
* Complete Practical Example::
* Technical Tutorial::
* Pure Python Equivalents::
Primer
* Simple example; A descriptor that returns a constant: Simple example A descriptor that returns a constant.
* Dynamic lookups::
* Managed attributes::
* Customized names::
* Closing thoughts::
Complete Practical Example
* Validator class::
* Custom validators::
* Practical application::
Technical Tutorial
* Abstract::
* Definition and introduction::
* Descriptor protocol::
* Overview of descriptor invocation::
* Invocation from an instance::
* Invocation from a class::
* Invocation from super::
* Summary of invocation logic::
* Automatic name notification::
* ORM example::
Pure Python Equivalents
* Properties::
* Functions and methods::
* Kinds of methods::
* Static methods::
* Class methods::
* Member objects and __slots__::
Debugging C API extensions and CPython Internals with GDB
* Prerequisites::
* Using the Debug build and Development mode::
* Using the python-gdb extension::
* Use with GDB commands::
Prerequisites
* Setup with Python built from source::
* Setup for Python from a Linux distro::
Using the python-gdb extension
* Pretty-printers::
* py-list::
* py-up and py-down::
* py-bt::
* py-print::
* py-locals::
Enum HOWTO
* Programmatic access to enumeration members and their attributes::
* Duplicating enum members and values::
* Ensuring unique enumeration values::
* Using automatic values::
* Iteration: Iteration<2>.
* Comparisons: Comparisons<3>.
* Allowed members and attributes of enumerations::
* Restricted Enum subclassing::
* Dataclass support::
* Pickling::
* Functional API: Functional API<3>.
* Derived Enumerations::
* When to use __new__() vs. __init__(): When to use __new__ vs __init__.
* How are Enums and Flags different?::
* Enum Cookbook::
* Subclassing EnumType::
Derived Enumerations
* IntEnum::
* StrEnum::
* IntFlag::
* Flag::
* Others: Others<2>.
When to use __new__() vs. __init__()
* Finer Points::
Finer Points
* Supported __dunder__ names: Supported __dunder__ names<2>.
* Supported _sunder_ names: Supported _sunder_ names<2>.
* _Private__names::
* Enum member type::
* Creating members that are mixed with other data types::
* Boolean value of Enum classes and members::
* Enum classes with methods::
* Combining members of Flag::
* Flag and IntFlag minutia::
How are Enums and Flags different?
* Enum Classes::
* Flag Classes::
* Enum Members (aka instances): Enum Members aka instances.
* Flag Members::
Enum Cookbook
* Omitting values::
* OrderedEnum::
* DuplicateFreeEnum::
* MultiValueEnum::
* Planet::
* TimePeriod::
Omitting values
* Using auto::
* Using object::
* Using a descriptive string::
* Using a custom __new__(): Using a custom __new__.
Functional Programming HOWTO
* Introduction: Introduction<14>.
* Iterators: Iterators<2>.
* Generator expressions and list comprehensions::
* Generators: Generators<2>.
* Built-in functions: Built-in functions<2>.
* The itertools module::
* The functools module::
* Small functions and the lambda expression::
* Revision History and Acknowledgements::
* References: References<2>.
Introduction
* Formal provability::
* Modularity::
* Ease of debugging and testing::
* Composability::
Iterators
* Data Types That Support Iterators::
Generators
* Passing values into a generator::
The itertools module
* Creating new iterators::
* Calling functions on elements::
* Selecting elements::
* Combinatoric functions::
* Grouping elements::
The functools module
* The operator module::
References
* General::
* Python-specific::
* Python documentation::
Logging HOWTO
* Basic Logging Tutorial::
* Advanced Logging Tutorial::
* Logging Levels: Logging Levels<2>.
* Useful Handlers::
* Exceptions raised during logging::
* Using arbitrary objects as messages::
* Optimization::
* Other resources::
Basic Logging Tutorial
* When to use logging::
* A simple example::
* Logging to a file::
* Logging variable data::
* Changing the format of displayed messages::
* Displaying the date/time in messages::
* Next Steps::
Advanced Logging Tutorial
* Logging Flow::
* Loggers::
* Handlers::
* Formatters::
* Configuring Logging::
* What happens if no configuration is provided::
* Configuring Logging for a Library::
Logging Levels
* Custom Levels::
Logging Cookbook
* Using logging in multiple modules::
* Logging from multiple threads::
* Multiple handlers and formatters::
* Logging to multiple destinations::
* Custom handling of levels::
* Configuration server example::
* Dealing with handlers that block::
* Sending and receiving logging events across a network::
* Adding contextual information to your logging output::
* Use of contextvars::
* Imparting contextual information in handlers::
* Logging to a single file from multiple processes::
* Using file rotation::
* Use of alternative formatting styles::
* Customizing LogRecord::
* Subclassing QueueHandler and QueueListener- a ZeroMQ example::
* Subclassing QueueHandler and QueueListener- a pynng example::
* An example dictionary-based configuration::
* Using a rotator and namer to customize log rotation processing::
* A more elaborate multiprocessing example::
* Inserting a BOM into messages sent to a SysLogHandler::
* Implementing structured logging::
* Customizing handlers with dictConfig(): Customizing handlers with dictConfig.
* Using particular formatting styles throughout your application::
* Configuring filters with dictConfig(): Configuring filters with dictConfig.
* Customized exception formatting::
* Speaking logging messages::
* Buffering logging messages and outputting them conditionally::
* Sending logging messages to email, with buffering: Sending logging messages to email with buffering.
* Formatting times using UTC (GMT) via configuration: Formatting times using UTC GMT via configuration.
* Using a context manager for selective logging::
* A CLI application starter template::
* A Qt GUI for logging::
* Logging to syslog with RFC5424 support::
* How to treat a logger like an output stream::
* How to uniformly handle newlines in logging output::
* Patterns to avoid::
* Other resources: Other resources<2>.
Sending and receiving logging events across a network
* Running a logging socket listener in production::
Adding contextual information to your logging output
* Using LoggerAdapters to impart contextual information::
* Using Filters to impart contextual information::
Using LoggerAdapters to impart contextual information
* Using objects other than dicts to pass contextual information::
Logging to a single file from multiple processes
* Using concurrent.futures.ProcessPoolExecutor: Using concurrent futures ProcessPoolExecutor.
* Deploying Web applications using Gunicorn and uWSGI::
Subclassing QueueHandler and QueueListener- a ZeroMQ example
* Subclass QueueHandler::
* Subclass QueueListener::
Subclassing QueueHandler and QueueListener- a pynng example
* Subclass QueueListener: Subclass QueueListener<2>.
* Subclass QueueHandler: Subclass QueueHandler<2>.
Using particular formatting styles throughout your application
* Using LogRecord factories::
* Using custom message objects::
Patterns to avoid
* Opening the same log file multiple times::
* Using loggers as attributes in a class or passing them as parameters::
* Adding handlers other than NullHandler to a logger in a library::
* Creating a lot of loggers::
Regular Expression HOWTO
* Introduction: Introduction<15>.
* Simple Patterns::
* Using Regular Expressions::
* More Pattern Power::
* Modifying Strings::
* Common Problems::
* Feedback::
Simple Patterns
* Matching Characters::
* Repeating Things::
Using Regular Expressions
* Compiling Regular Expressions::
* The Backslash Plague::
* Performing Matches::
* Module-Level Functions: Module-Level Functions<3>.
* Compilation Flags::
More Pattern Power
* More Metacharacters::
* Grouping::
* Non-capturing and Named Groups::
* Lookahead Assertions::
Modifying Strings
* Splitting Strings::
* Search and Replace: Search and Replace<2>.
Common Problems
* Use String Methods::
* match() versus search(): match versus search.
* Greedy versus Non-Greedy::
* Using re.VERBOSE: Using re VERBOSE.
Socket Programming HOWTO
* Sockets::
* Creating a Socket::
* Using a Socket::
* Disconnecting::
* Non-blocking Sockets::
Sockets
* History::
Creating a Socket
* IPC::
Using a Socket
* Binary Data::
Disconnecting
* When Sockets Die::
Sorting Techniques
* Sorting Basics::
* Key Functions::
* Operator Module Functions and Partial Function Evaluation::
* Ascending and Descending::
* Sort Stability and Complex Sorts::
* Decorate-Sort-Undecorate::
* Comparison Functions::
* Odds and Ends: Odds and Ends<2>.
* Partial Sorts::
Unicode HOWTO
* Introduction to Unicode::
* Python’s Unicode Support::
* Reading and Writing Unicode Data::
* Acknowledgements: Acknowledgements<10>.
Introduction to Unicode
* Definitions::
* Encodings::
* References: References<3>.
Python’s Unicode Support
* The String Type::
* Converting to Bytes::
* Unicode Literals in Python Source Code::
* Unicode Properties::
* Comparing Strings::
* Unicode Regular Expressions::
* References: References<4>.
Reading and Writing Unicode Data
* Unicode filenames::
* Tips for Writing Unicode-aware Programs::
* References: References<5>.
Tips for Writing Unicode-aware Programs
* Converting Between File Encodings::
* Files in an Unknown Encoding::
HOWTO Fetch Internet Resources Using The urllib Package
* Introduction: Introduction<16>.
* Fetching URLs::
* Handling Exceptions: Handling Exceptions<2>.
* info and geturl::
* Openers and Handlers::
* Basic Authentication::
* Proxies::
* Sockets and Layers::
* Footnotes::
Fetching URLs
* Data::
* Headers::
Handling Exceptions
* URLError::
* HTTPError::
* Wrapping it Up::
HTTPError
* Error Codes::
Wrapping it Up
* Number 1::
* Number 2::
An introduction to the ipaddress module
* Creating Address/Network/Interface objects::
* Inspecting Address/Network/Interface Objects::
* Networks as lists of Addresses::
* Comparisons: Comparisons<4>.
* Using IP Addresses with other modules::
* Getting more detail when instance creation fails::
Creating Address/Network/Interface objects
* A Note on IP Versions::
* IP Host Addresses::
* Defining Networks::
* Host Interfaces::
Instrumenting CPython with DTrace and SystemTap
* Enabling the static markers::
* Static DTrace probes::
* Static SystemTap markers::
* Available static markers::
* SystemTap Tapsets::
* Examples: Examples<39>.
Python support for the Linux perf profiler
* How to enable perf profiling support::
* How to obtain the best results::
* How to work without frame pointers::
Annotations Best Practices
* Accessing The Annotations Dict Of An Object In Python 3.10 And Newer: Accessing The Annotations Dict Of An Object In Python 3 10 And Newer.
* Accessing The Annotations Dict Of An Object In Python 3.9 And Older: Accessing The Annotations Dict Of An Object In Python 3 9 And Older.
* Manually Un-Stringizing Stringized Annotations::
* Best Practices For __annotations__ In Any Python Version::
* __annotations__ Quirks::
Isolating Extension Modules
* Who should read this::
* Background: Background<2>.
* Making Modules Safe with Multiple Interpreters::
* Heap Types: Heap Types<2>.
* Open Issues::
Background
* Enter Per-Module State::
* Isolated Module Objects::
* Surprising Edge Cases::
Making Modules Safe with Multiple Interpreters
* Managing Global State::
* Managing Per-Module State::
* Opt-Out; Limiting to One Module Object per Process: Opt-Out Limiting to One Module Object per Process.
* Module State Access from Functions::
Heap Types
* Changing Static Types to Heap Types::
* Defining Heap Types::
* Garbage-Collection Protocol::
* Module State Access from Classes::
* Module State Access from Regular Methods::
* Module State Access from Slot Methods, Getters and Setters: Module State Access from Slot Methods Getters and Setters.
* Lifetime of the Module State::
Garbage-Collection Protocol
* tp_traverse in Python 3.8 and lower: tp_traverse in Python 3 8 and lower.
* Delegating tp_traverse::
* Defining tp_dealloc::
* Not overriding tp_free::
* Avoiding PyObject_New::
Open Issues
* Per-Class Scope::
* Lossless Conversion to Heap Types::
timer file descriptor HOWTO
* Examples: Examples<40>.
The Python 2.3 Method Resolution Order
* The beginning::
* The C3 Method Resolution Order::
* Examples: Examples<41>.
* Bad Method Resolution Orders::
* The end::
* Resources::
Python experimental support for free threading
* Installation::
* Identifying free-threaded Python::
* The global interpreter lock in free-threaded Python::
* Thread safety::
* Known limitations::
Known limitations
* Immortalization::
* Frame objects: Frame objects<2>.
* Iterators: Iterators<3>.
* Single-threaded performance::
C API Extension Support for Free Threading
* Identifying the Free-Threaded Build in C::
* Module Initialization::
* General API Guidelines::
* Borrowed References::
* Memory Allocation APIs::
* Thread State and GIL APIs::
* Protecting Internal Extension State::
* Building Extensions for the Free-Threaded Build::
Module Initialization
* Multi-Phase Initialization::
* Single-Phase Initialization::
General API Guidelines
* Container Thread Safety::
Container Thread Safety
* PyDict_Next::
Building Extensions for the Free-Threaded Build
* Limited C API and Stable ABI::
* Windows: Windows<80>.
Python Frequently Asked Questions
* General Python FAQ::
* Programming FAQ::
* Design and History FAQ::
* Library and Extension FAQ::
* Extending/Embedding FAQ::
* Python on Windows FAQ::
* Graphic User Interface FAQ::
* “Why is Python Installed on my Computer?” FAQ::
General Python FAQ
* General Information::
* Python in the real world::
General Information
* What is Python?::
* What is the Python Software Foundation?::
* Are there copyright restrictions on the use of Python?::
* Why was Python created in the first place?::
* What is Python good for?::
* How does the Python version numbering scheme work?::
* How do I obtain a copy of the Python source?::
* How do I get documentation on Python?::
* I’ve never programmed before. Is there a Python tutorial?: I’ve never programmed before Is there a Python tutorial?.
* Is there a newsgroup or mailing list devoted to Python?::
* How do I get a beta test version of Python?::
* How do I submit bug reports and patches for Python?::
* Are there any published articles about Python that I can reference?::
* Are there any books on Python?::
* Where in the world is www.python.org located?: Where in the world is www python org located?.
* Why is it called Python?::
* Do I have to like “Monty Python’s Flying Circus”?::
Python in the real world
* How stable is Python?::
* How many people are using Python?::
* Have any significant projects been done in Python?::
* What new developments are expected for Python in the future?::
* Is it reasonable to propose incompatible changes to Python?::
* Is Python a good language for beginning programmers?::
Programming FAQ
* General Questions::
* Core Language::
* Numbers and strings::
* Performance: Performance<4>.
* Sequences (Tuples/Lists): Sequences Tuples/Lists.
* Objects::
* Modules: Modules<5>.
General Questions
* Is there a source code level debugger with breakpoints, single-stepping, etc.?: Is there a source code level debugger with breakpoints single-stepping etc ?.
* Are there tools to help find bugs or perform static analysis?::
* How can I create a stand-alone binary from a Python script?::
* Are there coding standards or a style guide for Python programs?::
Core Language
* Why am I getting an UnboundLocalError when the variable has a value?::
* What are the rules for local and global variables in Python?::
* Why do lambdas defined in a loop with different values all return the same result?::
* How do I share global variables across modules?::
* What are the “best practices” for using import in a module?::
* Why are default values shared between objects?::
* How can I pass optional or keyword parameters from one function to another?::
* What is the difference between arguments and parameters?::
* Why did changing list ‘y’ also change list ‘x’?::
* How do I write a function with output parameters (call by reference)?: How do I write a function with output parameters call by reference ?.
* How do you make a higher order function in Python?::
* How do I copy an object in Python?::
* How can I find the methods or attributes of an object?::
* How can my code discover the name of an object?::
* What’s up with the comma operator’s precedence?::
* Is there an equivalent of C’s “?;” ternary operator?: Is there an equivalent of C’s “? ” ternary operator?.
* Is it possible to write obfuscated one-liners in Python?::
* What does the slash(/) in the parameter list of a function mean?: What does the slash / in the parameter list of a function mean?.
Numbers and strings
* How do I specify hexadecimal and octal integers?::
* Why does -22 // 10 return -3?::
* How do I get int literal attribute instead of SyntaxError?::
* How do I convert a string to a number?::
* How do I convert a number to a string?::
* How do I modify a string in place?::
* How do I use strings to call functions/methods?::
* Is there an equivalent to Perl’s chomp() for removing trailing newlines from strings?: Is there an equivalent to Perl’s chomp for removing trailing newlines from strings?.
* Is there a scanf() or sscanf() equivalent?: Is there a scanf or sscanf equivalent?.
* What does UnicodeDecodeError or UnicodeEncodeError error mean?::
* Can I end a raw string with an odd number of backslashes?::
Performance
* My program is too slow. How do I speed it up?: My program is too slow How do I speed it up?.
* What is the most efficient way to concatenate many strings together?::
Sequences (Tuples/Lists)
* How do I convert between tuples and lists?::
* What’s a negative index?::
* How do I iterate over a sequence in reverse order?::
* How do you remove duplicates from a list?::
* How do you remove multiple items from a list::
* How do you make an array in Python?::
* How do I create a multidimensional list?::
* How do I apply a method or function to a sequence of objects?::
* Why does a_tuple[i] += [‘item’] raise an exception when the addition works?::
* I want to do a complicated sort; can you do a Schwartzian Transform in Python?: I want to do a complicated sort can you do a Schwartzian Transform in Python?.
* How can I sort one list by values from another list?::
Objects
* What is a class?::
* What is a method?::
* What is self?::
* How do I check if an object is an instance of a given class or of a subclass of it?::
* What is delegation?::
* How do I call a method defined in a base class from a derived class that extends it?::
* How can I organize my code to make it easier to change the base class?::
* How do I create static class data and static class methods?::
* How can I overload constructors (or methods) in Python?: How can I overload constructors or methods in Python?.
* I try to use __spam and I get an error about _SomeClassName__spam.: I try to use __spam and I get an error about _SomeClassName__spam.
* My class defines __del__ but it is not called when I delete the object.: My class defines __del__ but it is not called when I delete the object.
* How do I get a list of all instances of a given class?::
* Why does the result of id() appear to be not unique?: Why does the result of id appear to be not unique?.
* When can I rely on identity tests with the is operator?::
* How can a subclass control what data is stored in an immutable instance?::
* How do I cache method calls?::
Modules
* How do I create a .pyc file?: How do I create a pyc file?.
* How do I find the current module name?::
* How can I have modules that mutually import each other?::
* __import__(‘x.y.z’) returns ; how do I get z?: __import__ ‘x y z’ returns ; how do I get z?.
* When I edit an imported module and reimport it, the changes don’t show up. Why does this happen?: When I edit an imported module and reimport it the changes don’t show up Why does this happen?.
Design and History FAQ
* Why does Python use indentation for grouping of statements?::
* Why am I getting strange results with simple arithmetic operations?::
* Why are floating-point calculations so inaccurate?::
* Why are Python strings immutable?::
* Why must ‘self’ be used explicitly in method definitions and calls?::
* Why can’t I use an assignment in an expression?::
* Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?: Why does Python use methods for some functionality e g list index but functions for other e g len list ?.
* Why is join() a string method instead of a list or tuple method?: Why is join a string method instead of a list or tuple method?.
* How fast are exceptions?::
* Why isn’t there a switch or case statement in Python?::
* Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?::
* Why can’t lambda expressions contain statements?::
* Can Python be compiled to machine code, C or some other language?: Can Python be compiled to machine code C or some other language?.
* How does Python manage memory?::
* Why doesn’t CPython use a more traditional garbage collection scheme?::
* Why isn’t all memory freed when CPython exits?::
* Why are there separate tuple and list data types?::
* How are lists implemented in CPython?::
* How are dictionaries implemented in CPython?::
* Why must dictionary keys be immutable?::
* Why doesn’t list.sort() return the sorted list?: Why doesn’t list sort return the sorted list?.
* How do you specify and enforce an interface spec in Python?::
* Why is there no goto?::
* Why can’t raw strings (r-strings) end with a backslash?: Why can’t raw strings r-strings end with a backslash?.
* Why doesn’t Python have a “with” statement for attribute assignments?::
* Why don’t generators support the with statement?::
* Why are colons required for the if/while/def/class statements?::
* Why does Python allow commas at the end of lists and tuples?::
Library and Extension FAQ
* General Library Questions::
* Common tasks::
* Threads::
* Input and Output: Input and Output<2>.
* Network/Internet Programming::
* Databases::
* Mathematics and Numerics::
General Library Questions
* How do I find a module or application to perform task X?::
* Where is the math.py (socket.py, regex.py, etc.) source file?: Where is the math py socket py regex py etc source file?.
* How do I make a Python script executable on Unix?::
* Is there a curses/termcap package for Python?::
* Is there an equivalent to C’s onexit() in Python?: Is there an equivalent to C’s onexit in Python?.
* Why don’t my signal handlers work?::
Common tasks
* How do I test a Python program or component?::
* How do I create documentation from doc strings?::
* How do I get a single keypress at a time?::
Threads
* How do I program using threads?::
* None of my threads seem to run; why?: None of my threads seem to run why?.
* How do I parcel out work among a bunch of worker threads?::
* What kinds of global value mutation are thread-safe?::
* Can’t we get rid of the Global Interpreter Lock?::
Input and Output
* How do I delete a file? (And other file questions…): How do I delete a file? And other file questions….
* How do I copy a file?::
* How do I read (or write) binary data?: How do I read or write binary data?.
* I can’t seem to use os.read() on a pipe created with os.popen(); why?: I can’t seem to use os read on a pipe created with os popen ; why?.
* How do I access the serial (RS232) port?: How do I access the serial RS232 port?.
* Why doesn’t closing sys.stdout (stdin, stderr) really close it?: Why doesn’t closing sys stdout stdin stderr really close it?.
Network/Internet Programming
* What WWW tools are there for Python?::
* What module should I use to help with generating HTML?::
* How do I send mail from a Python script?::
* How do I avoid blocking in the connect() method of a socket?: How do I avoid blocking in the connect method of a socket?.
Databases
* Are there any interfaces to database packages in Python?::
* How do you implement persistent objects in Python?::
Mathematics and Numerics
* How do I generate random numbers in Python?::
Extending/Embedding FAQ
* Can I create my own functions in C?::
* Can I create my own functions in C++?::
* Writing C is hard; are there any alternatives?::
* How can I execute arbitrary Python statements from C?::
* How can I evaluate an arbitrary Python expression from C?::
* How do I extract C values from a Python object?::
* How do I use Py_BuildValue() to create a tuple of arbitrary length?: How do I use Py_BuildValue to create a tuple of arbitrary length?.
* How do I call an object’s method from C?::
* How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)?: How do I catch the output from PyErr_Print or anything that prints to stdout/stderr ?.
* How do I access a module written in Python from C?::
* How do I interface to C++ objects from Python?::
* I added a module using the Setup file and the make fails; why?::
* How do I debug an extension?::
* I want to compile a Python module on my Linux system, but some files are missing. Why?: I want to compile a Python module on my Linux system but some files are missing Why?.
* How do I tell “incomplete input” from “invalid input”?::
* How do I find undefined g++ symbols __builtin_new or __pure_virtual?::
* Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)?: Can I create an object class with some methods implemented in C and others in Python e g through inheritance ?.
Python on Windows FAQ
* How do I run a Python program under Windows?::
* How do I make Python scripts executable?::
* Why does Python sometimes take so long to start?::
* How do I make an executable from a Python script?::
* Is a *.pyd file the same as a DLL?: Is a * pyd file the same as a DLL?.
* How can I embed Python into a Windows application?::
* How do I keep editors from inserting tabs into my Python source?::
* How do I check for a keypress without blocking?::
* How do I solve the missing api-ms-win-crt-runtime-l1-1-0.dll error?: How do I solve the missing api-ms-win-crt-runtime-l1-1-0 dll error?.
Graphic User Interface FAQ
* General GUI Questions::
* What GUI toolkits exist for Python?::
* Tkinter questions::
Tkinter questions
* How do I freeze Tkinter applications?::
* Can I have Tk events handled while waiting for I/O?::
* I can’t get key bindings to work in Tkinter; why?: I can’t get key bindings to work in Tkinter why?.
“Why is Python Installed on my Computer?” FAQ
* What is Python?: What is Python?<2>.
* Why is Python installed on my machine?::
* Can I delete Python?::
Deprecations
* Pending Removal in Python 3.14: Pending Removal in Python 3 14<5>.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15<5>.
* Pending removal in Python 3.16: Pending removal in Python 3 16<5>.
* Pending Removal in Future Versions: Pending Removal in Future Versions<5>.
* C API Deprecations::
C API Deprecations
* Pending Removal in Python 3.14: Pending Removal in Python 3 14<6>.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15<6>.
* Pending Removal in Future Versions: Pending Removal in Future Versions<6>.
About this documentation
* Contributors to the Python documentation::
Dealing with Bugs
* Documentation bugs::
* Using the Python issue tracker::
* Getting started contributing to Python yourself::
History and License
* History of the software::
* Terms and conditions for accessing or otherwise using Python::
* Licenses and Acknowledgements for Incorporated Software::
Terms and conditions for accessing or otherwise using Python
* PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2::
* BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0: BEOPEN COM LICENSE AGREEMENT FOR PYTHON 2 0.
* CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1: CNRI LICENSE AGREEMENT FOR PYTHON 1 6 1.
* CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2: CWI LICENSE AGREEMENT FOR PYTHON 0 9 0 THROUGH 1 2.
* ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION::
Licenses and Acknowledgements for Incorporated Software
* Mersenne Twister::
* Sockets: Sockets<2>.
* Asynchronous socket services::
* Cookie management::
* Execution tracing::
* UUencode and UUdecode functions::
* XML Remote Procedure Calls::
* test_epoll::
* Select kqueue::
* SipHash24::
* strtod and dtoa::
* OpenSSL: OpenSSL<2>.
* expat::
* libffi::
* zlib: zlib<3>.
* cfuhash::
* libmpdec::
* W3C C14N test suite::
* mimalloc::
* asyncio: asyncio<12>.
* Global Unbounded Sequences (GUS): Global Unbounded Sequences GUS.
File: python.info, Node: What’s New in Python, Next: The Python Tutorial, Prev: Top, Up: Top
1 What’s New in Python
**********************
The “What’s New in Python” series of essays takes tours through the most
important changes between major Python versions. They are a “must read”
for anyone wishing to stay up-to-date after a new release.
* Menu:
* What’s New In Python 3.13: What’s New In Python 3 13.
* What’s New In Python 3.12: What’s New In Python 3 12.
* What’s New In Python 3.11: What’s New In Python 3 11.
* What’s New In Python 3.10: What’s New In Python 3 10.
* What’s New In Python 3.9: What’s New In Python 3 9.
* What’s New In Python 3.8: What’s New In Python 3 8.
* What’s New In Python 3.7: What’s New In Python 3 7.
* What’s New In Python 3.6: What’s New In Python 3 6.
* What’s New In Python 3.5: What’s New In Python 3 5.
* What’s New In Python 3.4: What’s New In Python 3 4.
* What’s New In Python 3.3: What’s New In Python 3 3.
* What’s New In Python 3.2: What’s New In Python 3 2.
* What’s New In Python 3.1: What’s New In Python 3 1.
* What’s New In Python 3.0: What’s New In Python 3 0.
* What’s New in Python 2.7: What’s New in Python 2 7.
* What’s New in Python 2.6: What’s New in Python 2 6.
* What’s New in Python 2.5: What’s New in Python 2 5.
* What’s New in Python 2.4: What’s New in Python 2 4.
* What’s New in Python 2.3: What’s New in Python 2 3.
* What’s New in Python 2.2: What’s New in Python 2 2.
* What’s New in Python 2.1: What’s New in Python 2 1.
* What’s New in Python 2.0: What’s New in Python 2 0.
* Changelog::
File: python.info, Node: What’s New In Python 3 13, Next: What’s New In Python 3 12, Up: What’s New in Python
1.1 What’s New In Python 3.13
=============================
Editors: Adam Turner and Thomas Wouters
This article explains the new features in Python 3.13, compared to 3.12.
Python 3.13 was released on October 7, 2024. For full details, see the
*note changelog: 13c.
See also
........
PEP 719(1) – Python 3.13 Release Schedule
* Menu:
* Summary – Release Highlights::
* New Features::
* Other Language Changes::
* New Modules::
* Improved Modules::
* Optimizations::
* Removed Modules And APIs::
* New Deprecations::
* CPython Bytecode Changes::
* C API Changes::
* Build Changes::
* Porting to Python 3.13: Porting to Python 3 13.
* Regression Test Changes::
* Notable changes in 3.13.1: Notable changes in 3 13 1.
* Notable changes in 3.13.4: Notable changes in 3 13 4.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0719/
File: python.info, Node: Summary – Release Highlights, Next: New Features, Up: What’s New In Python 3 13
1.1.1 Summary – Release Highlights
----------------------------------
Python 3.13 is the latest stable release of the Python programming
language, with a mix of changes to the language, the implementation and
the standard library. The biggest changes include a new *note
interactive interpreter: 13e., experimental support for running in a
*note free-threaded mode: 13f.( PEP 703(1)), and a *note Just-In-Time
compiler: 140.( PEP 744(2)).
Error messages continue to improve, with tracebacks now highlighted in
color by default. The *note locals(): 141. builtin now has *note
defined semantics: 142. for changing the returned mapping, and type
parameters now support default values.
The library changes contain removal of deprecated APIs and modules, as
well as the usual improvements in user-friendliness and correctness.
Several legacy standard library modules have now *note been removed:
143.following their deprecation in Python 3.11 ( PEP 594(3)).
This article doesn’t attempt to provide a complete specification of all
new features, but instead gives a convenient overview. For full details
refer to the documentation, such as the *note Library Reference: 144.
and *note Language Reference: 145. To understand the complete
implementation and design rationale for a change, refer to the PEP for a
particular new feature; but note that PEPs usually are not kept
up-to-date once a feature has been fully implemented. See *note Porting
to Python 3.13: 146. for guidance on upgrading from earlier versions of
Python.
__________________________________________________________________
Interpreter improvements:
* A greatly improved *note interactive interpreter: 13e. and *note
improved error messages: 147.
* PEP 667(4): The *note locals(): 141. builtin now has *note defined
semantics: 142. when mutating the returned mapping. Python
debuggers and similar tools may now more reliably update local
variables in optimized scopes even during concurrent code
execution.
* PEP 703(5): CPython 3.13 has experimental support for running with
the *note global interpreter lock: 148. disabled. See *note
Free-threaded CPython: 13f. for more details.
* PEP 744(6): A basic *note JIT compiler: 140. was added. It is
currently disabled by default (though we may turn it on later).
Performance improvements are modest – we expect to improve this
over the next few releases.
* Color support in the new *note interactive interpreter: 13e, as
well as in *note tracebacks: 147. and *note doctest: 149. output.
This can be disabled through the *note PYTHON_COLORS: 14a. and
NO_COLOR(7) environment variables.
Python data model improvements:
* *note __static_attributes__: 14b. stores the names of attributes
accessed through ‘self.X’ in any function in a class body.
* *note __firstlineno__: 14c. records the first line number of a
class definition.
Significant improvements in the standard library:
* Add a new *note PythonFinalizationError: 14d. exception, raised
when an operation is blocked during *note finalization: 14e.
* The *note argparse: 6. module now supports deprecating command-line
options, positional arguments, and subcommands.
* The new functions *note base64.z85encode(): 14f. and *note
base64.z85decode(): 150. support encoding and decoding Z85 data(8).
* The *note copy: 25. module now has a *note copy.replace(): 151.
function, with support for many builtin types and any class
defining the *note __replace__(): 152. method.
* The new *note dbm.sqlite3: 35. module is now the default *note dbm:
31. backend.
* The *note os: a1. module has a *note suite of new functions: 153.
for working with Linux’s timer notification file descriptors.
* The *note random: b8. module now has a *note command-line
interface: 154.
Security improvements:
* *note ssl.create_default_context(): 155. sets *note
ssl.VERIFY_X509_PARTIAL_CHAIN: 156. and *note
ssl.VERIFY_X509_STRICT: 157. as default flags.
C API improvements:
* The *note Py_mod_gil: 158. slot is now used to indicate that an
extension module supports running with the *note GIL: 159.
disabled.
* The *note PyTime C API: 15a. has been added, providing access to
system clocks.
* *note PyMutex: 15b. is a new lightweight mutex that occupies a
single byte.
* There is a new *note suite of functions: 15c. for generating PEP
669(9) monitoring events in the C API.
New typing features:
* PEP 696(10): Type parameters (*note typing.TypeVar: 15d, *note
typing.ParamSpec: 15e, and *note typing.TypeVarTuple: 15f.) now
support defaults.
* PEP 702(11): The new *note warnings.deprecated(): 160. decorator
adds support for marking deprecations in the type system and at
runtime.
* PEP 705(12): *note typing.ReadOnly: 161. can be used to mark an
item of a *note typing.TypedDict: 162. as read-only for type
checkers.
* PEP 742(13): *note typing.TypeIs: 163. provides more intuitive type
narrowing behavior, as an alternative to *note typing.TypeGuard:
164.
Platform support:
* PEP 730(14): Apple’s iOS is now an *note officially supported
platform: 165, at tier 3(15).
* PEP 738(16): Android is now an *note officially supported platform:
165, at tier 3(17).
* ‘wasm32-wasi’ is now supported as a tier 2(18) platform.
* ‘wasm32-emscripten’ is no longer an officially supported platform.
Important removals:
* *note PEP 594: 143.: The remaining 19 “dead batteries” (legacy
stdlib modules) have been removed from the standard library:
‘aifc’, ‘audioop’, ‘cgi’, ‘cgitb’, ‘chunk’, ‘crypt’, ‘imghdr’,
‘mailcap’, ‘msilib’, ‘nis’, ‘nntplib’, ‘ossaudiodev’, ‘pipes’,
‘sndhdr’, ‘spwd’, ‘sunau’, ‘telnetlib’, ‘uu’ and ‘xdrlib’.
* Remove the ‘2to3’ tool and ‘lib2to3’ module (deprecated in Python
3.11).
* Remove the ‘tkinter.tix’ module (deprecated in Python 3.6).
* Remove the ‘locale.resetlocale()’ function.
* Remove the ‘typing.io’ and ‘typing.re’ namespaces.
* Remove chained *note classmethod: 166. descriptors.
Release schedule changes:
PEP 602(19) (“Annual Release Cycle for Python”) has been updated to
extend the full support (‘bugfix’) period for new releases to two years.
This updated policy means that:
* Python 3.9–3.12 have one and a half years of full support, followed
by three and a half years of security fixes.
* Python 3.13 and later have two years of full support, followed by
three years of security fixes.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0703/
(2) https://peps.python.org/pep-0744/
(3) https://peps.python.org/pep-0594/
(4) https://peps.python.org/pep-0667/
(5) https://peps.python.org/pep-0703/
(6) https://peps.python.org/pep-0744/
(7) https://no-color.org/
(8) https://rfc.zeromq.org/spec/32/
(9) https://peps.python.org/pep-0669/
(10) https://peps.python.org/pep-0696/
(11) https://peps.python.org/pep-0702/
(12) https://peps.python.org/pep-0705/
(13) https://peps.python.org/pep-0742/
(14) https://peps.python.org/pep-0730/
(15) https://peps.python.org/pep-0011/#tier-3
(16) https://peps.python.org/pep-0738/
(17) https://peps.python.org/pep-0011/#tier-3
(18) https://peps.python.org/pep-0011/#tier-2
(19) https://peps.python.org/pep-0602/
File: python.info, Node: New Features, Next: Other Language Changes, Prev: Summary – Release Highlights, Up: What’s New In Python 3 13
1.1.2 New Features
------------------
* Menu:
* A better interactive interpreter::
* Improved error messages::
* Free-threaded CPython::
* An experimental just-in-time (JIT) compiler: An experimental just-in-time JIT compiler.
* Defined mutation semantics for locals(): Defined mutation semantics for locals.
* Support for mobile platforms::
File: python.info, Node: A better interactive interpreter, Next: Improved error messages, Up: New Features
1.1.2.1 A better interactive interpreter
........................................
Python now uses a new *note interactive: 169. shell by default, based on
code from the PyPy project(1). When the user starts the *note REPL:
16a. from an interactive terminal, the following new features are now
supported:
* Multiline editing with history preservation.
* Direct support for REPL-specific commands like ‘help’, ‘exit’, and
‘quit’, without the need to call them as functions.
* Prompts and tracebacks with *note color enabled by default: 16b.
* Interactive help browsing using ‘F1’ with a separate command
history.
* History browsing using ‘F2’ that skips output as well as the *note
>>>: 16c. and *note …: 16d. prompts.
* “Paste mode” with ‘F3’ that makes pasting larger blocks of code
easier (press ‘F3’ again to return to the regular prompt).
To disable the new interactive shell, set the *note PYTHON_BASIC_REPL:
16e. environment variable. For more on interactive mode, see *note
Interactive Mode: 16f.
(Contributed by Pablo Galindo Salgado, Łukasz Langa, and Lysandros
Nikolaou in gh-111201(2) based on code from the PyPy project. Windows
support contributed by Dino Viehland and Anthony Shaw.)
---------- Footnotes ----------
(1) https://pypy.org/
(2) https://github.com/python/cpython/issues/111201
File: python.info, Node: Improved error messages, Next: Free-threaded CPython, Prev: A better interactive interpreter, Up: New Features
1.1.2.2 Improved error messages
...............................
* The interpreter now uses color by default when displaying
tracebacks in the terminal. This feature *note can be controlled:
16b. via the new *note PYTHON_COLORS: 14a. environment variable as
well as the canonical NO_COLOR(1) and FORCE_COLOR(2) environment
variables. (Contributed by Pablo Galindo Salgado in gh-112730(3).)
* A common mistake is to write a script with the same name as a
standard library module. When this results in errors, we now
display a more helpful error message:
$ python random.py
Traceback (most recent call last):
File "/home/me/random.py", line 1, in
import random
File "/home/me/random.py", line 3, in
print(random.randint(5))
^^^^^^^^^^^^^^
AttributeError: module 'random' has no attribute 'randint' (consider renaming '/home/me/random.py' since it has the same name as the standard library module named 'random' and prevents importing that standard library module)
Similarly, if a script has the same name as a third-party module
that it attempts to import and this results in errors, we also
display a more helpful error message:
$ python numpy.py
Traceback (most recent call last):
File "/home/me/numpy.py", line 1, in
import numpy as np
File "/home/me/numpy.py", line 3, in
np.array([1, 2, 3])
^^^^^^^^
AttributeError: module 'numpy' has no attribute 'array' (consider renaming '/home/me/numpy.py' if it has the same name as a library you intended to import)
(Contributed by Shantanu Jain in gh-95754(4).)
* The error message now tries to suggest the correct keyword argument
when an incorrect keyword argument is passed to a function.
>>> "Better error messages!".split(max_split=1)
Traceback (most recent call last):
File "", line 1, in
"Better error messages!".split(max_split=1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
TypeError: split() got an unexpected keyword argument 'max_split'. Did you mean 'maxsplit'?
(Contributed by Pablo Galindo Salgado and Shantanu Jain in
gh-107944(5).)
---------- Footnotes ----------
(1) https://no-color.org/
(2) https://force-color.org/
(3) https://github.com/python/cpython/issues/112730
(4) https://github.com/python/cpython/issues/95754
(5) https://github.com/python/cpython/issues/107944
File: python.info, Node: Free-threaded CPython, Next: An experimental just-in-time JIT compiler, Prev: Improved error messages, Up: New Features
1.1.2.3 Free-threaded CPython
.............................
CPython now has experimental support for running in a free-threaded
mode, with the *note global interpreter lock: 148. (GIL) disabled. This
is an experimental feature and therefore is not enabled by default. The
free-threaded mode requires a different executable, usually called
‘python3.13t’ or ‘python3.13t.exe’. Pre-built binaries marked as
'free-threaded' can be installed as part of the official *note Windows:
172. and *note macOS: 173. installers, or CPython can be built from
source with the *note -disable-gil: 174. option.
Free-threaded execution allows for full utilization of the available
processing power by running threads in parallel on available CPU cores.
While not all software will benefit from this automatically, programs
designed with threading in mind will run faster on multi-core hardware.
'The free-threaded mode is experimental' and work is ongoing to improve
it: expect some bugs and a substantial single-threaded performance hit.
Free-threaded builds of CPython support optionally running with the GIL
enabled at runtime using the environment variable *note PYTHON_GIL: 175.
or the command-line option *note -X gil=1: 176.
To check if the current interpreter supports free-threading, *note
python -VV: 177. and *note sys.version: 178. contain “experimental
free-threading build”. The new ‘sys._is_gil_enabled()’ function can be
used to check whether the GIL is actually disabled in the running
process.
C-API extension modules need to be built specifically for the
free-threaded build. Extensions that support running with the *note
GIL: 159. disabled should use the *note Py_mod_gil: 158. slot.
Extensions using single-phase init should use *note
PyUnstable_Module_SetGIL(): 179. to indicate whether they support
running with the GIL disabled. Importing C extensions that don’t use
these mechanisms will cause the GIL to be enabled, unless the GIL was
explicitly disabled with the *note PYTHON_GIL: 175. environment variable
or the *note -X gil=0: 176. option. pip 24.1 or newer is required to
install packages with C extensions in the free-threaded build.
This work was made possible thanks to many individuals and
organizations, including the large community of contributors to Python
and third-party projects to test and enable free-threading support.
Notable contributors include: Sam Gross, Ken Jin, Donghee Na, Itamar
Oren, Matt Page, Brett Simmers, Dino Viehland, Carl Meyer, Nathan
Goldbaum, Ralf Gommers, Lysandros Nikolaou, and many others. Many of
these contributors are employed by Meta, which has provided significant
engineering resources to support this project.
See also
........
PEP 703(1) “Making the Global Interpreter Lock Optional in CPython”
contains rationale and information surrounding this work.
Porting Extension Modules to Support Free-Threading(2): A
community-maintained porting guide for extension authors.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0703/
(2) https://py-free-threading.github.io/porting/
File: python.info, Node: An experimental just-in-time JIT compiler, Next: Defined mutation semantics for locals, Prev: Free-threaded CPython, Up: New Features
1.1.2.4 An experimental just-in-time (JIT) compiler
...................................................
When CPython is configured and built using the
‘--enable-experimental-jit’ option, a just-in-time (JIT) compiler is
added which may speed up some Python programs. On Windows, use
‘PCbuild/build.bat --experimental-jit’ to enable the JIT or
‘--experimental-jit-interpreter’ to enable the Tier 2 interpreter.
Build requirements and further supporting information are contained
at(1) ‘Tools/jit/README.md’.
The ‘--enable-experimental-jit’ option takes these (optional) values,
defaulting to ‘yes’ if ‘--enable-experimental-jit’ is present without
the optional value.
* ‘no’: Disable the entire Tier 2 and JIT pipeline.
* ‘yes’: Enable the JIT. To disable the JIT at runtime, pass the
environment variable *note PYTHON_JIT=0: 17b.
* ‘yes-off’: Build the JIT but disable it by default. To enable the
JIT at runtime, pass the environment variable *note PYTHON_JIT=1:
17b.
* ‘interpreter’: Enable the Tier 2 interpreter but disable the JIT.
The interpreter can be disabled by running with *note PYTHON_JIT=0:
17b.
The internal architecture is roughly as follows:
* We start with specialized 'Tier 1 bytecode'. See *note What’s new
in 3.11: 17c. for details.
* When the Tier 1 bytecode gets hot enough, it gets translated to a
new purely internal intermediate representation (IR), called the
'Tier 2 IR', and sometimes referred to as micro-ops (“uops”).
* The Tier 2 IR uses the same stack-based virtual machine as Tier 1,
but the instruction format is better suited to translation to
machine code.
* We have several optimization passes for Tier 2 IR, which are
applied before it is interpreted or translated to machine code.
* There is a Tier 2 interpreter, but it is mostly intended for
debugging the earlier stages of the optimization pipeline. The
Tier 2 interpreter can be enabled by configuring Python with
‘--enable-experimental-jit=interpreter’.
* When the JIT is enabled, the optimized Tier 2 IR is translated to
machine code, which is then executed.
* The machine code translation process uses a technique called
'copy-and-patch'. It has no runtime dependencies, but there is a
new build-time dependency on LLVM.
See also
........
PEP 744(2)
(JIT by Brandt Bucher, inspired by a paper by Haoran Xu and Fredrik
Kjolstad. Tier 2 IR by Mark Shannon and Guido van Rossum. Tier 2
optimizer by Ken Jin.)
---------- Footnotes ----------
(1) https://github.com/python/cpython/blob/main/Tools/jit/README.md
(2) https://peps.python.org/pep-0744/
File: python.info, Node: Defined mutation semantics for locals, Next: Support for mobile platforms, Prev: An experimental just-in-time JIT compiler, Up: New Features
1.1.2.5 Defined mutation semantics for ‘locals()’
.................................................
Historically, the expected result of mutating the return value of *note
locals(): 141. has been left to individual Python implementations to
define. Starting from Python 3.13, PEP 667(1) standardises the
historical behavior of CPython for most code execution scopes, but
changes *note optimized scopes: 17e. (functions, generators, coroutines,
comprehensions, and generator expressions) to explicitly return
independent snapshots of the currently assigned local variables,
including locally referenced nonlocal variables captured in closures.
This change to the semantics of *note locals(): 141. in optimized scopes
also affects the default behavior of code execution functions that
implicitly target ‘locals()’ if no explicit namespace is provided (such
as *note exec(): 17f. and *note eval(): 180.). In previous versions,
whether or not changes could be accessed by calling ‘locals()’ after
calling the code execution function was implementation-dependent. In
CPython specifically, such code would typically appear to work as
desired, but could sometimes fail in optimized scopes based on other
code (including debuggers and code execution tracing tools) potentially
resetting the shared snapshot in that scope. Now, the code will always
run against an independent snapshot of the local variables in optimized
scopes, and hence the changes will never be visible in subsequent calls
to ‘locals()’. To access the changes made in these cases, an explicit
namespace reference must now be passed to the relevant function.
Alternatively, it may make sense to update affected code to use a higher
level code execution API that returns the resulting code execution
namespace (e.g. *note runpy.run_path(): 181. when executing Python
files from disk).
To ensure debuggers and similar tools can reliably update local
variables in scopes affected by this change, *note FrameType.f_locals:
182. now returns a write-through proxy to the frame’s local and locally
referenced nonlocal variables in these scopes, rather than returning an
inconsistently updated shared ‘dict’ instance with undefined runtime
semantics.
See PEP 667(2) for more details, including related C API changes and
deprecations. Porting notes are also provided below for the affected
*note Python APIs: 183. and *note C APIs: 184.
(PEP and implementation contributed by Mark Shannon and Tian Gao in
gh-74929(3). Documentation updates provided by Guido van Rossum and
Alyssa Coghlan.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0667/
(2) https://peps.python.org/pep-0667/
(3) https://github.com/python/cpython/issues/74929
File: python.info, Node: Support for mobile platforms, Prev: Defined mutation semantics for locals, Up: New Features
1.1.2.6 Support for mobile platforms
....................................
PEP 730(1): iOS is now a PEP 11(2) supported platform, with the
‘arm64-apple-ios’ and ‘arm64-apple-ios-simulator’ targets at tier 3
(iPhone and iPad devices released after 2013 and the Xcode iOS simulator
running on Apple silicon hardware, respectively).
‘x86_64-apple-ios-simulator’ (the Xcode iOS simulator running on older
‘x86_64’ hardware) is not a tier 3 supported platform, but will have
best-effort support. (PEP written and implementation contributed by
Russell Keith-Magee in gh-114099(3).)
PEP 738(4): Android is now a PEP 11(5) supported platform, with the
‘aarch64-linux-android’ and ‘x86_64-linux-android’ targets at tier 3.
The 32-bit targets ‘arm-linux-androideabi’ and ‘i686-linux-android’ are
not tier 3 supported platforms, but will have best-effort support. (PEP
written and implementation contributed by Malcolm Smith in
gh-116622(6).)
See also
........
PEP 730(7), PEP 738(8)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0730/
(2) https://peps.python.org/pep-0011/
(3) https://github.com/python/cpython/issues/114099
(4) https://peps.python.org/pep-0738/
(5) https://peps.python.org/pep-0011/
(6) https://github.com/python/cpython/issues/116622
(7) https://peps.python.org/pep-0730/
(8) https://peps.python.org/pep-0738/
File: python.info, Node: Other Language Changes, Next: New Modules, Prev: New Features, Up: What’s New In Python 3 13
1.1.3 Other Language Changes
----------------------------
* The compiler now strips common leading whitespace from every line
in a docstring. This reduces the size of the *note bytecode cache:
187. (such as ‘.pyc’ files), with reductions in file size of around
5%, for example in ‘sqlalchemy.orm.session’ from SQLAlchemy 2.0.
This change affects tools that use docstrings, such as *note
doctest: 3a.
>>> def spam():
... """
... This is a docstring with
... leading whitespace.
...
... It even has multiple paragraphs!
... """
...
>>> spam.__doc__
'\nThis is a docstring with\n leading whitespace.\n\nIt even has multiple paragraphs!\n'
(Contributed by Inada Naoki in gh-81283(1).)
* *note Annotation scopes: 188. within class scopes can now contain
lambdas and comprehensions. Comprehensions that are located within
class scopes are not inlined into their parent scope.
class C[T]:
type Alias = lambda: T
(Contributed by Jelle Zijlstra in gh-109118(2) and gh-118160(3).)
* *note Future statements: 189. are no longer triggered by relative
imports of the *note __future__: 0. module, meaning that statements
of the form ‘from .__future__ import ...’ are now simply standard
relative imports, with no special features activated. (Contributed
by Jeremiah Gabriel Pascual in gh-118216(4).)
* *note global: 18a. declarations are now permitted in *note except:
18b. blocks when that global is used in the *note else: 18c. block.
Previously this raised an erroneous *note SyntaxError: 18d.
(Contributed by Irit Katriel in gh-111123(5).)
* Add *note PYTHON_FROZEN_MODULES: 18e, a new environment variable
that determines whether frozen modules are ignored by the import
machinery, equivalent to the *note -X frozen_modules: 176.
command-line option. (Contributed by Yilei Yang in gh-111374(6).)
* Add *note support for the perf profiler: 18f. working without frame
pointers(7) through the new environment variable *note
PYTHON_PERF_JIT_SUPPORT: 190. and command-line option *note -X
perf_jit: 176. (Contributed by Pablo Galindo in gh-118518(8).)
* The location of a ‘.python_history’ file can be changed via the new
*note PYTHON_HISTORY: 191. environment variable. (Contributed by
Levi Sabah, Zackery Spytz and Hugo van Kemenade in gh-73965(9).)
* Classes have a new *note __static_attributes__: 14b. attribute.
This is populated by the compiler with a tuple of the class’s
attribute names which are assigned through ‘self.’ from any
function in its body. (Contributed by Irit Katriel in
gh-115775(10).)
* The compiler now creates a ‘__firstlineno__’ attribute on classes
with the line number of the first line of the class definition.
(Contributed by Serhiy Storchaka in gh-118465(11).)
* The *note exec(): 17f. and *note eval(): 180. builtins now accept
the 'globals' and 'locals' arguments as keywords. (Contributed by
Raphael Gaschignard in gh-105879(12))
* The *note compile(): 192. builtin now accepts a new flag,
‘ast.PyCF_OPTIMIZED_AST’, which is similar to ‘ast.PyCF_ONLY_AST’
except that the returned AST is optimized according to the value of
the 'optimize' argument. (Contributed by Irit Katriel in
gh-108113(13)).
* Add a *note __name__: 193. attribute on *note property: 194.
objects. (Contributed by Eugene Toder in gh-101860(14).)
* Add *note PythonFinalizationError: 14d, a new exception derived
from *note RuntimeError: 195. and used to signal when operations
are blocked during *note finalization: 14e. The following
callables now raise ‘PythonFinalizationError’, instead of *note
RuntimeError: 195.:
* *note _thread.start_new_thread(): 196.
* *note os.fork(): 197.
* *note os.forkpty(): 198.
* *note subprocess.Popen: 199.
(Contributed by Victor Stinner in gh-114570(15).)
* Allow the 'count' argument of *note str.replace(): 19a. to be a
keyword. (Contributed by Hugo van Kemenade in gh-106487(16).)
* Many functions now emit a warning if a boolean value is passed as a
file descriptor argument. This can help catch some errors earlier.
(Contributed by Serhiy Storchaka in gh-82626(17).)
* Added ‘name’ and ‘mode’ attributes for compressed and archived
file-like objects in the *note bz2: 13, *note lzma: 8a, *note
tarfile: de, and *note zipfile: 131. modules. (Contributed by
Serhiy Storchaka in gh-115961(18).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/81283
(2) https://github.com/python/cpython/issues/109118
(3) https://github.com/python/cpython/issues/118160
(4) https://github.com/python/cpython/issues/118216
(5) https://github.com/python/cpython/issues/111123
(6) https://github.com/python/cpython/issues/111374
(7) https://en.wikipedia.org/wiki/Call_stack
(8) https://github.com/python/cpython/issues/118518
(9) https://github.com/python/cpython/issues/73965
(10) https://github.com/python/cpython/issues/115775
(11) https://github.com/python/cpython/issues/118465
(12) https://github.com/python/cpython/issues/105879
(13) https://github.com/python/cpython/issues/108113
(14) https://github.com/python/cpython/issues/101860
(15) https://github.com/python/cpython/issues/114570
(16) https://github.com/python/cpython/issues/106487
(17) https://github.com/python/cpython/issues/82626
(18) https://github.com/python/cpython/issues/115961
File: python.info, Node: New Modules, Next: Improved Modules, Prev: Other Language Changes, Up: What’s New In Python 3 13
1.1.4 New Modules
-----------------
* *note dbm.sqlite3: 35.: An SQLite backend for *note dbm: 31.
(Contributed by Raymond Hettinger and Erlend E. Aasland in
gh-100414(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/100414
File: python.info, Node: Improved Modules, Next: Optimizations, Prev: New Modules, Up: What’s New In Python 3 13
1.1.5 Improved Modules
----------------------
* Menu:
* argparse::
* array::
* ast::
* asyncio::
* base64::
* compileall::
* concurrent.futures: concurrent futures.
* configparser::
* copy::
* ctypes::
* dbm::
* dis::
* doctest::
* email::
* enum::
* fractions::
* glob::
* importlib::
* io::
* ipaddress::
* itertools::
* marshal::
* math::
* mimetypes::
* mmap::
* multiprocessing::
* os::
* os.path: os path.
* pathlib::
* pdb::
* queue::
* random::
* re::
* shutil::
* site::
* sqlite3::
* ssl::
* statistics::
* subprocess::
* sys::
* tempfile::
* time::
* tkinter::
* traceback::
* types::
* typing::
* unicodedata::
* venv::
* warnings::
* xml::
* zipimport::
File: python.info, Node: argparse, Next: array, Up: Improved Modules
1.1.5.1 argparse
................
* Add the 'deprecated' parameter to the *note add_argument(): 19e.
and ‘add_parser()’ methods, to enable deprecating command-line
options, positional arguments, and subcommands. (Contributed by
Serhiy Storchaka in gh-83648(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/83648
File: python.info, Node: array, Next: ast, Prev: argparse, Up: Improved Modules
1.1.5.2 array
.............
* Add the ‘'w'’ type code (‘Py_UCS4’) for Unicode characters. It
should be used instead of the deprecated ‘'u'’ type code.
(Contributed by Inada Naoki in gh-80480(1).)
* Register *note array.array: 1a0. as a *note MutableSequence: 1a1.
by implementing the *note clear(): 1a2. method. (Contributed by
Mike Zimin in gh-114894(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/80480
(2) https://github.com/python/cpython/issues/114894
File: python.info, Node: ast, Next: asyncio, Prev: array, Up: Improved Modules
1.1.5.3 ast
...........
* The constructors of node types in the *note ast: 8. module are now
stricter in the arguments they accept, with more intuitive behavior
when arguments are omitted.
If an optional field on an AST node is not included as an argument
when constructing an instance, the field will now be set to ‘None’.
Similarly, if a list field is omitted, that field will now be set
to an empty list, and if an ‘expr_context’ field is omitted, it
defaults to *note Load(): 1a4. (Previously, in all cases, the
attribute would be missing on the newly constructed AST node
instance.)
In all other cases, where a required argument is omitted, the node
constructor will emit a *note DeprecationWarning: 1a5. This will
raise an exception in Python 3.15. Similarly, passing a keyword
argument to the constructor that does not map to a field on the AST
node is now deprecated, and will raise an exception in Python 3.15.
These changes do not apply to user-defined subclasses of *note
ast.AST: 1a6. unless the class opts in to the new behavior by
defining the *note AST._field_types: 1a7. mapping.
(Contributed by Jelle Zijlstra in gh-105858(1), gh-117486(2), and
gh-118851(3).)
* *note ast.parse(): 1a8. now accepts an optional argument 'optimize'
which is passed on to *note compile(): 192. This makes it possible
to obtain an optimized AST. (Contributed by Irit Katriel in
gh-108113(4).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/105858
(2) https://github.com/python/cpython/issues/117486
(3) https://github.com/python/cpython/issues/118851
(4) https://github.com/python/cpython/issues/108113
File: python.info, Node: asyncio, Next: base64, Prev: ast, Up: Improved Modules
1.1.5.4 asyncio
...............
* *note asyncio.as_completed(): 1aa. now returns an object that is
both an *note asynchronous iterator: 1ab. and a plain *note
iterator: 1ac. of *note awaitables: 1ad. The awaitables yielded by
asynchronous iteration include original task or future objects that
were passed in, making it easier to associate results with the
tasks being completed. (Contributed by Justin Arthur in
gh-77714(1).)
* *note asyncio.loop.create_unix_server(): 1ae. will now
automatically remove the Unix socket when the server is closed.
(Contributed by Pierre Ossman in gh-111246(2).)
* *note DatagramTransport.sendto(): 1af. will now send zero-length
datagrams if called with an empty bytes object. The transport flow
control also now accounts for the datagram header when calculating
the buffer size. (Contributed by Jamie Phan in gh-115199(3).)
* Add *note Queue.shutdown: 1b0. and *note QueueShutDown: 1b1. to
manage queue termination. (Contributed by Laurie Opperman and Yves
Duprat in gh-104228(4).)
* Add the *note Server.close_clients(): 1b2. and *note
Server.abort_clients(): 1b3. methods, which more forcefully close
an asyncio server. (Contributed by Pierre Ossman in gh-113538(5).)
* Accept a tuple of separators in *note StreamReader.readuntil():
1b4, stopping when any one of them is encountered. (Contributed by
Bruce Merry in gh-81322(6).)
* Improve the behavior of *note TaskGroup: 1b5. when an external
cancellation collides with an internal cancellation. For example,
when two task groups are nested and both experience an exception in
a child task simultaneously, it was possible that the outer task
group would hang, because its internal cancellation was swallowed
by the inner task group.
In the case where a task group is cancelled externally and also
must raise an *note ExceptionGroup: 1b6, it will now call the
parent task’s *note cancel(): 1b7. method. This ensures that a
*note CancelledError: 1b8. will be raised at the next *note await:
1b9, so the cancellation is not lost.
An added benefit of these changes is that task groups now preserve
the cancellation count (*note cancelling(): 1ba.).
In order to handle some corner cases, *note uncancel(): 1bb. may
now reset the undocumented ‘_must_cancel’ flag when the
cancellation count reaches zero.
(Inspired by an issue reported by Arthur Tacca in gh-116720(7).)
* When *note TaskGroup.create_task(): 1bc. is called on an inactive
*note TaskGroup: 1b5, the given coroutine will be closed (which
prevents a *note RuntimeWarning: 1bd. about the given coroutine
being never awaited). (Contributed by Arthur Tacca and Jason Zhang
in gh-115957(8).)
* The function and methods named ‘create_task’ have received a new
‘**kwargs’ argument that is passed through to the task constructor.
This change was accidentally added in 3.13.3, and broke the API
contract for custom task factories. Several third-party task
factories implemented workarounds for this. In 3.13.4 and later
releases the old factory contract is honored once again (until
3.14). To keep the workarounds working, the extra ‘**kwargs’
argument still allows passing additional keyword arguments to *note
Task: 1be. and to custom task factories.
This affects the following function and methods: *note
asyncio.create_task(): 1bf, *note asyncio.loop.create_task(): 1c0,
*note asyncio.TaskGroup.create_task(): 1bc. (Contributed by Thomas
Grainger in gh-128307(9).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/77714
(2) https://github.com/python/cpython/issues/111246
(3) https://github.com/python/cpython/issues/115199
(4) https://github.com/python/cpython/issues/104228
(5) https://github.com/python/cpython/issues/113538
(6) https://github.com/python/cpython/issues/81322
(7) https://github.com/python/cpython/issues/116720
(8) https://github.com/python/cpython/issues/115957
(9) https://github.com/python/cpython/issues/128307
File: python.info, Node: base64, Next: compileall, Prev: asyncio, Up: Improved Modules
1.1.5.5 base64
..............
* Add *note z85encode(): 14f. and *note z85decode(): 150. functions
for encoding *note bytes: 1c2. as Z85 data(1) and decoding
Z85-encoded data to ‘bytes’. (Contributed by Matan Perelman in
gh-75299(2).)
---------- Footnotes ----------
(1) https://rfc.zeromq.org/spec/32/
(2) https://github.com/python/cpython/issues/75299
File: python.info, Node: compileall, Next: concurrent futures, Prev: base64, Up: Improved Modules
1.1.5.6 compileall
..................
* The default number of worker threads and processes is now selected
using *note os.process_cpu_count(): 1c4. instead of *note
os.cpu_count(): 1c5. (Contributed by Victor Stinner in
gh-109649(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/109649
File: python.info, Node: concurrent futures, Next: configparser, Prev: compileall, Up: Improved Modules
1.1.5.7 concurrent.futures
..........................
* The default number of worker threads and processes is now selected
using *note os.process_cpu_count(): 1c4. instead of *note
os.cpu_count(): 1c5. (Contributed by Victor Stinner in
gh-109649(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/109649
File: python.info, Node: configparser, Next: copy, Prev: concurrent futures, Up: Improved Modules
1.1.5.8 configparser
....................
* *note ConfigParser: 1c8. now has support for unnamed sections,
which allows for top-level key-value pairs. This can be enabled
with the new 'allow_unnamed_section' parameter. (Contributed by
Pedro Sousa Lacerda in gh-66449(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/66449
File: python.info, Node: copy, Next: ctypes, Prev: configparser, Up: Improved Modules
1.1.5.9 copy
............
* The new *note replace(): 151. function and the *note replace
protocol: 152. make creating modified copies of objects much
simpler. This is especially useful when working with immutable
objects. The following types support the *note replace(): 151.
function and implement the replace protocol:
* *note collections.namedtuple(): 1ca.
* *note dataclasses.dataclass: 1cb.
* *note datetime.datetime: 1cc, *note datetime.date: 1cd, *note
datetime.time: 1ce.
* *note inspect.Signature: 1cf, *note inspect.Parameter: 1d0.
* *note types.SimpleNamespace: 1d1.
* *note code objects: 1d2.
Any user-defined class can also support *note copy.replace(): 151.
by defining the *note __replace__(): 152. method. (Contributed by
Serhiy Storchaka in gh-108751(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/108751
File: python.info, Node: ctypes, Next: dbm, Prev: copy, Up: Improved Modules
1.1.5.10 ctypes
...............
* As a consequence of necessary internal refactoring, initialization
of internal metaclasses now happens in ‘__init__’ rather than in
‘__new__’. This affects projects that subclass these internal
metaclasses to provide custom initialization. Generally:
- Custom logic that was done in ‘__new__’ after calling
‘super().__new__’ should be moved to ‘__init__’.
- To create a class, call the metaclass, not only the
metaclass’s ‘__new__’ method.
See gh-124520(1) for discussion and links to changes in some
affected projects.
* *note ctypes.Structure: 1d4. objects have a new *note _align_: 1d5.
attribute which allows the alignment of the structure being packed
to/from memory to be specified explicitly. (Contributed by Matt
Sanderson in gh-112433(2))
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/124520
(2) https://github.com/python/cpython/issues/112433
File: python.info, Node: dbm, Next: dis, Prev: ctypes, Up: Improved Modules
1.1.5.11 dbm
............
* Add *note dbm.sqlite3: 35, a new module which implements an SQLite
backend, and make it the default ‘dbm’ backend. (Contributed by
Raymond Hettinger and Erlend E. Aasland in gh-100414(1).)
* Allow removing all items from the database through the new *note
gdbm.clear(): 1d7. and *note ndbm.clear(): 1d8. methods.
(Contributed by Donghee Na in gh-107122(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/100414
(2) https://github.com/python/cpython/issues/107122
File: python.info, Node: dis, Next: doctest, Prev: dbm, Up: Improved Modules
1.1.5.12 dis
............
* Change the output of *note dis: 38. module functions to show
logical labels for jump targets and exception handlers, rather than
offsets. The offsets can be added with the new *note -O: 1da.
command-line option or the 'show_offsets' argument. (Contributed
by Irit Katriel in gh-112137(1).)
* *note get_instructions(): 1db. no longer represents cache entries
as separate instructions. Instead, it returns them as part of the
*note Instruction: 1dc, in the new 'cache_info' field. The
'show_caches' argument to *note get_instructions(): 1db. is
deprecated and no longer has any effect. (Contributed by Irit
Katriel in gh-112962(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/112137
(2) https://github.com/python/cpython/issues/112962
File: python.info, Node: doctest, Next: email, Prev: dis, Up: Improved Modules
1.1.5.13 doctest
................
* *note doctest: 3a. output is now colored by default. This can be
controlled via the new *note PYTHON_COLORS: 14a. environment
variable as well as the canonical NO_COLOR(1) and FORCE_COLOR(2)
environment variables. See also *note Controlling color: 16b.
(Contributed by Hugo van Kemenade in gh-117225(3).)
* The *note DocTestRunner.run(): 1de. method now counts the number of
skipped tests. Add the *note DocTestRunner.skips: 1df. and *note
TestResults.skipped: 1e0. attributes. (Contributed by Victor
Stinner in gh-108794(4).)
---------- Footnotes ----------
(1) https://no-color.org/
(2) https://force-color.org/
(3) https://github.com/python/cpython/issues/117225
(4) https://github.com/python/cpython/issues/108794
File: python.info, Node: email, Next: enum, Prev: doctest, Up: Improved Modules
1.1.5.14 email
..............
* Headers with embedded newlines are now quoted on output. The *note
generator: 40. will now refuse to serialize (write) headers that
are improperly folded or delimited, such that they would be parsed
as multiple headers or joined with adjacent data. If you need to
turn this safety feature off, set *note verify_generated_headers:
1e2. (Contributed by Bas Bloemsaat and Petr Viktorin in
gh-121650(1).)
* *note getaddresses(): 1e3. and *note parseaddr(): 1e4. now return
‘('', '')’ pairs in more situations where invalid email addresses
are encountered instead of potentially inaccurate values. The two
functions have a new optional 'strict' parameter (default ‘True’).
To get the old behavior (accepting malformed input), use
‘strict=False’. ‘getattr(email.utils, 'supports_strict_parsing',
False)’ can be used to check if the 'strict' parameter is
available. (Contributed by Thomas Dwyer and Victor Stinner for
gh-102988(2) to improve the CVE 2023-27043(3) fix.)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/121650
(2) https://github.com/python/cpython/issues/102988
(3) https://www.cve.org/CVERecord?id=CVE-2023-27043
File: python.info, Node: enum, Next: fractions, Prev: email, Up: Improved Modules
1.1.5.15 enum
.............
* *note EnumDict: 1e6. has been made public to better support
subclassing *note EnumType: 1e7.
File: python.info, Node: fractions, Next: glob, Prev: enum, Up: Improved Modules
1.1.5.16 fractions
..................
* *note Fraction: 1e9. objects now support the standard *note format
specification mini-language: 1ea. rules for fill, alignment, sign
handling, minimum width, and grouping. (Contributed by Mark
Dickinson in gh-111320(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/111320
File: python.info, Node: glob, Next: importlib, Prev: fractions, Up: Improved Modules
1.1.5.17 glob
.............
* Add *note translate(): 1ec, a function to convert a path
specification with shell-style wildcards to a regular expression.
(Contributed by Barney Gale in gh-72904(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/72904
File: python.info, Node: importlib, Next: io, Prev: glob, Up: Improved Modules
1.1.5.18 importlib
..................
* The following functions in *note importlib.resources: 7b. now allow
accessing a directory (or tree) of resources, using multiple
positional arguments (the 'encoding' and 'errors' arguments in the
text-reading functions are now keyword-only):
* *note is_resource(): 1ee.
* *note open_binary(): 1ef.
* *note open_text(): 1f0.
* *note path(): 1f1.
* *note read_binary(): 1f2.
* *note read_text(): 1f3.
These functions are no longer deprecated and are not scheduled for
removal. (Contributed by Petr Viktorin in gh-116608(1).)
* *note contents(): 1f4. remains deprecated in favor of the
fully-featured *note Traversable: 1f5. API. However, there is now
no plan to remove it. (Contributed by Petr Viktorin in
gh-116608(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/116608
(2) https://github.com/python/cpython/issues/116608
File: python.info, Node: io, Next: ipaddress, Prev: importlib, Up: Improved Modules
1.1.5.19 io
...........
* The *note IOBase: 1f7. finalizer now logs any errors raised by the
*note close(): 1f8. method with *note sys.unraisablehook: 1f9.
Previously, errors were ignored silently by default, and only
logged in *note Python Development Mode: 1fa. or when using a *note
Python debug build: 1fb. (Contributed by Victor Stinner in
gh-62948(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/62948
File: python.info, Node: ipaddress, Next: itertools, Prev: io, Up: Improved Modules
1.1.5.20 ipaddress
..................
* Add the *note IPv4Address.ipv6_mapped: 1fd. property, which returns
the IPv4-mapped IPv6 address. (Contributed by Charles Machalow in
gh-109466(1).)
* Fix ‘is_global’ and ‘is_private’ behavior in *note IPv4Address:
1fe, *note IPv6Address: 1ff, *note IPv4Network: 200, and *note
IPv6Network: 201. (Contributed by Jakub Stasiak in gh-113171(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/109466
(2) https://github.com/python/cpython/issues/113171
File: python.info, Node: itertools, Next: marshal, Prev: ipaddress, Up: Improved Modules
1.1.5.21 itertools
..................
* *note batched(): 203. has a new 'strict' parameter, which raises a
*note ValueError: 204. if the final batch is shorter than the
specified batch size. (Contributed by Raymond Hettinger in
gh-113202(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/113202
File: python.info, Node: marshal, Next: math, Prev: itertools, Up: Improved Modules
1.1.5.22 marshal
................
* Add the 'allow_code' parameter in module functions. Passing
‘allow_code=False’ prevents serialization and de-serialization of
code objects which are incompatible between Python versions.
(Contributed by Serhiy Storchaka in gh-113626(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/113626
File: python.info, Node: math, Next: mimetypes, Prev: marshal, Up: Improved Modules
1.1.5.23 math
.............
* The new function *note fma(): 207. performs fused multiply-add
operations. This computes ‘x * y + z’ with only a single round,
and so avoids any intermediate loss of precision. It wraps the
‘fma()’ function provided by C99, and follows the specification of
the IEEE 754 “fusedMultiplyAdd” operation for special cases.
(Contributed by Mark Dickinson and Victor Stinner in gh-73468(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/73468
File: python.info, Node: mimetypes, Next: mmap, Prev: math, Up: Improved Modules
1.1.5.24 mimetypes
..................
* Add the *note guess_file_type(): 209. function to guess a MIME type
from a filesystem path. Using paths with *note guess_type(): 20a.
is now *note soft deprecated: 20b. (Contributed by Serhiy
Storchaka in gh-66543(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/66543
File: python.info, Node: mmap, Next: multiprocessing, Prev: mimetypes, Up: Improved Modules
1.1.5.25 mmap
.............
* *note mmap: 20d. is now protected from crashing on Windows when the
mapped memory is inaccessible due to file system errors or access
violations. (Contributed by Jannis Weigend in gh-118209(1).)
* *note mmap: 20d. has a new *note seekable(): 20e. method that can
be used when a seekable file-like object is required. The *note
seek(): 20f. method now returns the new absolute position.
(Contributed by Donghee Na and Sylvie Liberman in gh-111835(2).)
* The new UNIX-only 'trackfd' parameter for *note mmap: 20d. controls
file descriptor duplication; if false, the file descriptor
specified by 'fileno' will not be duplicated. (Contributed by
Zackery Spytz and Petr Viktorin in gh-78502(3).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/118209
(2) https://github.com/python/cpython/issues/111835
(3) https://github.com/python/cpython/issues/78502
File: python.info, Node: multiprocessing, Next: os, Prev: mmap, Up: Improved Modules
1.1.5.26 multiprocessing
........................
* The default number of worker threads and processes is now selected
using *note os.process_cpu_count(): 1c4. instead of *note
os.cpu_count(): 1c5. (Contributed by Victor Stinner in
gh-109649(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/109649
File: python.info, Node: os, Next: os path, Prev: multiprocessing, Up: Improved Modules
1.1.5.27 os
...........
* Add *note process_cpu_count(): 1c4. function to get the number of
logical CPU cores usable by the calling thread of the current
process. (Contributed by Victor Stinner in gh-109649(1).)
* *note cpu_count(): 1c5. and *note process_cpu_count(): 1c4. can be
overridden through the new environment variable *note
PYTHON_CPU_COUNT: 212. or the new command-line option *note -X
cpu_count: 176. This option is useful for users who need to limit
CPU resources of a container system without having to modify
application code or the container itself. (Contributed by Donghee
Na in gh-109595(2).)
* Add a *note low level interface: 153. to Linux’s ‘timer file
descriptors(3)’ via *note timerfd_create(): 213, *note
timerfd_settime(): 214, *note timerfd_settime_ns(): 215, *note
timerfd_gettime(): 216, *note timerfd_gettime_ns(): 217, *note
TFD_NONBLOCK: 218, *note TFD_CLOEXEC: 219, *note TFD_TIMER_ABSTIME:
21a, and *note TFD_TIMER_CANCEL_ON_SET: 21b. (Contributed by Masaru
Tsuchiyama in gh-108277(4).)
* *note lchmod(): 21c. and the 'follow_symlinks' argument of *note
chmod(): 21d. are both now available on Windows. Note that the
default value of 'follow_symlinks' in ‘lchmod()’ is ‘False’ on
Windows. (Contributed by Serhiy Storchaka in gh-59616(5).)
* *note fchmod(): 21e. and support for file descriptors in *note
chmod(): 21d. are both now available on Windows. (Contributed by
Serhiy Storchaka in gh-113191(6).)
* On Windows, *note mkdir(): 21f. and *note makedirs(): 220. now
support passing a 'mode' value of ‘0o700’ to apply access control
to the new directory. This implicitly affects *note
tempfile.mkdtemp(): 221. and is a mitigation for CVE 2024-4030(7).
Other values for 'mode' continue to be ignored. (Contributed by
Steve Dower in gh-118486(8).)
* *note posix_spawn(): 222. now accepts ‘None’ for the 'env'
argument, which makes the newly spawned process use the current
process environment. (Contributed by Jakub Kulik in gh-113119(9).)
* *note posix_spawn(): 222. can now use the *note
POSIX_SPAWN_CLOSEFROM: 223. attribute in the 'file_actions'
parameter on platforms that support
‘posix_spawn_file_actions_addclosefrom_np()’. (Contributed by
Jakub Kulik in gh-113117(10).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/109649
(2) https://github.com/python/cpython/issues/109595
(3) https://manpages.debian.org/timerfd_create(2)
(4) https://github.com/python/cpython/issues/108277
(5) https://github.com/python/cpython/issues/59616
(6) https://github.com/python/cpython/issues/113191
(7) https://www.cve.org/CVERecord?id=CVE-2024-4030
(8) https://github.com/python/cpython/issues/118486
(9) https://github.com/python/cpython/issues/113119
(10) https://github.com/python/cpython/issues/113117
File: python.info, Node: os path, Next: pathlib, Prev: os, Up: Improved Modules
1.1.5.28 os.path
................
* Add *note isreserved(): 225. to check if a path is reserved on the
current system. This function is only available on Windows.
(Contributed by Barney Gale in gh-88569(1).)
* On Windows, *note isabs(): 226. no longer considers paths starting
with exactly one slash (‘\’ or ‘/’) to be absolute. (Contributed
by Barney Gale and Jon Foster in gh-44626(2).)
* *note realpath(): 227. now resolves MS-DOS style file names even if
the file is not accessible. (Contributed by Moonsik Park in
gh-82367(3).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/88569
(2) https://github.com/python/cpython/issues/44626
(3) https://github.com/python/cpython/issues/82367
File: python.info, Node: pathlib, Next: pdb, Prev: os path, Up: Improved Modules
1.1.5.29 pathlib
................
* Add *note UnsupportedOperation: 229, which is raised instead of
*note NotImplementedError: 22a. when a path operation isn’t
supported. (Contributed by Barney Gale in gh-89812(1).)
* Add a new constructor for creating *note Path: 22b. objects from
‘file’ URIs (‘file:///’), *note Path.from_uri(): 22c. (Contributed
by Barney Gale in gh-107465(2).)
* Add *note PurePath.full_match(): 22d. for matching paths with
shell-style wildcards, including the recursive wildcard “‘**’”.
(Contributed by Barney Gale in gh-73435(3).)
* Add the *note PurePath.parser: 22e. class attribute to store the
implementation of *note os.path: a2. used for low-level path
parsing and joining. This will be either ‘posixpath’ or ‘ntpath’.
* Add 'recurse_symlinks' keyword-only argument to *note Path.glob():
22f. and *note rglob(): 230. (Contributed by Barney Gale in
gh-77609(4).)
* *note Path.glob(): 22f. and *note rglob(): 230. now return files
and directories when given a pattern that ends with “‘**’”.
Previously, only directories were returned. (Contributed by Barney
Gale in gh-70303(5).)
* Add the 'follow_symlinks' keyword-only argument to *note
Path.is_file: 231, *note Path.is_dir: 232, *note Path.owner(): 233,
and *note Path.group(): 234. (Contributed by Barney Gale in
gh-105793(6) and Kamil Turek in gh-107962(7).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/89812
(2) https://github.com/python/cpython/issues/107465
(3) https://github.com/python/cpython/issues/73435
(4) https://github.com/python/cpython/issues/77609
(5) https://github.com/python/cpython/issues/70303
(6) https://github.com/python/cpython/issues/105793
(7) https://github.com/python/cpython/issues/107962
File: python.info, Node: pdb, Next: queue, Prev: pathlib, Up: Improved Modules
1.1.5.30 pdb
............
* *note breakpoint(): 236. and *note set_trace(): 237. now enter the
debugger immediately rather than on the next line of code to be
executed. This change prevents the debugger from breaking outside
of the context when ‘breakpoint()’ is positioned at the end of the
context. (Contributed by Tian Gao in gh-118579(1).)
* ‘sys.path[0]’ is no longer replaced by the directory of the script
being debugged when *note sys.flags.safe_path: 238. is set.
(Contributed by Tian Gao and Christian Walther in gh-111762(2).)
* *note zipapp: 130. is now supported as a debugging target.
(Contributed by Tian Gao in gh-118501(3).)
* Add ability to move between chained exceptions during post-mortem
debugging in *note pm(): 239. using the new *note exceptions
[exc_number]: 23a. command for Pdb. (Contributed by Matthias
Bussonnier in gh-106676(4).)
* Expressions and statements whose prefix is a pdb command are now
correctly identified and executed. (Contributed by Tian Gao in
gh-108464(5).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/118579
(2) https://github.com/python/cpython/issues/111762
(3) https://github.com/python/cpython/issues/118501
(4) https://github.com/python/cpython/issues/106676
(5) https://github.com/python/cpython/issues/108464
File: python.info, Node: queue, Next: random, Prev: pdb, Up: Improved Modules
1.1.5.31 queue
..............
* Add *note Queue.shutdown: 23c. and *note ShutDown: 23d. to manage
queue termination. (Contributed by Laurie Opperman and Yves Duprat
in gh-104750(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/104750
File: python.info, Node: random, Next: re, Prev: queue, Up: Improved Modules
1.1.5.32 random
...............
* Add a *note command-line interface: 154. (Contributed by Hugo van
Kemenade in gh-118131(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/118131
File: python.info, Node: re, Next: shutil, Prev: random, Up: Improved Modules
1.1.5.33 re
...........
* Rename ‘re.error’ to *note PatternError: 240. for improved clarity.
‘re.error’ is kept for backward compatibility.
File: python.info, Node: shutil, Next: site, Prev: re, Up: Improved Modules
1.1.5.34 shutil
...............
* Support the 'dir_fd' and 'follow_symlinks' keyword arguments in
*note chown(): 242. (Contributed by Berker Peksag and Tahia K in
gh-62308(1))
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/62308
File: python.info, Node: site, Next: sqlite3, Prev: shutil, Up: Improved Modules
1.1.5.35 site
.............
* ‘.pth’ files are now decoded using UTF-8 first, and then with the
*note locale encoding: 244. if UTF-8 decoding fails. (Contributed
by Inada Naoki in gh-117802(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/117802
File: python.info, Node: sqlite3, Next: ssl, Prev: site, Up: Improved Modules
1.1.5.36 sqlite3
................
* A *note ResourceWarning: 246. is now emitted if a *note Connection:
247. object is not *note closed: 248. explicitly. (Contributed by
Erlend E. Aasland in gh-105539(1).)
* Add the 'filter' keyword-only parameter to *note
Connection.iterdump(): 249. for filtering database objects to dump.
(Contributed by Mariusz Felisiak in gh-91602(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/105539
(2) https://github.com/python/cpython/issues/91602
File: python.info, Node: ssl, Next: statistics, Prev: sqlite3, Up: Improved Modules
1.1.5.37 ssl
............
* The *note create_default_context(): 155. API now includes *note
VERIFY_X509_PARTIAL_CHAIN: 156. and *note VERIFY_X509_STRICT: 157.
in its default flags.
Note: *note VERIFY_X509_STRICT: 157. may reject pre- RFC
5280(1) or malformed certificates that the underlying OpenSSL
implementation might otherwise accept. Whilst disabling this
is not recommended, you can do so using:
import ssl
ctx = ssl.create_default_context()
ctx.verify_flags &= ~ssl.VERIFY_X509_STRICT
(Contributed by William Woodruff in gh-112389(2).)
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc5280.html
(2) https://github.com/python/cpython/issues/112389
File: python.info, Node: statistics, Next: subprocess, Prev: ssl, Up: Improved Modules
1.1.5.38 statistics
...................
* Add *note kde(): 24c. for kernel density estimation. This makes it
possible to estimate a continuous probability density function from
a fixed number of discrete samples. (Contributed by Raymond
Hettinger in gh-115863(1).)
* Add *note kde_random(): 24d. for sampling from an estimated
probability density function created by *note kde(): 24c.
(Contributed by Raymond Hettinger in gh-115863(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/115863
(2) https://github.com/python/cpython/issues/115863
File: python.info, Node: subprocess, Next: sys, Prev: statistics, Up: Improved Modules
1.1.5.39 subprocess
...................
* The *note subprocess: d6. module now uses the *note posix_spawn():
222. function in more situations.
Notably, when 'close_fds' is ‘True’ (the default), *note
posix_spawn(): 222. will be used when the C library provides
‘posix_spawn_file_actions_addclosefrom_np()’, which includes recent
versions of Linux, FreeBSD, and Solaris. On Linux, this should
perform similarly to the existing Linux ‘vfork()’ based code.
A private control knob ‘subprocess._USE_POSIX_SPAWN’ can be set to
‘False’ if you need to force *note subprocess: d6. to never use
*note posix_spawn(): 222. Please report your reason and platform
details in the *note issue tracker: 250. if you set this so that we
can improve our API selection logic for everyone. (Contributed by
Jakub Kulik in gh-113117(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/113117
File: python.info, Node: sys, Next: tempfile, Prev: subprocess, Up: Improved Modules
1.1.5.40 sys
............
* Add the *note _is_interned(): 252. function to test if a string was
interned. This function is not guaranteed to exist in all
implementations of Python. (Contributed by Serhiy Storchaka in
gh-78573(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/78573
File: python.info, Node: tempfile, Next: time, Prev: sys, Up: Improved Modules
1.1.5.41 tempfile
.................
* On Windows, the default mode ‘0o700’ used by *note
tempfile.mkdtemp(): 221. now limits access to the new directory due
to changes to *note os.mkdir(): 21f. This is a mitigation for CVE
2024-4030(1). (Contributed by Steve Dower in gh-118486(2).)
---------- Footnotes ----------
(1) https://www.cve.org/CVERecord?id=CVE-2024-4030
(2) https://github.com/python/cpython/issues/118486
File: python.info, Node: time, Next: tkinter, Prev: tempfile, Up: Improved Modules
1.1.5.42 time
.............
* On Windows, *note monotonic(): 255. now uses the
‘QueryPerformanceCounter()’ clock for a resolution of 1
microsecond, instead of the ‘GetTickCount64()’ clock which has a
resolution of 15.6 milliseconds. (Contributed by Victor Stinner in
gh-88494(1).)
* On Windows, *note time(): 256. now uses the
‘GetSystemTimePreciseAsFileTime()’ clock for a resolution of 1
microsecond, instead of the ‘GetSystemTimeAsFileTime()’ clock which
has a resolution of 15.6 milliseconds. (Contributed by Victor
Stinner in gh-63207(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/88494
(2) https://github.com/python/cpython/issues/63207
File: python.info, Node: tkinter, Next: traceback, Prev: time, Up: Improved Modules
1.1.5.43 tkinter
................
* Add *note tkinter: f0. widget methods: ‘tk_busy_hold()’,
‘tk_busy_configure()’, ‘tk_busy_cget()’, ‘tk_busy_forget()’,
‘tk_busy_current()’, and ‘tk_busy_status()’. (Contributed by
Miguel, klappnase and Serhiy Storchaka in gh-72684(1).)
* The *note tkinter: f0. widget method ‘wm_attributes()’ now accepts
the attribute name without the minus prefix to get window
attributes, for example ‘w.wm_attributes('alpha')’ and allows
specifying attributes and values to set as keyword arguments, for
example ‘w.wm_attributes(alpha=0.5)’. (Contributed by Serhiy
Storchaka in gh-43457(2).)
* ‘wm_attributes()’ can now return attributes as a *note dict: 258,
by using the new optional keyword-only parameter
'return_python_dict'. (Contributed by Serhiy Storchaka in
gh-43457(3).)
* ‘Text.count()’ can now return a simple *note int: 259. when the new
optional keyword-only parameter 'return_ints' is used. Otherwise,
the single count is returned as a 1-tuple or ‘None’. (Contributed
by Serhiy Storchaka in gh-97928(4).)
* Support the “vsapi” element type in the *note element_create():
25a. method of *note tkinter.ttk.Style: 25b. (Contributed by
Serhiy Storchaka in gh-68166(5).)
* Add the ‘after_info()’ method for Tkinter widgets. (Contributed by
Cheryl Sabella in gh-77020(6).)
* Add a new ‘copy_replace()’ method to ‘PhotoImage’ to copy a region
from one image to another, possibly with pixel zooming,
subsampling, or both. (Contributed by Serhiy Storchaka in
gh-118225(7).)
* Add 'from_coords' parameter to the ‘PhotoImage’ methods ‘copy()’,
‘zoom()’ and ‘subsample()’. Add 'zoom' and 'subsample' parameters
to the ‘PhotoImage’ method ‘copy()’. (Contributed by Serhiy
Storchaka in gh-118225(8).)
* Add the ‘PhotoImage’ methods ‘read()’ to read an image from a file
and ‘data()’ to get the image data. Add 'background' and
'grayscale' parameters to the ‘write()’ method. (Contributed by
Serhiy Storchaka in gh-118271(9).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/72684
(2) https://github.com/python/cpython/issues/43457
(3) https://github.com/python/cpython/issues/43457
(4) https://github.com/python/cpython/issues/97928
(5) https://github.com/python/cpython/issues/68166
(6) https://github.com/python/cpython/issues/77020
(7) https://github.com/python/cpython/issues/118225
(8) https://github.com/python/cpython/issues/118225
(9) https://github.com/python/cpython/issues/118271
File: python.info, Node: traceback, Next: types, Prev: tkinter, Up: Improved Modules
1.1.5.44 traceback
..................
* Add the *note exc_type_str: 25d. attribute to *note
TracebackException: 25e, which holds a string display of the
'exc_type'. Deprecate the *note exc_type: 25f. attribute, which
holds the type object itself. Add parameter 'save_exc_type'
(default ‘True’) to indicate whether ‘exc_type’ should be saved.
(Contributed by Irit Katriel in gh-112332(1).)
* Add a new 'show_group' keyword-only parameter to *note
TracebackException.format_exception_only(): 260. to (recursively)
format the nested exceptions of a *note BaseExceptionGroup: 261.
instance. (Contributed by Irit Katriel in gh-105292(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/112332
(2) https://github.com/python/cpython/issues/105292
File: python.info, Node: types, Next: typing, Prev: traceback, Up: Improved Modules
1.1.5.45 types
..............
* *note SimpleNamespace: 1d1. can now take a single positional
argument to initialise the namespace’s arguments. This argument
must either be a mapping or an iterable of key-value pairs.
(Contributed by Serhiy Storchaka in gh-108191(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/108191
File: python.info, Node: typing, Next: unicodedata, Prev: types, Up: Improved Modules
1.1.5.46 typing
...............
* PEP 705(1): Add *note ReadOnly: 161, a special typing construct to
mark a *note TypedDict: 162. item as read-only for type checkers.
* PEP 742(2): Add *note TypeIs: 163, a typing construct that can be
used to instruct a type checker how to narrow a type.
* Add *note NoDefault: 264, a sentinel object used to represent the
defaults of some parameters in the *note typing: 104. module.
(Contributed by Jelle Zijlstra in gh-116126(3).)
* Add *note get_protocol_members(): 265. to return the set of members
defining a *note typing.Protocol: 266. (Contributed by Jelle
Zijlstra in gh-104873(4).)
* Add *note is_protocol(): 267. to check whether a class is a *note
Protocol: 266. (Contributed by Jelle Zijlstra in gh-104873(5).)
* *note ClassVar: 268. can now be nested in *note Final: 269, and
vice versa. (Contributed by Mehdi Drissi in gh-89547(6).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0705/
(2) https://peps.python.org/pep-0742/
(3) https://github.com/python/cpython/issues/116126
(4) https://github.com/python/cpython/issues/104873
(5) https://github.com/python/cpython/issues/104873
(6) https://github.com/python/cpython/issues/89547
File: python.info, Node: unicodedata, Next: venv, Prev: typing, Up: Improved Modules
1.1.5.47 unicodedata
....................
* Update the Unicode database to version 15.1.0(1). (Contributed by
James Gerity in gh-109559(2).)
---------- Footnotes ----------
(1) https://www.unicode.org/versions/Unicode15.1.0/
(2) https://github.com/python/cpython/issues/109559
File: python.info, Node: venv, Next: warnings, Prev: unicodedata, Up: Improved Modules
1.1.5.48 venv
.............
* Add support for creating source control management (SCM) ignore
files in a virtual environment’s directory. By default, Git is
supported. This is implemented as opt-in via the API, which can be
extended to support other SCMs (*note EnvBuilder: 26c. and *note
create(): 26d.), and opt-out via the CLI, using
‘--without-scm-ignore-files’. (Contributed by Brett Cannon in
gh-108125(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/108125
File: python.info, Node: warnings, Next: xml, Prev: venv, Up: Improved Modules
1.1.5.49 warnings
.................
* PEP 702(1): The new *note warnings.deprecated(): 160. decorator
provides a way to communicate deprecations to a *note static type
checker: 26f. and to warn on usage of deprecated classes and
functions. A *note DeprecationWarning: 1a5. may also be emitted
when a decorated function or class is used at runtime.
(Contributed by Jelle Zijlstra in gh-104003(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0702/
(2) https://github.com/python/cpython/issues/104003
File: python.info, Node: xml, Next: zipimport, Prev: warnings, Up: Improved Modules
1.1.5.50 xml
............
* Allow controlling Expat >=2.6.0 reparse deferral ( CVE
2023-52425(1)) by adding five new methods:
* *note xml.etree.ElementTree.XMLParser.flush(): 271.
* *note xml.etree.ElementTree.XMLPullParser.flush(): 272.
* *note xml.parsers.expat.xmlparser.GetReparseDeferralEnabled():
273.
* *note xml.parsers.expat.xmlparser.SetReparseDeferralEnabled():
274.
* ‘xml.sax.expatreader.ExpatParser.flush()’
(Contributed by Sebastian Pipping in gh-115623(2).)
* Add the ‘close()’ method for the iterator returned by *note
iterparse(): 275. for explicit cleanup. (Contributed by Serhiy
Storchaka in gh-69893(3).)
---------- Footnotes ----------
(1) https://www.cve.org/CVERecord?id=CVE-2023-52425
(2) https://github.com/python/cpython/issues/115623
(3) https://github.com/python/cpython/issues/69893
File: python.info, Node: zipimport, Prev: xml, Up: Improved Modules
1.1.5.51 zipimport
..................
* Add support for ZIP64(1) format files. Everybody loves huge data,
right? (Contributed by Tim Hatch in gh-94146(2).)
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64
(2) https://github.com/python/cpython/issues/94146
File: python.info, Node: Optimizations, Next: Removed Modules And APIs, Prev: Improved Modules, Up: What’s New In Python 3 13
1.1.6 Optimizations
-------------------
* Several standard library modules have had their import times
significantly improved. For example, the import time of the *note
typing: 104. module has been reduced by around a third by removing
dependencies on *note re: b9. and *note contextlib: 23. Other
modules to enjoy import-time speedups include *note email.utils:
50, *note enum: 56, *note functools: 5f, *note importlib.metadata:
7a, and *note threading: ed. (Contributed by Alex Waygood,
Shantanu Jain, Adam Turner, Daniel Hollas, and others in
gh-109653(1).)
* *note textwrap.indent(): 278. is now around 30% faster than before
for large input. (Contributed by Inada Naoki in gh-107369(2).)
* The *note subprocess: d6. module now uses the *note posix_spawn():
222. function in more situations, including when 'close_fds' is
‘True’ (the default) on many modern platforms. This should provide
a notable performance increase when launching processes on FreeBSD
and Solaris. See the *note subprocess: 24f. section above for
details. (Contributed by Jakub Kulik in gh-113117(3).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/109653
(2) https://github.com/python/cpython/issues/107369
(3) https://github.com/python/cpython/issues/113117
File: python.info, Node: Removed Modules And APIs, Next: New Deprecations, Prev: Optimizations, Up: What’s New In Python 3 13
1.1.7 Removed Modules And APIs
------------------------------
* Menu:
* PEP 594; Remove “dead batteries” from the standard library: PEP 594 Remove “dead batteries” from the standard library.
* 2to3::
* builtins::
* configparser: configparser<2>.
* importlib.metadata: importlib metadata.
* locale::
* opcode::
* optparse::
* pathlib: pathlib<2>.
* re: re<2>.
* tkinter.tix: tkinter tix.
* turtle::
* typing: typing<2>.
* unittest::
* urllib::
* webbrowser::
File: python.info, Node: PEP 594 Remove “dead batteries” from the standard library, Next: 2to3, Up: Removed Modules And APIs
1.1.7.1 PEP 594: Remove “dead batteries” from the standard library
..................................................................
PEP 594(1) proposed removing 19 modules from the standard library,
colloquially referred to as ‘dead batteries’ due to their historic,
obsolete, or insecure status. All of the following modules were
deprecated in Python 3.11, and are now removed:
* ‘aifc’
* standard-aifc(2): Use the redistribution of ‘aifc’ library
from PyPI.
* ‘audioop’
* audioop-lts(3): Use ‘audioop-lts’ library from PyPI.
* ‘chunk’
* standard-chunk(4): Use the redistribution of ‘chunk’ library
from PyPI.
* ‘cgi’ and ‘cgitb’
* ‘cgi.FieldStorage’ can typically be replaced with *note
urllib.parse.parse_qsl(): 27b. for ‘GET’ and ‘HEAD’ requests,
and the *note email.message: 44. module or the multipart(5)
library for ‘POST’ and ‘PUT’ requests.
* ‘cgi.parse()’ can be replaced by calling *note
urllib.parse.parse_qs(): 27c. directly on the desired query
string, unless the input is ‘multipart/form-data’, which
should be replaced as described below for
‘cgi.parse_multipart()’.
* ‘cgi.parse_header()’ can be replaced with the functionality in
the *note email: 3b. package, which implements the same MIME
RFCs. For example, with *note email.message.EmailMessage:
27d.:
from email.message import EmailMessage
msg = EmailMessage()
msg['content-type'] = 'application/json; charset="utf8"'
main, params = msg.get_content_type(), msg['content-type'].params
* ‘cgi.parse_multipart()’ can be replaced with the functionality
in the *note email: 3b. package, which implements the same
MIME RFCs, or with the multipart(6) library. For example, the
*note email.message.EmailMessage: 27d. and *note
email.message.Message: 27e. classes.
* standard-cgi(7): and standard-cgitb(8): Use the redistribution
of ‘cgi’ and ‘cgitb’ library from PyPI.
* ‘crypt’ and the private ‘_crypt’ extension. The *note hashlib: 68.
module may be an appropriate replacement when simply hashing a
value is required. Otherwise, various third-party libraries on
PyPI are available:
* bcrypt(9): Modern password hashing for your software and your
servers.
* passlib(10): Comprehensive password hashing framework
supporting over 30 schemes.
* argon2-cffi(11): The secure Argon2 password hashing algorithm.
* legacycrypt(12): *note ctypes: 2a. wrapper to the POSIX crypt
library call and associated functionality.
* crypt_r(13): Fork of the ‘crypt’ module, wrapper to the
‘crypt_r(3)(14)’ library call and associated functionality.
* standard-crypt(15) and deprecated-crypt-alternative(16): Use
the redistribution of ‘crypt’ and reimplementation of ‘_crypt’
libraries from PyPI.
* ‘imghdr’: The filetype(17), puremagic(18), or python-magic(19)
libraries should be used as replacements. For example, the
‘puremagic.what()’ function can be used to replace the
‘imghdr.what()’ function for all file formats that were supported
by ‘imghdr’.
* standard-imghdr(20): Use the redistribution of ‘imghdr’
library from PyPI.
* ‘mailcap’: Use the *note mimetypes: 8f. module instead.
* standard-mailcap(21): Use the redistribution of ‘mailcap’
library from PyPI.
* ‘msilib’
* ‘nis’
* ‘nntplib’: Use the pynntp(22) library from PyPI instead.
* standard-nntplib(23): Use the redistribution of ‘nntplib’
library from PyPI.
* ‘ossaudiodev’: For audio playback, use the pygame(24) library from
PyPI instead.
* ‘pipes’: Use the *note subprocess: d6. module instead. Use *note
shlex.quote(): 27f. to replace the undocumented ‘pipes.quote’
function.
* standard-pipes(25): Use the redistribution of ‘pipes’ library
from PyPI.
* ‘sndhdr’: The filetype(26), puremagic(27), or python-magic(28)
libraries should be used as replacements.
* standard-sndhdr(29): Use the redistribution of ‘sndhdr’
library from PyPI.
* ‘spwd’: Use the python-pam(30) library from PyPI instead.
* ‘sunau’
* standard-sunau(31): Use the redistribution of ‘sunau’ library
from PyPI.
* ‘telnetlib’, Use the telnetlib3(32) or Exscript(33) libraries from
PyPI instead.
* standard-telnetlib(34): Use the redistribution of ‘telnetlib’
library from PyPI.
* ‘uu’: Use the *note base64: e. module instead, as a modern
alternative.
* standard-uu(35): Use the redistribution of ‘uu’ library from
PyPI.
* ‘xdrlib’
* standard-xdrlib(36): Use the redistribution of ‘xdrlib’
library from PyPI.
(Contributed by Victor Stinner and Zachary Ware in gh-104773(37) and
gh-104780(38).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0594/
(2) https://pypi.org/project/standard-aifc/
(3) https://pypi.org/project/audioop-lts/
(4) https://pypi.org/project/standard-chunk/
(5) https://pypi.org/project/multipart/
(6) https://pypi.org/project/multipart/
(7) https://pypi.org/project/standard-cgi/
(8) https://pypi.org/project/standard-cgitb/
(9) https://pypi.org/project/bcrypt/
(10) https://pypi.org/project/passlib/
(11) https://pypi.org/project/argon2-cffi/
(12) https://pypi.org/project/legacycrypt/
(13) https://pypi.org/project/crypt_r/
(14) https://manpages.debian.org/crypt_r(3)
(15) https://pypi.org/project/standard-crypt/
(16) https://pypi.org/project/deprecated-crypt-alternative/
(17) https://pypi.org/project/filetype/
(18) https://pypi.org/project/puremagic/
(19) https://pypi.org/project/python-magic/
(20) https://pypi.org/project/standard-imghdr/
(21) https://pypi.org/project/standard-mailcap/
(22) https://pypi.org/project/pynntp/
(23) https://pypi.org/project/standard-nntplib/
(24) https://pypi.org/project/pygame/
(25) https://pypi.org/project/standard-pipes/
(26) https://pypi.org/project/filetype/
(27) https://pypi.org/project/puremagic/
(28) https://pypi.org/project/python-magic/
(29) https://pypi.org/project/standard-sndhdr/
(30) https://pypi.org/project/python-pam/
(31) https://pypi.org/project/standard-sunau/
(32) https://pypi.org/project/telnetlib3/
(33) https://pypi.org/project/Exscript/
(34) https://pypi.org/project/standard-telnetlib/
(35) https://pypi.org/project/standard-uu/
(36) https://pypi.org/project/standard-xdrlib/
(37) https://github.com/python/cpython/issues/104773
(38) https://github.com/python/cpython/issues/104780
File: python.info, Node: 2to3, Next: builtins, Prev: PEP 594 Remove “dead batteries” from the standard library, Up: Removed Modules And APIs
1.1.7.2 2to3
............
* Remove the ‘2to3’ program and the ‘lib2to3’ module, previously
deprecated in Python 3.11. (Contributed by Victor Stinner in
gh-104780(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/104780
File: python.info, Node: builtins, Next: configparser<2>, Prev: 2to3, Up: Removed Modules And APIs
1.1.7.3 builtins
................
* Remove support for chained *note classmethod: 166. descriptors
(introduced in gh-63272(1)). These can no longer be used to wrap
other descriptors, such as *note property: 194. The core design of
this feature was flawed and led to several problems. To
“pass-through” a *note classmethod: 166, consider using the
‘__wrapped__’ attribute that was added in Python 3.10.
(Contributed by Raymond Hettinger in gh-89519(2).)
* Raise a *note RuntimeError: 195. when calling *note frame.clear():
282. on a suspended frame (as has always been the case for an
executing frame). (Contributed by Irit Katriel in gh-79932(3).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/63272
(2) https://github.com/python/cpython/issues/89519
(3) https://github.com/python/cpython/issues/79932
File: python.info, Node: configparser<2>, Next: importlib metadata, Prev: builtins, Up: Removed Modules And APIs
1.1.7.4 configparser
....................
* Remove the undocumented ‘LegacyInterpolation’ class, deprecated in
the docstring since Python 3.2, and at runtime since Python 3.11.
(Contributed by Hugo van Kemenade in gh-104886(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/104886
File: python.info, Node: importlib metadata, Next: locale, Prev: configparser<2>, Up: Removed Modules And APIs
1.1.7.5 importlib.metadata
..........................
* Remove deprecated subscript (*note __getitem__(): 285.) access for
*note EntryPoint: 286. objects. (Contributed by Jason R. Coombs in
gh-113175(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/113175
File: python.info, Node: locale, Next: opcode, Prev: importlib metadata, Up: Removed Modules And APIs
1.1.7.6 locale
..............
* Remove the ‘locale.resetlocale()’ function, deprecated in Python
3.11. Use ‘locale.setlocale(locale.LC_ALL, "")’ instead.
(Contributed by Victor Stinner in gh-104783(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/104783
File: python.info, Node: opcode, Next: optparse, Prev: locale, Up: Removed Modules And APIs
1.1.7.7 opcode
..............
* Move ‘opcode.ENABLE_SPECIALIZATION’ to
‘_opcode.ENABLE_SPECIALIZATION’. This field was added in 3.12, it
was never documented, and is not intended for external use.
(Contributed by Irit Katriel in gh-105481(1).)
* Remove ‘opcode.is_pseudo()’, ‘opcode.MIN_PSEUDO_OPCODE’, and
‘opcode.MAX_PSEUDO_OPCODE’, which were added in Python 3.12, but
were neither documented nor exposed through *note dis: 38, and were
not intended to be used externally. (Contributed by Irit Katriel
in gh-105481(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/105481
(2) https://github.com/python/cpython/issues/105481
File: python.info, Node: optparse, Next: pathlib<2>, Prev: opcode, Up: Removed Modules And APIs
1.1.7.8 optparse
................
* This module is no longer considered *note soft deprecated: 20b.
While *note argparse: 6. remains preferred for new projects that
aren’t using a third party command line argument processing
library, there are aspects of the way ‘argparse’ works that mean
the lower level ‘optparse’ module may provide a better foundation
for 'writing' argument processing libraries, and for implementing
command line applications which adhere more strictly than
‘argparse’ does to various Unix command line processing conventions
that originate in the behaviour of the C ‘getopt()’ function .
(Contributed by Alyssa Coghlan and Serhiy Storchaka in
gh-126180(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/126180
File: python.info, Node: pathlib<2>, Next: re<2>, Prev: optparse, Up: Removed Modules And APIs
1.1.7.9 pathlib
...............
* Remove the ability to use *note Path: 22b. objects as context
managers. This functionality was deprecated and has had no effect
since Python 3.9. (Contributed by Barney Gale in gh-83863(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/83863
File: python.info, Node: re<2>, Next: tkinter tix, Prev: pathlib<2>, Up: Removed Modules And APIs
1.1.7.10 re
...........
* Remove the undocumented, deprecated, and broken ‘re.template()’
function and ‘re.TEMPLATE’ / ‘re.T’ flag. (Contributed by Serhiy
Storchaka and Nikita Sobolev in gh-105687(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/105687
File: python.info, Node: tkinter tix, Next: turtle, Prev: re<2>, Up: Removed Modules And APIs
1.1.7.11 tkinter.tix
....................
* Remove the ‘tkinter.tix’ module, deprecated in Python 3.6. The
third-party Tix library which the module wrapped is unmaintained.
(Contributed by Zachary Ware in gh-75552(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/75552
File: python.info, Node: turtle, Next: typing<2>, Prev: tkinter tix, Up: Removed Modules And APIs
1.1.7.12 turtle
...............
* Remove the ‘RawTurtle.settiltangle()’ method, deprecated in the
documentation since Python 3.1 and at runtime since Python 3.11.
(Contributed by Hugo van Kemenade in gh-104876(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/104876
File: python.info, Node: typing<2>, Next: unittest, Prev: turtle, Up: Removed Modules And APIs
1.1.7.13 typing
...............
* Remove the ‘typing.io’ and ‘typing.re’ namespaces, deprecated since
Python 3.8. The items in those namespaces can be imported directly
from the *note typing: 104. module. (Contributed by Sebastian
Rittau in gh-92871(1).)
* Remove the keyword-argument method of creating *note TypedDict:
162. types, deprecated in Python 3.11. (Contributed by Tomas Roun
in gh-104786(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/92871
(2) https://github.com/python/cpython/issues/104786
File: python.info, Node: unittest, Next: urllib, Prev: typing<2>, Up: Removed Modules And APIs
1.1.7.14 unittest
.................
* Remove the following *note unittest: 106. functions, deprecated in
Python 3.11:
* ‘unittest.findTestCases()’
* ‘unittest.makeSuite()’
* ‘unittest.getTestCaseNames()’
Use *note TestLoader: 290. methods instead:
* *note loadTestsFromModule(): 291.
* *note loadTestsFromTestCase(): 292.
* *note getTestCaseNames(): 293.
(Contributed by Hugo van Kemenade in gh-104835(1).)
* Remove the untested and undocumented ‘TestProgram.usageExit()’
method, deprecated in Python 3.11. (Contributed by Hugo van
Kemenade in gh-104992(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/104835
(2) https://github.com/python/cpython/issues/104992
File: python.info, Node: urllib, Next: webbrowser, Prev: unittest, Up: Removed Modules And APIs
1.1.7.15 urllib
...............
* Remove the 'cafile', 'capath', and 'cadefault' parameters of the
*note urllib.request.urlopen(): 295. function, deprecated in Python
3.6. Use the 'context' parameter instead with an *note SSLContext:
296. instance. The *note ssl.SSLContext.load_cert_chain(): 297.
function can be used to load specific certificates, or let *note
ssl.create_default_context(): 155. select the operating system’s
trusted certificate authority (CA) certificates. (Contributed by
Victor Stinner in gh-105382(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/105382
File: python.info, Node: webbrowser, Prev: urllib, Up: Removed Modules And APIs
1.1.7.16 webbrowser
...................
* Remove the untested and undocumented ‘MacOSX’ class, deprecated in
Python 3.11. Use the ‘MacOSXOSAScript’ class (introduced in Python
3.2) instead. (Contributed by Hugo van Kemenade in gh-104804(1).)
* Remove the deprecated ‘MacOSXOSAScript._name’ attribute. Use the
*note MacOSXOSAScript.name: 299. attribute instead. (Contributed
by Nikita Sobolev in gh-105546(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/104804
(2) https://github.com/python/cpython/issues/105546
File: python.info, Node: New Deprecations, Next: CPython Bytecode Changes, Prev: Removed Modules And APIs, Up: What’s New In Python 3 13
1.1.8 New Deprecations
----------------------
* *note User-defined functions: 29b.:
* Deprecate assignment to a function’s *note __code__: 29c.
attribute, where the new code object’s type does not match the
function’s type. The different types are: plain function,
generator, async generator, and coroutine. (Contributed by
Irit Katriel in gh-81137(1).)
* *note array: 7.:
* Deprecate the ‘'u'’ format code (‘wchar_t’) at runtime. This
format code has been deprecated in documentation since Python
3.3, and will be removed in Python 3.16. Use the ‘'w'’ format
code (*note Py_UCS4: 29d.) for Unicode characters instead.
(Contributed by Hugo van Kemenade in gh-80480(2).)
* *note ctypes: 2a.:
* Deprecate the undocumented ‘SetPointerType()’ function, to be
removed in Python 3.15. (Contributed by Victor Stinner in
gh-105733(3).)
* *note Soft-deprecate: 20b. the *note ARRAY(): 29e. function in
favour of ‘type * length’ multiplication. (Contributed by
Victor Stinner in gh-105733(4).)
* *note decimal: 36.:
* Deprecate the non-standard and undocumented *note Decimal:
29f. format specifier ‘'N'’, which is only supported in the
‘decimal’ module’s C implementation. (Contributed by Serhiy
Storchaka in gh-89902(5).)
* *note dis: 38.:
* Deprecate the ‘HAVE_ARGUMENT’ separator. Check membership in
*note hasarg: 2a0. instead. (Contributed by Irit Katriel in
gh-109319(6).)
* *note gettext: 63.:
* Deprecate non-integer numbers as arguments to functions and
methods that consider plural forms in the ‘gettext’ module,
even if no translation was found. (Contributed by Serhiy
Storchaka in gh-88434(7).)
* *note glob: 64.:
* Deprecate the undocumented ‘glob0()’ and ‘glob1()’ functions.
Use *note glob(): 2a1. and pass a *note path-like object: 2a2.
specifying the root directory to the 'root_dir' parameter
instead. (Contributed by Barney Gale in gh-117337(8).)
* *note http.server: 72.:
* Deprecate *note CGIHTTPRequestHandler: 2a3, to be removed in
Python 3.15. Process-based CGI HTTP servers have been out of
favor for a very long time. This code was outdated,
unmaintained, and rarely used. It has a high potential for
both security and functionality bugs. (Contributed by Gregory
P. Smith in gh-109096(9).)
* Deprecate the ‘--cgi’ flag to the ‘python -m http.server’
command-line interface, to be removed in Python 3.15.
(Contributed by Gregory P. Smith in gh-109096(10).)
* *note mimetypes: 8f.:
* *note Soft-deprecate: 20b. file path arguments to *note
guess_type(): 20a, use *note guess_file_type(): 209. instead.
(Contributed by Serhiy Storchaka in gh-66543(11).)
* *note re: b9.:
* Deprecate passing the optional 'maxsplit', 'count', or 'flags'
arguments as positional arguments to the module-level *note
split(): 2a4, *note sub(): 2a5, and *note subn(): 2a6.
functions. These parameters will become *note keyword-only:
2a7. in a future version of Python. (Contributed by Serhiy
Storchaka in gh-56166(12).)
* *note pathlib: a4.:
* Deprecate *note PurePath.is_reserved(): 2a8, to be removed in
Python 3.15. Use *note os.path.isreserved(): 225. to detect
reserved paths on Windows. (Contributed by Barney Gale in
gh-88569(13).)
* *note platform: aa.:
* Deprecate *note java_ver(): 2a9, to be removed in Python 3.15.
This function is only useful for Jython support, has a
confusing API, and is largely untested. (Contributed by
Nikita Sobolev in gh-116349(14).)
* *note pydoc: b5.:
* Deprecate the undocumented ‘ispackage()’ function.
(Contributed by Zackery Spytz in gh-64020(15).)
* *note sqlite3: cf.:
* Deprecate passing more than one positional argument to the
*note connect(): 2aa. function and the *note Connection: 247.
constructor. The remaining parameters will become
keyword-only in Python 3.15. (Contributed by Erlend E.
Aasland in gh-107948(16).)
* Deprecate passing name, number of arguments, and the callable
as keyword arguments for *note Connection.create_function():
2ab. and *note Connection.create_aggregate(): 2ac. These
parameters will become positional-only in Python 3.15.
(Contributed by Erlend E. Aasland in gh-108278(17).)
* Deprecate passing the callback callable by keyword for the
*note set_authorizer(): 2ad, *note set_progress_handler():
2ae, and *note set_trace_callback(): 2af. *note Connection:
247. methods. The callback callables will become
positional-only in Python 3.15. (Contributed by Erlend E.
Aasland in gh-108278(18).)
* *note sys: d9.:
* Deprecate the *note _enablelegacywindowsfsencoding(): 2b0.
function, to be removed in Python 3.16. Use the *note
PYTHONLEGACYWINDOWSFSENCODING: 2b1. environment variable
instead. (Contributed by Inada Naoki in gh-73427(19).)
* *note tarfile: de.:
* Deprecate the undocumented and unused ‘TarFile.tarfile’
attribute, to be removed in Python 3.16. (Contributed in
gh-115256(20).)
* *note traceback: fe.:
* Deprecate the *note TracebackException.exc_type: 25f.
attribute. Use *note TracebackException.exc_type_str: 25d.
instead. (Contributed by Irit Katriel in gh-112332(21).)
* *note typing: 104.:
* Deprecate the undocumented keyword argument syntax for
creating *note NamedTuple: 2b2. classes (e.g. ‘Point =
NamedTuple("Point", x=int, y=int)’), to be removed in Python
3.15. Use the class-based syntax or the functional syntax
instead. (Contributed by Alex Waygood in gh-105566(22).)
* Deprecate omitting the 'fields' parameter when creating a
*note NamedTuple: 2b2. or *note typing.TypedDict: 162. class,
and deprecate passing ‘None’ to the 'fields' parameter of both
types. Python 3.15 will require a valid sequence for the
'fields' parameter. To create a NamedTuple class with zero
fields, use ‘class NT(NamedTuple): pass’ or ‘NT =
NamedTuple("NT", ())’. To create a TypedDict class with zero
fields, use ‘class TD(TypedDict): pass’ or ‘TD =
TypedDict("TD", {})’. (Contributed by Alex Waygood in
gh-105566(23) and gh-105570(24).)
* Deprecate the *note typing.no_type_check_decorator(): 2b3.
decorator function, to be removed in Python 3.15. After eight
years in the *note typing: 104. module, it has yet to be
supported by any major type checker. (Contributed by Alex
Waygood in gh-106309(25).)
* Deprecate *note typing.AnyStr: 2b4. In Python 3.16, it will
be removed from ‘typing.__all__’, and a *note
DeprecationWarning: 1a5. will be emitted at runtime when it is
imported or accessed. It will be removed entirely in Python
3.18. Use the new *note type parameter syntax: 2b5. instead.
(Contributed by Michael The in gh-107116(26).)
* *note wave: 113.:
* Deprecate the *note getmark(): 2b6, ‘setmark()’, and *note
getmarkers(): 2b7. methods of the *note Wave_read: 2b8. and
*note Wave_write: 2b9. classes, to be removed in Python 3.15.
(Contributed by Victor Stinner in gh-105096(27).)
* Menu:
* Pending Removal in Python 3.14: Pending Removal in Python 3 14.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15.
* Pending removal in Python 3.16: Pending removal in Python 3 16.
* Pending Removal in Future Versions::
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/81137
(2) https://github.com/python/cpython/issues/80480
(3) https://github.com/python/cpython/issues/105733
(4) https://github.com/python/cpython/issues/105733
(5) https://github.com/python/cpython/issues/89902
(6) https://github.com/python/cpython/issues/109319
(7) https://github.com/python/cpython/issues/88434
(8) https://github.com/python/cpython/issues/117337
(9) https://github.com/python/cpython/issues/109096
(10) https://github.com/python/cpython/issues/109096
(11) https://github.com/python/cpython/issues/66543
(12) https://github.com/python/cpython/issues/56166
(13) https://github.com/python/cpython/issues/88569
(14) https://github.com/python/cpython/issues/116349
(15) https://github.com/python/cpython/issues/64020
(16) https://github.com/python/cpython/issues/107948
(17) https://github.com/python/cpython/issues/108278
(18) https://github.com/python/cpython/issues/108278
(19) https://github.com/python/cpython/issues/73427
(20) https://github.com/python/cpython/issues/115256
(21) https://github.com/python/cpython/issues/112332
(22) https://github.com/python/cpython/issues/105566
(23) https://github.com/python/cpython/issues/105566
(24) https://github.com/python/cpython/issues/105570
(25) https://github.com/python/cpython/issues/106309
(26) https://github.com/python/cpython/issues/107116
(27) https://github.com/python/cpython/issues/105096
File: python.info, Node: Pending Removal in Python 3 14, Next: Pending Removal in Python 3 15, Up: New Deprecations
1.1.8.1 Pending Removal in Python 3.14
......................................
* *note argparse: 6.: The 'type', 'choices', and 'metavar' parameters
of ‘argparse.BooleanOptionalAction’ are deprecated and will be
removed in 3.14. (Contributed by Nikita Sobolev in gh-92248(1).)
* *note ast: 8.: The following features have been deprecated in
documentation since Python 3.8, now cause a *note
DeprecationWarning: 1a5. to be emitted at runtime when they are
accessed or used, and will be removed in Python 3.14:
* ‘ast.Num’
* ‘ast.Str’
* ‘ast.Bytes’
* ‘ast.NameConstant’
* ‘ast.Ellipsis’
Use *note ast.Constant: 2bb. instead. (Contributed by Serhiy
Storchaka in gh-90953(2).)
* *note asyncio: a.:
* The child watcher classes *note MultiLoopChildWatcher: 2bc,
*note FastChildWatcher: 2bd, *note AbstractChildWatcher: 2be.
and *note SafeChildWatcher: 2bf. are deprecated and will be
removed in Python 3.14. (Contributed by Kumar Aditya in
gh-94597(3).)
* *note asyncio.set_child_watcher(): 2c0, *note
asyncio.get_child_watcher(): 2c1, *note
asyncio.AbstractEventLoopPolicy.set_child_watcher(): 2c2. and
*note asyncio.AbstractEventLoopPolicy.get_child_watcher():
2c3. are deprecated and will be removed in Python 3.14.
(Contributed by Kumar Aditya in gh-94597(4).)
* The *note get_event_loop(): 2c4. method of the default event
loop policy now emits a *note DeprecationWarning: 1a5. if
there is no current event loop set and it decides to create
one. (Contributed by Serhiy Storchaka and Guido van Rossum in
gh-100160(5).)
* *note collections.abc: 1e.: Deprecated *note ByteString: 2c5.
Prefer ‘Sequence’ or *note Buffer: 2c6. For use in typing, prefer
a union, like ‘bytes | bytearray’, or *note collections.abc.Buffer:
2c6. (Contributed by Shantanu Jain in gh-91896(6).)
* *note email: 3b.: Deprecated the 'isdst' parameter in *note
email.utils.localtime(): 2c7. (Contributed by Alan Williams in
gh-72346(7).)
* *note importlib.abc: 78. deprecated classes:
* ‘importlib.abc.ResourceReader’
* ‘importlib.abc.Traversable’
* ‘importlib.abc.TraversableResources’
Use *note importlib.resources.abc: 7c. classes instead:
* *note importlib.resources.abc.Traversable: 1f5.
* *note importlib.resources.abc.TraversableResources: 2c8.
(Contributed by Jason R. Coombs and Hugo van Kemenade in
gh-93963(8).)
* *note itertools: 81. had undocumented, inefficient, historically
buggy, and inconsistent support for copy, deepcopy, and pickle
operations. This will be removed in 3.14 for a significant
reduction in code volume and maintenance burden. (Contributed by
Raymond Hettinger in gh-101588(9).)
* *note multiprocessing: 94.: The default start method will change to
a safer one on Linux, BSDs, and other non-macOS POSIX platforms
where ‘'fork'’ is currently the default (gh-84559(10)). Adding a
runtime warning about this was deemed too disruptive as the
majority of code is not expected to care. Use the *note
get_context(): 2c9. or *note set_start_method(): 2ca. APIs to
explicitly specify when your code 'requires' ‘'fork'’. See *note
Contexts and start methods: 2cb.
* *note pathlib: a4.: *note is_relative_to(): 2cc. and *note
relative_to(): 2cd.: passing additional arguments is deprecated.
* *note pkgutil: a9.: *note find_loader(): 2ce. and *note
get_loader(): 2cf. now raise *note DeprecationWarning: 1a5.; use
*note importlib.util.find_spec(): 2d0. instead. (Contributed by
Nikita Sobolev in gh-97850(11).)
* *note pty: b1.:
* ‘master_open()’: use *note pty.openpty(): 2d1.
* ‘slave_open()’: use *note pty.openpty(): 2d1.
* *note sqlite3: cf.:
* *note version: 2d2. and *note version_info: 2d3.
* *note execute(): 2d4. and *note executemany(): 2d5. if *note
named placeholders: 2d6. are used and 'parameters' is a
sequence instead of a *note dict: 258.
* *note typing: 104.: *note ByteString: 2d7, deprecated since Python
3.9, now causes a *note DeprecationWarning: 1a5. to be emitted when
it is used.
* *note urllib: 108.: ‘urllib.parse.Quoter’ is deprecated: it was not
intended to be a public API. (Contributed by Gregory P. Smith in
gh-88168(12).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/92248
(2) https://github.com/python/cpython/issues/90953
(3) https://github.com/python/cpython/issues/94597
(4) https://github.com/python/cpython/issues/94597
(5) https://github.com/python/cpython/issues/100160
(6) https://github.com/python/cpython/issues/91896
(7) https://github.com/python/cpython/issues/72346
(8) https://github.com/python/cpython/issues/93963
(9) https://github.com/python/cpython/issues/101588
(10) https://github.com/python/cpython/issues/84559
(11) https://github.com/python/cpython/issues/97850
(12) https://github.com/python/cpython/issues/88168
File: python.info, Node: Pending Removal in Python 3 15, Next: Pending removal in Python 3 16, Prev: Pending Removal in Python 3 14, Up: New Deprecations
1.1.8.2 Pending Removal in Python 3.15
......................................
* The import system:
* Setting *note __cached__: 2d9. on a module while failing to
set *note __spec__.cached: 2da. is deprecated. In Python
3.15, ‘__cached__’ will cease to be set or take into
consideration by the import system or standard library.
(gh-97879(1))
* Setting *note __package__: 2db. on a module while failing to
set *note __spec__.parent: 2dc. is deprecated. In Python
3.15, ‘__package__’ will cease to be set or take into
consideration by the import system or standard library.
(gh-97879(2))
* *note ctypes: 2a.:
* The undocumented ‘ctypes.SetPointerType()’ function has been
deprecated since Python 3.13.
* *note http.server: 72.:
* The obsolete and rarely used *note CGIHTTPRequestHandler: 2a3.
has been deprecated since Python 3.13. No direct replacement
exists. 'Anything' is better than CGI to interface a web
server with a request handler.
* The ‘--cgi’ flag to the ‘python -m http.server’ command-line
interface has been deprecated since Python 3.13.
* *note importlib: 77.:
* ‘load_module()’ method: use ‘exec_module()’ instead.
* *note locale: 86.:
* The *note getdefaultlocale(): 2dd. function has been
deprecated since Python 3.11. Its removal was originally
planned for Python 3.13 (gh-90817(3)), but has been postponed
to Python 3.15. Use *note getlocale(): 2de, *note
setlocale(): 2df, and *note getencoding(): 2e0. instead.
(Contributed by Hugo van Kemenade in gh-111187(4).)
* *note pathlib: a4.:
* *note PurePath.is_reserved(): 2a8. has been deprecated since
Python 3.13. Use *note os.path.isreserved(): 225. to detect
reserved paths on Windows.
* *note platform: aa.:
* *note java_ver(): 2a9. has been deprecated since Python 3.13.
This function is only useful for Jython support, has a
confusing API, and is largely untested.
* *note sysconfig: db.:
* The 'check_home' argument of *note
sysconfig.is_python_build(): 2e1. has been deprecated since
Python 3.12.
* *note threading: ed.:
* *note RLock(): 2e2. will take no arguments in Python 3.15.
Passing any arguments has been deprecated since Python 3.14,
as the Python version does not permit any arguments, but the C
version allows any number of positional or keyword arguments,
ignoring every argument.
* *note types: 103.:
* *note types.CodeType: 2e3.: Accessing *note co_lnotab: 2e4.
was deprecated in PEP 626(5) since 3.10 and was planned to be
removed in 3.12, but it only got a proper *note
DeprecationWarning: 1a5. in 3.12. May be removed in 3.15.
(Contributed by Nikita Sobolev in gh-101866(6).)
* *note typing: 104.:
* The undocumented keyword argument syntax for creating *note
NamedTuple: 2b2. classes (e.g. ‘Point = NamedTuple("Point",
x=int, y=int)’) has been deprecated since Python 3.13. Use
the class-based syntax or the functional syntax instead.
* When using the functional syntax of *note TypedDict: 162.s,
failing to pass a value to the 'fields' parameter (‘TD =
TypedDict("TD")’) or passing ‘None’ (‘TD = TypedDict("TD",
None)’) has been deprecated since Python 3.13. Use ‘class
TD(TypedDict): pass’ or ‘TD = TypedDict("TD", {})’ to create a
TypedDict with zero field.
* The *note typing.no_type_check_decorator(): 2b3. decorator
function has been deprecated since Python 3.13. After eight
years in the *note typing: 104. module, it has yet to be
supported by any major type checker.
* *note wave: 113.:
* The *note getmark(): 2b6, ‘setmark()’, and *note getmarkers():
2b7. methods of the *note Wave_read: 2b8. and *note
Wave_write: 2b9. classes have been deprecated since Python
3.13.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/97879
(2) https://github.com/python/cpython/issues/97879
(3) https://github.com/python/cpython/issues/90817
(4) https://github.com/python/cpython/issues/111187
(5) https://peps.python.org/pep-0626/
(6) https://github.com/python/cpython/issues/101866
File: python.info, Node: Pending removal in Python 3 16, Next: Pending Removal in Future Versions, Prev: Pending Removal in Python 3 15, Up: New Deprecations
1.1.8.3 Pending removal in Python 3.16
......................................
* The import system:
* Setting *note __loader__: 2e6. on a module while failing to
set *note __spec__.loader: 2e7. is deprecated. In Python
3.16, ‘__loader__’ will cease to be set or taken into
consideration by the import system or the standard library.
* *note array: 7.:
* The ‘'u'’ format code (‘wchar_t’) has been deprecated in
documentation since Python 3.3 and at runtime since Python
3.13. Use the ‘'w'’ format code (*note Py_UCS4: 29d.) for
Unicode characters instead.
* *note asyncio: a.:
* ‘asyncio.iscoroutinefunction()’ is deprecated and will be
removed in Python 3.16, use *note
inspect.iscoroutinefunction(): 2e8. instead. (Contributed by
Jiahao Li and Kumar Aditya in gh-122875(1).)
* *note builtins: 12.:
* Bitwise inversion on boolean types, ‘~True’ or ‘~False’ has
been deprecated since Python 3.12, as it produces surprising
and unintuitive results (‘-2’ and ‘-1’). Use ‘not x’ instead
for the logical negation of a Boolean. In the rare case that
you need the bitwise inversion of the underlying integer,
convert to ‘int’ explicitly (‘~int(x)’).
* *note shutil: c5.:
* The ‘ExecError’ exception has been deprecated since Python
3.14. It has not been used by any function in ‘shutil’ since
Python 3.4, and is now an alias of *note RuntimeError: 195.
* *note symtable: d8.:
* The *note Class.get_methods: 2e9. method has been deprecated
since Python 3.14.
* *note sys: d9.:
* The *note _enablelegacywindowsfsencoding(): 2b0. function has
been deprecated since Python 3.13. Use the *note
PYTHONLEGACYWINDOWSFSENCODING: 2b1. environment variable
instead.
* *note tarfile: de.:
* The undocumented and unused ‘TarFile.tarfile’ attribute has
been deprecated since Python 3.13.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/122875
File: python.info, Node: Pending Removal in Future Versions, Prev: Pending removal in Python 3 16, Up: New Deprecations
1.1.8.4 Pending Removal in Future Versions
..........................................
The following APIs will be removed in the future, although there is
currently no date scheduled for their removal.
* *note argparse: 6.: Nesting argument groups and nesting mutually
exclusive groups are deprecated.
* *note builtins: 12.:
* ‘bool(NotImplemented)’.
* Generators: ‘throw(type, exc, tb)’ and ‘athrow(type, exc, tb)’
signature is deprecated: use ‘throw(exc)’ and ‘athrow(exc)’
instead, the single argument signature.
* Currently Python accepts numeric literals immediately followed
by keywords, for example ‘0in x’, ‘1or x’, ‘0if 1else 2’. It
allows confusing and ambiguous expressions like ‘[0x1for x in
y]’ (which can be interpreted as ‘[0x1 for x in y]’ or ‘[0x1f
or x in y]’). A syntax warning is raised if the numeric
literal is immediately followed by one of keywords *note and:
2eb, *note else: 18c, *note for: 2ec, *note if: 2ed, *note in:
2ee, *note is: 2ef. and *note or: 2f0. In a future release it
will be changed to a syntax error. (gh-87999(1))
* Support for ‘__index__()’ and ‘__int__()’ method returning
non-int type: these methods will be required to return an
instance of a strict subclass of *note int: 259.
* Support for ‘__float__()’ method returning a strict subclass
of *note float: 2f1.: these methods will be required to return
an instance of *note float: 2f1.
* Support for ‘__complex__()’ method returning a strict subclass
of *note complex: 2f2.: these methods will be required to
return an instance of *note complex: 2f2.
* Delegation of ‘int()’ to ‘__trunc__()’ method.
* Passing a complex number as the 'real' or 'imag' argument in
the *note complex(): 2f2. constructor is now deprecated; it
should only be passed as a single positional argument.
(Contributed by Serhiy Storchaka in gh-109218(2).)
* *note calendar: 14.: ‘calendar.January’ and ‘calendar.February’
constants are deprecated and replaced by *note calendar.JANUARY:
2f3. and *note calendar.FEBRUARY: 2f4. (Contributed by Prince
Roshan in gh-103636(3).)
* *note codeobject.co_lnotab: 2e4.: use the *note
codeobject.co_lines(): 2f5. method instead.
* *note datetime: 30.:
* *note utcnow(): 2f6.: use
‘datetime.datetime.now(tz=datetime.UTC)’.
* *note utcfromtimestamp(): 2f7.: use
‘datetime.datetime.fromtimestamp(timestamp, tz=datetime.UTC)’.
* *note gettext: 63.: Plural value must be an integer.
* *note importlib: 77.:
* *note cache_from_source(): 2f8. 'debug_override' parameter is
deprecated: use the 'optimization' parameter instead.
* *note importlib.metadata: 7a.:
* ‘EntryPoints’ tuple interface.
* Implicit ‘None’ on return values.
* *note logging: 87.: the ‘warn()’ method has been deprecated since
Python 3.3, use *note warning(): 2f9. instead.
* *note mailbox: 8b.: Use of StringIO input and text mode is
deprecated, use BytesIO and binary mode instead.
* *note os: a1.: Calling *note os.register_at_fork(): 2fa. in
multi-threaded process.
* ‘pydoc.ErrorDuringImport’: A tuple value for 'exc_info' parameter
is deprecated, use an exception instance.
* *note re: b9.: More strict rules are now applied for numerical
group references and group names in regular expressions. Only
sequence of ASCII digits is now accepted as a numerical reference.
The group name in bytes patterns and replacement strings can now
only contain ASCII letters and digits and underscore. (Contributed
by Serhiy Storchaka in gh-91760(4).)
* ‘sre_compile’, ‘sre_constants’ and ‘sre_parse’ modules.
* *note shutil: c5.: *note rmtree(): 2fb.’s 'onerror' parameter is
deprecated in Python 3.12; use the 'onexc' parameter instead.
* *note ssl: d0. options and protocols:
* *note ssl.SSLContext: 296. without protocol argument is
deprecated.
* *note ssl.SSLContext: 296.: *note set_npn_protocols(): 2fc.
and ‘selected_npn_protocol()’ are deprecated: use ALPN
instead.
* ‘ssl.OP_NO_SSL*’ options
* ‘ssl.OP_NO_TLS*’ options
* ‘ssl.PROTOCOL_SSLv3’
* ‘ssl.PROTOCOL_TLS’
* ‘ssl.PROTOCOL_TLSv1’
* ‘ssl.PROTOCOL_TLSv1_1’
* ‘ssl.PROTOCOL_TLSv1_2’
* ‘ssl.TLSVersion.SSLv3’
* ‘ssl.TLSVersion.TLSv1’
* ‘ssl.TLSVersion.TLSv1_1’
* *note threading: ed. methods:
* ‘threading.Condition.notifyAll()’: use *note notify_all():
2fd.
* ‘threading.Event.isSet()’: use *note is_set(): 2fe.
* ‘threading.Thread.isDaemon()’, *note
threading.Thread.setDaemon(): 2ff.: use *note
threading.Thread.daemon: 300. attribute.
* ‘threading.Thread.getName()’, *note
threading.Thread.setName(): 301.: use *note
threading.Thread.name: 302. attribute.
* ‘threading.currentThread()’: use *note
threading.current_thread(): 303.
* ‘threading.activeCount()’: use *note threading.active_count():
304.
* *note typing.Text: 305. (gh-92332(5)).
* *note unittest.IsolatedAsyncioTestCase: 306.: it is deprecated to
return a value that is not ‘None’ from a test case.
* *note urllib.parse: 10a. deprecated functions: *note urlparse():
307. instead
* ‘splitattr()’
* ‘splithost()’
* ‘splitnport()’
* ‘splitpasswd()’
* ‘splitport()’
* ‘splitquery()’
* ‘splittag()’
* ‘splittype()’
* ‘splituser()’
* ‘splitvalue()’
* ‘to_bytes()’
* *note urllib.request: 10b.: *note URLopener: 308. and *note
FancyURLopener: 309. style of invoking requests is deprecated. Use
newer *note urlopen(): 295. functions and methods.
* *note wsgiref: 118.: ‘SimpleHandler.stdout.write()’ should not do
partial writes.
* *note xml.etree.ElementTree: 125.: Testing the truth value of an
*note Element: 30a. is deprecated. In a future release it will
always return ‘True’. Prefer explicit ‘len(elem)’ or ‘elem is not
None’ tests instead.
* *note zipimport.zipimporter.load_module(): 30b. is deprecated: use
*note exec_module(): 30c. instead.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/87999
(2) https://github.com/python/cpython/issues/109218
(3) https://github.com/python/cpython/issues/103636
(4) https://github.com/python/cpython/issues/91760
(5) https://github.com/python/cpython/issues/92332
File: python.info, Node: CPython Bytecode Changes, Next: C API Changes, Prev: New Deprecations, Up: What’s New In Python 3 13
1.1.9 CPython Bytecode Changes
------------------------------
* The oparg of *note YIELD_VALUE: 30e. is now ‘1’ if the yield is
part of a yield-from or await, and ‘0’ otherwise. The oparg of
*note RESUME: 30f. was changed to add a bit indicating if the
except-depth is 1, which is needed to optimize closing of
generators. (Contributed by Irit Katriel in gh-111354(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/111354
File: python.info, Node: C API Changes, Next: Build Changes, Prev: CPython Bytecode Changes, Up: What’s New In Python 3 13
1.1.10 C API Changes
--------------------
* Menu:
* New Features: New Features<2>.
* Changed C APIs::
* Limited C API Changes::
* Removed C APIs::
* Deprecated C APIs::
File: python.info, Node: New Features<2>, Next: Changed C APIs, Up: C API Changes
1.1.10.1 New Features
.....................
* Add the *note PyMonitoring C API: 15c. for generating PEP 669(1)
monitoring events:
* *note PyMonitoringState: 312.
* *note PyMonitoring_FirePyStartEvent(): 313.
* *note PyMonitoring_FirePyResumeEvent(): 314.
* *note PyMonitoring_FirePyReturnEvent(): 315.
* *note PyMonitoring_FirePyYieldEvent(): 316.
* *note PyMonitoring_FireCallEvent(): 317.
* *note PyMonitoring_FireLineEvent(): 318.
* *note PyMonitoring_FireJumpEvent(): 319.
* *note PyMonitoring_FireBranchEvent(): 31a.
* *note PyMonitoring_FireCReturnEvent(): 31b.
* *note PyMonitoring_FirePyThrowEvent(): 31c.
* *note PyMonitoring_FireRaiseEvent(): 31d.
* *note PyMonitoring_FireCRaiseEvent(): 31e.
* *note PyMonitoring_FireReraiseEvent(): 31f.
* *note PyMonitoring_FireExceptionHandledEvent(): 320.
* *note PyMonitoring_FirePyUnwindEvent(): 321.
* *note PyMonitoring_FireStopIterationEvent(): 322.
* *note PyMonitoring_EnterScope(): 323.
* *note PyMonitoring_ExitScope(): 324.
(Contributed by Irit Katriel in gh-111997(2)).
* Add *note PyMutex: 15b, a lightweight mutex that occupies a single
byte, and the new *note PyMutex_Lock(): 325. and *note
PyMutex_Unlock(): 326. functions. ‘PyMutex_Lock()’ will release
the *note GIL: 159. (if currently held) if the operation needs to
block. (Contributed by Sam Gross in gh-108724(3).)
* Add the *note PyTime C API: 327. to provide access to system
clocks:
* *note PyTime_t: 328.
* *note PyTime_MIN: 329. and *note PyTime_MAX: 32a.
* *note PyTime_AsSecondsDouble(): 32b.
* *note PyTime_Monotonic(): 32c.
* *note PyTime_MonotonicRaw(): 32d.
* *note PyTime_PerfCounter(): 32e.
* *note PyTime_PerfCounterRaw(): 32f.
* *note PyTime_Time(): 330.
* *note PyTime_TimeRaw(): 331.
(Contributed by Victor Stinner and Petr Viktorin in gh-110850(4).)
* Add the *note PyDict_ContainsString(): 332. function with the same
behavior as *note PyDict_Contains(): 333, but 'key' is specified as
a const char* UTF-8 encoded bytes string, rather than a *note
PyObject: 334.*. (Contributed by Victor Stinner in gh-108314(5).)
* Add the *note PyDict_GetItemRef(): 335. and *note
PyDict_GetItemStringRef(): 336. functions, which behave similarly
to *note PyDict_GetItemWithError(): 337, but return a *note strong
reference: 338. instead of a *note borrowed reference: 339.
Moreover, these functions return ‘-1’ on error, removing the need
to check ‘PyErr_Occurred()’. (Contributed by Victor Stinner in
gh-106004(6).)
* Add the *note PyDict_SetDefaultRef(): 33a. function, which behaves
similarly to *note PyDict_SetDefault(): 33b, but returns a *note
strong reference: 338. instead of a *note borrowed reference: 339.
This function returns ‘-1’ on error, ‘0’ on insertion, and ‘1’ if
the key was already present in the dictionary. (Contributed by Sam
Gross in gh-112066(7).)
* Add the *note PyDict_Pop(): 33c. and *note PyDict_PopString(): 33d.
functions to remove a key from a dictionary and optionally return
the removed value. This is similar to *note dict.pop(): 33e,
though there is no default value, and *note KeyError: 33f. is not
raised for missing keys. (Contributed by Stefan Behnel and Victor
Stinner in gh-111262(8).)
* Add the *note PyMapping_GetOptionalItem(): 340. and *note
PyMapping_GetOptionalItemString(): 341. functions as alternatives
to *note PyObject_GetItem(): 342. and *note
PyMapping_GetItemString(): 343. respectively. The new functions do
not raise *note KeyError: 33f. if the requested key is missing from
the mapping. These variants are more convenient and faster if a
missing key should not be treated as a failure. (Contributed by
Serhiy Storchaka in gh-106307(9).)
* Add the *note PyObject_GetOptionalAttr(): 344. and *note
PyObject_GetOptionalAttrString(): 345. functions as alternatives to
*note PyObject_GetAttr(): 346. and *note PyObject_GetAttrString():
347. respectively. The new functions do not raise *note
AttributeError: 348. if the requested attribute is not found on the
object. These variants are more convenient and faster if the
missing attribute should not be treated as a failure. (Contributed
by Serhiy Storchaka in gh-106521(10).)
* Add the *note PyErr_FormatUnraisable(): 349. function as an
extension to *note PyErr_WriteUnraisable(): 34a. that allows
customizing the warning message. (Contributed by Serhiy Storchaka
in gh-108082(11).)
* Add new functions that return a *note strong reference: 338.
instead of a *note borrowed reference: 339. for frame locals,
globals, and builtins, as part of *note PEP 667: 142.:
* *note PyEval_GetFrameBuiltins(): 34b. replaces *note
PyEval_GetBuiltins(): 34c.
* *note PyEval_GetFrameGlobals(): 34d. replaces *note
PyEval_GetGlobals(): 34e.
* *note PyEval_GetFrameLocals(): 34f. replaces *note
PyEval_GetLocals(): 350.
(Contributed by Mark Shannon and Tian Gao in gh-74929(12).)
* Add the *note Py_GetConstant(): 351. and *note
Py_GetConstantBorrowed(): 352. functions to get *note strong: 338.
or *note borrowed: 339. references to constants. For example,
‘Py_GetConstant(Py_CONSTANT_ZERO)’ returns a strong reference to
the constant zero. (Contributed by Victor Stinner in
gh-115754(13).)
* Add the *note PyImport_AddModuleRef(): 353. function as a
replacement for *note PyImport_AddModule(): 354. that returns a
*note strong reference: 338. instead of a *note borrowed reference:
339. (Contributed by Victor Stinner in gh-105922(14).)
* Add the *note Py_IsFinalizing(): 355. function to check whether the
main Python interpreter is *note shutting down: 14e. (Contributed
by Victor Stinner in gh-108014(15).)
* Add the *note PyList_GetItemRef(): 356. function as a replacement
for *note PyList_GetItem(): 357. that returns a *note strong
reference: 338. instead of a *note borrowed reference: 339.
(Contributed by Sam Gross in gh-114329(16).)
* Add the *note PyList_Extend(): 358. and *note PyList_Clear(): 359.
functions, mirroring the Python ‘list.extend()’ and ‘list.clear()’
methods. (Contributed by Victor Stinner in gh-111138(17).)
* Add the *note PyLong_AsInt(): 35a. function. It behaves similarly
to *note PyLong_AsLong(): 35b, but stores the result in a C int
instead of a C long. (Contributed by Victor Stinner in
gh-108014(18).)
* Add the *note PyLong_AsNativeBytes(): 35c, *note
PyLong_FromNativeBytes(): 35d, and *note
PyLong_FromUnsignedNativeBytes(): 35e. functions to simplify
converting between native integer types and Python *note int: 259.
objects. (Contributed by Steve Dower in gh-111140(19).)
* Add *note PyModule_Add(): 35f. function, which is similar to *note
PyModule_AddObjectRef(): 360. and *note PyModule_AddObject(): 361,
but always steals a reference to the value. (Contributed by Serhiy
Storchaka in gh-86493(20).)
* Add the *note PyObject_GenericHash(): 362. function that implements
the default hashing function of a Python object. (Contributed by
Serhiy Storchaka in gh-113024(21).)
* Add the *note Py_HashPointer(): 363. function to hash a raw
pointer. (Contributed by Victor Stinner in gh-111545(22).)
* Add the *note PyObject_VisitManagedDict(): 364. and *note
PyObject_ClearManagedDict(): 365. functions. which must be called
by the traverse and clear functions of a type using the *note
Py_TPFLAGS_MANAGED_DICT: 366. flag. The pythoncapi-compat
project(23) can be used to use these functions with Python 3.11 and
3.12. (Contributed by Victor Stinner in gh-107073(24).)
* Add the *note PyRefTracer_SetTracer(): 367. and *note
PyRefTracer_GetTracer(): 368. functions, which enable tracking
object creation and destruction in the same way that the *note
tracemalloc: ff. module does. (Contributed by Pablo Galindo in
gh-93502(25).)
* Add the *note PySys_AuditTuple(): 369. function as an alternative
to *note PySys_Audit(): 36a. that takes event arguments as a Python
*note tuple: 36b. object. (Contributed by Victor Stinner in
gh-85283(26).)
* Add the *note PyThreadState_GetUnchecked(): 36c. function as an
alternative to *note PyThreadState_Get(): 36d. that doesn’t kill
the process with a fatal error if it is ‘NULL’. The caller is
responsible for checking if the result is ‘NULL’. (Contributed by
Victor Stinner in gh-108867(27).)
* Add the *note PyType_GetFullyQualifiedName(): 36e. function to get
the type’s fully qualified name. The module name is prepended if
*note type.__module__: 36f. is a string and is not equal to either
‘'builtins'’ or ‘'__main__'’. (Contributed by Victor Stinner in
gh-111696(28).)
* Add the *note PyType_GetModuleName(): 370. function to get the
type’s module name. This is equivalent to getting the *note
type.__module__: 36f. attribute. (Contributed by Eric Snow and
Victor Stinner in gh-111696(29).)
* Add the *note PyUnicode_EqualToUTF8AndSize(): 371. and *note
PyUnicode_EqualToUTF8(): 372. functions to compare a Unicode object
with a const char* UTF-8 encoded string and ‘1’ if they are equal
or ‘0’ otherwise. These functions do not raise exceptions.
(Contributed by Serhiy Storchaka in gh-110289(30).)
* Add the *note PyWeakref_GetRef(): 373. function as an alternative
to *note PyWeakref_GetObject(): 374. that returns a *note strong
reference: 338. or ‘NULL’ if the referent is no longer live.
(Contributed by Victor Stinner in gh-105927(31).)
* Add fixed variants of functions which silently ignore errors:
* *note PyObject_HasAttrWithError(): 375. replaces *note
PyObject_HasAttr(): 376.
* *note PyObject_HasAttrStringWithError(): 377. replaces *note
PyObject_HasAttrString(): 378.
* *note PyMapping_HasKeyWithError(): 379. replaces *note
PyMapping_HasKey(): 37a.
* *note PyMapping_HasKeyStringWithError(): 37b. replaces *note
PyMapping_HasKeyString(): 37c.
The new functions return ‘-1’ for errors and the standard ‘1’ for
true and ‘0’ for false.
(Contributed by Serhiy Storchaka in gh-108511(32).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0669/
(2) https://github.com/python/cpython/issues/111997
(3) https://github.com/python/cpython/issues/108724
(4) https://github.com/python/cpython/issues/110850
(5) https://github.com/python/cpython/issues/108314
(6) https://github.com/python/cpython/issues/106004
(7) https://github.com/python/cpython/issues/112066
(8) https://github.com/python/cpython/issues/111262
(9) https://github.com/python/cpython/issues/106307
(10) https://github.com/python/cpython/issues/106521
(11) https://github.com/python/cpython/issues/108082
(12) https://github.com/python/cpython/issues/74929
(13) https://github.com/python/cpython/issues/115754
(14) https://github.com/python/cpython/issues/105922
(15) https://github.com/python/cpython/issues/108014
(16) https://github.com/python/cpython/issues/114329
(17) https://github.com/python/cpython/issues/111138
(18) https://github.com/python/cpython/issues/108014
(19) https://github.com/python/cpython/issues/111140
(20) https://github.com/python/cpython/issues/86493
(21) https://github.com/python/cpython/issues/113024
(22) https://github.com/python/cpython/issues/111545
(23) https://github.com/python/pythoncapi-compat/
(24) https://github.com/python/cpython/issues/107073
(25) https://github.com/python/cpython/issues/93502
(26) https://github.com/python/cpython/issues/85283
(27) https://github.com/python/cpython/issues/108867
(28) https://github.com/python/cpython/issues/111696
(29) https://github.com/python/cpython/issues/111696
(30) https://github.com/python/cpython/issues/110289
(31) https://github.com/python/cpython/issues/105927
(32) https://github.com/python/cpython/issues/108511
File: python.info, Node: Changed C APIs, Next: Limited C API Changes, Prev: New Features<2>, Up: C API Changes
1.1.10.2 Changed C APIs
.......................
* The 'keywords' parameter of *note PyArg_ParseTupleAndKeywords():
37e. and *note PyArg_VaParseTupleAndKeywords(): 37f. now has type
char *const* in C and const char *const* in C++, instead of char**.
In C++, this makes these functions compatible with arguments of
type const char *const*, const char**, or char *const* without an
explicit type cast. In C, the functions only support arguments of
type char *const*. This can be overridden with the *note
PY_CXX_CONST: 380. macro. (Contributed by Serhiy Storchaka in
gh-65210(1).)
* *note PyArg_ParseTupleAndKeywords(): 37e. now supports non-ASCII
keyword parameter names. (Contributed by Serhiy Storchaka in
gh-110815(2).)
* The ‘PyCode_GetFirstFree()’ function is now unstable API and is now
named *note PyUnstable_Code_GetFirstFree(): 381. (Contributed by
Bogdan Romanyuk in gh-115781(3).)
* The *note PyDict_GetItem(): 382, *note PyDict_GetItemString(): 383,
*note PyMapping_HasKey(): 37a, *note PyMapping_HasKeyString(): 37c,
*note PyObject_HasAttr(): 376, *note PyObject_HasAttrString(): 378,
and *note PySys_GetObject(): 384. functions, each of which clears
all errors which occurred when calling them now reports these
errors using *note sys.unraisablehook(): 1f9. You may replace them
with other functions as recommended in the documentation.
(Contributed by Serhiy Storchaka in gh-106672(4).)
* Add support for the ‘%T’, ‘%#T’, ‘%N’ and ‘%#N’ formats to *note
PyUnicode_FromFormat(): 385.:
* ‘%T’: Get the fully qualified name of an object type
* ‘%#T’: As above, but use a colon as the separator
* ‘%N’: Get the fully qualified name of a type
* ‘%#N’: As above, but use a colon as the separator
See PEP 737(5) for more information. (Contributed by Victor
Stinner in gh-111696(6).)
* You no longer have to define the ‘PY_SSIZE_T_CLEAN’ macro before
including ‘Python.h’ when using ‘#’ formats in *note format codes:
386. APIs accepting the format codes always use ‘Py_ssize_t’ for
‘#’ formats. (Contributed by Inada Naoki in gh-104922(7).)
* If Python is built in *note debug mode: 1fb. or *note with
assertions: 387, *note PyTuple_SET_ITEM(): 388. and *note
PyList_SET_ITEM(): 389. now check the index argument with an
assertion. (Contributed by Victor Stinner in gh-106168(8).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/65210
(2) https://github.com/python/cpython/issues/110815
(3) https://github.com/python/cpython/issues/115781
(4) https://github.com/python/cpython/issues/106672
(5) https://peps.python.org/pep-0737/
(6) https://github.com/python/cpython/issues/111696
(7) https://github.com/python/cpython/issues/104922
(8) https://github.com/python/cpython/issues/106168
File: python.info, Node: Limited C API Changes, Next: Removed C APIs, Prev: Changed C APIs, Up: C API Changes
1.1.10.3 Limited C API Changes
..............................
* The following functions are now included in the Limited C API:
* *note PyMem_RawMalloc(): 38b.
* *note PyMem_RawCalloc(): 38c.
* *note PyMem_RawRealloc(): 38d.
* *note PyMem_RawFree(): 38e.
* *note PySys_Audit(): 36a.
* *note PySys_AuditTuple(): 369.
* *note PyType_GetModuleByDef(): 38f.
(Contributed by Victor Stinner in gh-85283(1), gh-85283(2), and
gh-116936(3).)
* Python built with *note -with-trace-refs: 390. (tracing references)
now supports the *note Limited API: 391. (Contributed by Victor
Stinner in gh-108634(4).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/85283
(2) https://github.com/python/cpython/issues/85283
(3) https://github.com/python/cpython/issues/116936
(4) https://github.com/python/cpython/issues/108634
File: python.info, Node: Removed C APIs, Next: Deprecated C APIs, Prev: Limited C API Changes, Up: C API Changes
1.1.10.4 Removed C APIs
.......................
* Remove several functions, macros, variables, etc with names
prefixed by ‘_Py’ or ‘_PY’ (which are considered private). If your
project is affected by one of these removals and you believe that
the removed API should remain available, please *note open a new
issue: 250. to request a public C API and add ‘cc: @vstinner’ to
the issue to notify Victor Stinner. (Contributed by Victor Stinner
in gh-106320(1).)
* Remove old buffer protocols deprecated in Python 3.0. Use *note
Buffer Protocol: 393. instead.
* ‘PyObject_CheckReadBuffer()’: Use *note
PyObject_CheckBuffer(): 394. to test whether the object
supports the buffer protocol. Note that *note
PyObject_CheckBuffer(): 394. doesn’t guarantee that *note
PyObject_GetBuffer(): 395. will succeed. To test if the
object is actually readable, see the next example of *note
PyObject_GetBuffer(): 395.
* ‘PyObject_AsCharBuffer()’, ‘PyObject_AsReadBuffer()’: Use
*note PyObject_GetBuffer(): 395. and *note PyBuffer_Release():
396. instead:
Py_buffer view;
if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) < 0) {
return NULL;
}
// Use `view.buf` and `view.len` to read from the buffer.
// You may need to cast buf as `(const char*)view.buf`.
PyBuffer_Release(&view);
* ‘PyObject_AsWriteBuffer()’: Use *note PyObject_GetBuffer():
395. and *note PyBuffer_Release(): 396. instead:
Py_buffer view;
if (PyObject_GetBuffer(obj, &view, PyBUF_WRITABLE) < 0) {
return NULL;
}
// Use `view.buf` and `view.len` to write to the buffer.
PyBuffer_Release(&view);
(Contributed by Inada Naoki in gh-85275(2).)
* Remove various functions deprecated in Python 3.9:
* ‘PyEval_CallObject()’, ‘PyEval_CallObjectWithKeywords()’: Use
*note PyObject_CallNoArgs(): 397. or *note PyObject_Call():
398. instead.
Warning: In *note PyObject_Call(): 398, positional
arguments must be a *note tuple: 36b. and must not be
‘NULL’, and keyword arguments must be a *note dict: 258.
or ‘NULL’, whereas the removed functions checked argument
types and accepted ‘NULL’ positional and keyword
arguments. To replace
‘PyEval_CallObjectWithKeywords(func, NULL, kwargs)’ with
*note PyObject_Call(): 398, pass an empty tuple as
positional arguments using *note PyTuple_New(0): 399.
* ‘PyEval_CallFunction()’: Use *note PyObject_CallFunction():
39a. instead.
* ‘PyEval_CallMethod()’: Use *note PyObject_CallMethod(): 39b.
instead.
* ‘PyCFunction_Call()’: Use *note PyObject_Call(): 398. instead.
(Contributed by Victor Stinner in gh-105107(3).)
* Remove the following old functions to configure the Python
initialization, deprecated in Python 3.11:
* ‘PySys_AddWarnOptionUnicode()’: Use *note
PyConfig.warnoptions: 39c. instead.
* ‘PySys_AddWarnOption()’: Use *note PyConfig.warnoptions: 39c.
instead.
* ‘PySys_AddXOption()’: Use *note PyConfig.xoptions: 39d.
instead.
* ‘PySys_HasWarnOptions()’: Use *note PyConfig.xoptions: 39d.
instead.
* ‘PySys_SetPath()’: Set *note PyConfig.module_search_paths:
39e. instead.
* ‘Py_SetPath()’: Set *note PyConfig.module_search_paths: 39e.
instead.
* ‘Py_SetStandardStreamEncoding()’: Set *note
PyConfig.stdio_encoding: 39f. instead, and set also maybe
*note PyConfig.legacy_windows_stdio: 3a0. (on Windows).
* ‘_Py_SetProgramFullPath()’: Set *note PyConfig.executable:
3a1. instead.
Use the new *note PyConfig: 3a2. API of the *note Python
Initialization Configuration: 3a3. instead ( PEP 587(4)), added to
Python 3.8. (Contributed by Victor Stinner in gh-105145(5).)
* Remove ‘PyEval_AcquireLock()’ and ‘PyEval_ReleaseLock()’ functions,
deprecated in Python 3.2. They didn’t update the current thread
state. They can be replaced with:
* *note PyEval_SaveThread(): 3a4. and *note
PyEval_RestoreThread(): 3a5.;
* low-level *note PyEval_AcquireThread(): 3a6. and *note
PyEval_RestoreThread(): 3a5.;
* or *note PyGILState_Ensure(): 3a7. and *note
PyGILState_Release(): 3a8.
(Contributed by Victor Stinner in gh-105182(6).)
* Remove the ‘PyEval_ThreadsInitialized()’ function, deprecated in
Python 3.9. Since Python 3.7, ‘Py_Initialize()’ always creates the
GIL: calling ‘PyEval_InitThreads()’ does nothing and
‘PyEval_ThreadsInitialized()’ always returns non-zero.
(Contributed by Victor Stinner in gh-105182(7).)
* Remove the ‘_PyInterpreterState_Get()’ alias to *note
PyInterpreterState_Get(): 3a9. which was kept for backward
compatibility with Python 3.8. The pythoncapi-compat project(8)
can be used to get *note PyInterpreterState_Get(): 3a9. on Python
3.8 and older. (Contributed by Victor Stinner in gh-106320(9).)
* Remove the private ‘_PyObject_FastCall()’ function: use
‘PyObject_Vectorcall()’ which is available since Python 3.8 ( PEP
590(10)). (Contributed by Victor Stinner in gh-106023(11).)
* Remove the ‘cpython/pytime.h’ header file, which only contained
private functions. (Contributed by Victor Stinner in
gh-106316(12).)
* Remove the undocumented ‘PY_TIMEOUT_MAX’ constant from the limited
C API. (Contributed by Victor Stinner in gh-110014(13).)
* Remove the old trashcan macros ‘Py_TRASHCAN_SAFE_BEGIN’ and
‘Py_TRASHCAN_SAFE_END’. Replace both with the new macros
‘Py_TRASHCAN_BEGIN’ and ‘Py_TRASHCAN_END’. (Contributed by Irit
Katriel in gh-105111(14).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/106320
(2) https://github.com/python/cpython/issues/85275
(3) https://github.com/python/cpython/issues/105107
(4) https://peps.python.org/pep-0587/
(5) https://github.com/python/cpython/issues/105145
(6) https://github.com/python/cpython/issues/105182
(7) https://github.com/python/cpython/issues/105182
(8) https://github.com/python/pythoncapi-compat/
(9) https://github.com/python/cpython/issues/106320
(10) https://peps.python.org/pep-0590/
(11) https://github.com/python/cpython/issues/106023
(12) https://github.com/python/cpython/issues/106316
(13) https://github.com/python/cpython/issues/110014
(14) https://github.com/python/cpython/issues/105111
File: python.info, Node: Deprecated C APIs, Prev: Removed C APIs, Up: C API Changes
1.1.10.5 Deprecated C APIs
..........................
* Deprecate old Python initialization functions:
* *note PySys_ResetWarnOptions(): 3ab.: Clear *note
sys.warnoptions: 3ac. and ‘warnings.filters’ instead.
* *note Py_GetExecPrefix(): 3ad.: Get *note sys.exec_prefix:
3ae. instead.
* *note Py_GetPath(): 3af.: Get *note sys.path: 3b0. instead.
* *note Py_GetPrefix(): 3b1.: Get *note sys.prefix: 3b2.
instead.
* *note Py_GetProgramFullPath(): 3b3.: Get *note sys.executable:
3b4. instead.
* *note Py_GetProgramName(): 3b5.: Get *note sys.executable:
3b4. instead.
* *note Py_GetPythonHome(): 3b6.: Get *note PyConfig.home: 3b7.
or the *note PYTHONHOME: 3b8. environment variable instead.
(Contributed by Victor Stinner in gh-105145(1).)
* *note Soft deprecate: 20b. the *note PyEval_GetBuiltins(): 34c,
*note PyEval_GetGlobals(): 34e, and *note PyEval_GetLocals(): 350.
functions, which return a *note borrowed reference: 339. (Soft
deprecated as part of PEP 667(2).)
* Deprecate the *note PyImport_ImportModuleNoBlock(): 3b9. function,
which is just an alias to *note PyImport_ImportModule(): 3ba. since
Python 3.3. (Contributed by Victor Stinner in gh-105396(3).)
* *note Soft deprecate: 20b. the *note PyModule_AddObject(): 361.
function. It should be replaced with *note PyModule_Add(): 35f. or
*note PyModule_AddObjectRef(): 360. (Contributed by Serhiy
Storchaka in gh-86493(4).)
* Deprecate the old ‘Py_UNICODE’ and ‘PY_UNICODE_TYPE’ types and the
‘Py_UNICODE_WIDE’ define. Use the ‘wchar_t’ type directly instead.
Since Python 3.3, ‘Py_UNICODE’ and ‘PY_UNICODE_TYPE’ are just
aliases to ‘wchar_t’. (Contributed by Victor Stinner in
gh-105156(5).)
* Deprecate the *note PyWeakref_GetObject(): 374. and *note
PyWeakref_GET_OBJECT(): 3bb. functions, which return a *note
borrowed reference: 339. Replace them with the new *note
PyWeakref_GetRef(): 373. function, which returns a *note strong
reference: 338. The pythoncapi-compat project(6) can be used to
get *note PyWeakref_GetRef(): 373. on Python 3.12 and older.
(Contributed by Victor Stinner in gh-105927(7).)
* Menu:
* Pending Removal in Python 3.14: Pending Removal in Python 3 14<2>.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15<2>.
* Pending removal in Python 3.16: Pending removal in Python 3 16<2>.
* Pending Removal in Future Versions: Pending Removal in Future Versions<2>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/105145
(2) https://peps.python.org/pep-0667/
(3) https://github.com/python/cpython/issues/105396
(4) https://github.com/python/cpython/issues/86493
(5) https://github.com/python/cpython/issues/105156
(6) https://github.com/python/pythoncapi-compat/
(7) https://github.com/python/cpython/issues/105927
File: python.info, Node: Pending Removal in Python 3 14<2>, Next: Pending Removal in Python 3 15<2>, Up: Deprecated C APIs
1.1.10.6 Pending Removal in Python 3.14
.......................................
* The ‘ma_version_tag’ field in *note PyDictObject: 3bd. for
extension modules ( PEP 699(1); gh-101193(2)).
* Creating *note immutable types: 3be. with mutable bases
(gh-95388(3)).
* Functions to configure Python’s initialization, deprecated in
Python 3.11:
* ‘PySys_SetArgvEx()’: Set *note PyConfig.argv: 3bf. instead.
* ‘PySys_SetArgv()’: Set *note PyConfig.argv: 3bf. instead.
* ‘Py_SetProgramName()’: Set *note PyConfig.program_name: 3c0.
instead.
* ‘Py_SetPythonHome()’: Set *note PyConfig.home: 3b7. instead.
The *note Py_InitializeFromConfig(): 3c1. API should be used with
*note PyConfig: 3a2. instead.
* Global configuration variables:
* *note Py_DebugFlag: 3c2.: Use *note PyConfig.parser_debug:
3c3. instead.
* *note Py_VerboseFlag: 3c4.: Use *note PyConfig.verbose: 3c5.
instead.
* *note Py_QuietFlag: 3c6.: Use *note PyConfig.quiet: 3c7.
instead.
* *note Py_InteractiveFlag: 3c8.: Use *note
PyConfig.interactive: 3c9. instead.
* *note Py_InspectFlag: 3ca.: Use *note PyConfig.inspect: 3cb.
instead.
* *note Py_OptimizeFlag: 3cc.: Use *note
PyConfig.optimization_level: 3cd. instead.
* *note Py_NoSiteFlag: 3ce.: Use *note PyConfig.site_import:
3cf. instead.
* *note Py_BytesWarningFlag: 3d0.: Use *note
PyConfig.bytes_warning: 3d1. instead.
* *note Py_FrozenFlag: 3d2.: Use *note
PyConfig.pathconfig_warnings: 3d3. instead.
* *note Py_IgnoreEnvironmentFlag: 3d4.: Use *note
PyConfig.use_environment: 3d5. instead.
* *note Py_DontWriteBytecodeFlag: 3d6.: Use *note
PyConfig.write_bytecode: 3d7. instead.
* *note Py_NoUserSiteDirectory: 3d8.: Use *note
PyConfig.user_site_directory: 3d9. instead.
* *note Py_UnbufferedStdioFlag: 3da.: Use *note
PyConfig.buffered_stdio: 3db. instead.
* *note Py_HashRandomizationFlag: 3dc.: Use *note
PyConfig.use_hash_seed: 3dd. and *note PyConfig.hash_seed:
3de. instead.
* *note Py_IsolatedFlag: 3df.: Use *note PyConfig.isolated: 3e0.
instead.
* *note Py_LegacyWindowsFSEncodingFlag: 3e1.: Use *note
PyPreConfig.legacy_windows_fs_encoding: 3e2. instead.
* *note Py_LegacyWindowsStdioFlag: 3e3.: Use *note
PyConfig.legacy_windows_stdio: 3a0. instead.
* ‘Py_FileSystemDefaultEncoding’: Use *note
PyConfig.filesystem_encoding: 3e4. instead.
* ‘Py_HasFileSystemDefaultEncoding’: Use *note
PyConfig.filesystem_encoding: 3e4. instead.
* ‘Py_FileSystemDefaultEncodeErrors’: Use *note
PyConfig.filesystem_errors: 3e5. instead.
* ‘Py_UTF8Mode’: Use *note PyPreConfig.utf8_mode: 3e6. instead.
(see *note Py_PreInitialize(): 3e7.)
The *note Py_InitializeFromConfig(): 3c1. API should be used with
*note PyConfig: 3a2. instead.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0699/
(2) https://github.com/python/cpython/issues/101193
(3) https://github.com/python/cpython/issues/95388
File: python.info, Node: Pending Removal in Python 3 15<2>, Next: Pending removal in Python 3 16<2>, Prev: Pending Removal in Python 3 14<2>, Up: Deprecated C APIs
1.1.10.7 Pending Removal in Python 3.15
.......................................
* The *note PyImport_ImportModuleNoBlock(): 3b9.: Use *note
PyImport_ImportModule(): 3ba. instead.
* *note PyWeakref_GetObject(): 374. and *note PyWeakref_GET_OBJECT():
3bb.: Use *note PyWeakref_GetRef(): 373. instead.
* *note Py_UNICODE: 3e9. type and the ‘Py_UNICODE_WIDE’ macro: Use
‘wchar_t’ instead.
* Python initialization functions:
* *note PySys_ResetWarnOptions(): 3ab.: Clear *note
sys.warnoptions: 3ac. and ‘warnings.filters’ instead.
* *note Py_GetExecPrefix(): 3ad.: Get *note
sys.base_exec_prefix: 3ea. and *note sys.exec_prefix: 3ae.
instead.
* *note Py_GetPath(): 3af.: Get *note sys.path: 3b0. instead.
* *note Py_GetPrefix(): 3b1.: Get *note sys.base_prefix: 3eb.
and *note sys.prefix: 3b2. instead.
* *note Py_GetProgramFullPath(): 3b3.: Get *note sys.executable:
3b4. instead.
* *note Py_GetProgramName(): 3b5.: Get *note sys.executable:
3b4. instead.
* *note Py_GetPythonHome(): 3b6.: Get *note PyConfig.home: 3b7.
or the *note PYTHONHOME: 3b8. environment variable instead.
File: python.info, Node: Pending removal in Python 3 16<2>, Next: Pending Removal in Future Versions<2>, Prev: Pending Removal in Python 3 15<2>, Up: Deprecated C APIs
1.1.10.8 Pending removal in Python 3.16
.......................................
* The bundled copy of ‘libmpdec’.
File: python.info, Node: Pending Removal in Future Versions<2>, Prev: Pending removal in Python 3 16<2>, Up: Deprecated C APIs
1.1.10.9 Pending Removal in Future Versions
...........................................
The following APIs are deprecated and will be removed, although there is
currently no date scheduled for their removal.
* *note Py_TPFLAGS_HAVE_FINALIZE: 3ee.: Unneeded since Python 3.8.
* *note PyErr_Fetch(): 3ef.: Use *note PyErr_GetRaisedException():
3f0. instead.
* *note PyErr_NormalizeException(): 3f1.: Use *note
PyErr_GetRaisedException(): 3f0. instead.
* *note PyErr_Restore(): 3f2.: Use *note PyErr_SetRaisedException():
3f3. instead.
* *note PyModule_GetFilename(): 3f4.: Use *note
PyModule_GetFilenameObject(): 3f5. instead.
* *note PyOS_AfterFork(): 3f6.: Use *note PyOS_AfterFork_Child():
3f7. instead.
* *note PySlice_GetIndicesEx(): 3f8.: Use *note PySlice_Unpack():
3f9. and *note PySlice_AdjustIndices(): 3fa. instead.
* ‘PyUnicode_AsDecodedObject()’: Use *note PyCodec_Decode(): 3fb.
instead.
* ‘PyUnicode_AsDecodedUnicode()’: Use *note PyCodec_Decode(): 3fb.
instead.
* ‘PyUnicode_AsEncodedObject()’: Use *note PyCodec_Encode(): 3fc.
instead.
* ‘PyUnicode_AsEncodedUnicode()’: Use *note PyCodec_Encode(): 3fc.
instead.
* *note PyUnicode_READY(): 3fd.: Unneeded since Python 3.12
* ‘PyErr_Display()’: Use *note PyErr_DisplayException(): 3fe.
instead.
* ‘_PyErr_ChainExceptions()’: Use ‘_PyErr_ChainExceptions1()’
instead.
* ‘PyBytesObject.ob_shash’ member: call *note PyObject_Hash(): 3ff.
instead.
* ‘PyDictObject.ma_version_tag’ member.
* Thread Local Storage (TLS) API:
* *note PyThread_create_key(): 400.: Use *note
PyThread_tss_alloc(): 401. instead.
* *note PyThread_delete_key(): 402.: Use *note
PyThread_tss_free(): 403. instead.
* *note PyThread_set_key_value(): 404.: Use *note
PyThread_tss_set(): 405. instead.
* *note PyThread_get_key_value(): 406.: Use *note
PyThread_tss_get(): 407. instead.
* *note PyThread_delete_key_value(): 408.: Use *note
PyThread_tss_delete(): 409. instead.
* *note PyThread_ReInitTLS(): 40a.: Unneeded since Python 3.7.
File: python.info, Node: Build Changes, Next: Porting to Python 3 13, Prev: C API Changes, Up: What’s New In Python 3 13
1.1.11 Build Changes
--------------------
* ‘arm64-apple-ios’ and ‘arm64-apple-ios-simulator’ are both now PEP
11(1) tier 3 platforms. (*note PEP 730: 165. written and
implementation contributed by Russell Keith-Magee in gh-114099(2).)
* ‘aarch64-linux-android’ and ‘x86_64-linux-android’ are both now PEP
11(3) tier 3 platforms. (*note PEP 738: 165. written and
implementation contributed by Malcolm Smith in gh-116622(4).)
* ‘wasm32-wasi’ is now a PEP 11(5) tier 2 platform. (Contributed by
Brett Cannon in gh-115192(6).)
* ‘wasm32-emscripten’ is no longer a PEP 11(7) supported platform.
(Contributed by Brett Cannon in gh-115192(8).)
* Building CPython now requires a compiler with support for the C11
atomic library, GCC built-in atomic functions, or MSVC interlocked
intrinsics.
* Autoconf 2.71 and aclocal 1.16.5 are now required to regenerate the
‘configure’ script. (Contributed by Christian Heimes in
gh-89886(9) and by Victor Stinner in gh-112090(10).)
* SQLite 3.15.2 or newer is required to build the *note sqlite3: cf.
extension module. (Contributed by Erlend Aasland in
gh-105875(11).)
* CPython now bundles the mimalloc library(12) by default. It is
licensed under the MIT license; see *note mimalloc license: 40d.
The bundled mimalloc has custom changes, see gh-113141(13) for
details. (Contributed by Dino Viehland in gh-109914(14).)
* The ‘configure’ option *note -with-system-libmpdec: 40e. now
defaults to ‘yes’. The bundled copy of ‘libmpdec’ will be removed
in Python 3.16.
* Python built with ‘configure’ *note -with-trace-refs: 390. (tracing
references) is now ABI compatible with the Python release build and
*note debug build: 1fb. (Contributed by Victor Stinner in
gh-108634(15).)
* On POSIX systems, the pkg-config (‘.pc’) filenames now include the
ABI flags. For example, the free-threaded build generates
‘python-3.13t.pc’ and the debug build generates ‘python-3.13d.pc’.
* The ‘errno’, ‘fcntl’, ‘grp’, ‘md5’, ‘pwd’, ‘resource’, ‘termios’,
‘winsound’, ‘_ctypes_test’, ‘_multiprocessing.posixshmem’,
‘_scproxy’, ‘_stat’, ‘_statistics’, ‘_testconsole’,
‘_testimportmultiple’ and ‘_uuid’ C extensions are now built with
the *note limited C API: 391. (Contributed by Victor Stinner in
gh-85283(16).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0011/
(2) https://github.com/python/cpython/issues/114099
(3) https://peps.python.org/pep-0011/
(4) https://github.com/python/cpython/issues/116622
(5) https://peps.python.org/pep-0011/
(6) https://github.com/python/cpython/issues/115192
(7) https://peps.python.org/pep-0011/
(8) https://github.com/python/cpython/issues/115192
(9) https://github.com/python/cpython/issues/89886
(10) https://github.com/python/cpython/issues/112090
(11) https://github.com/python/cpython/issues/105875
(12) https://github.com/microsoft/mimalloc/
(13) https://github.com/python/cpython/issues/113141
(14) https://github.com/python/cpython/issues/109914
(15) https://github.com/python/cpython/issues/108634
(16) https://github.com/python/cpython/issues/85283
File: python.info, Node: Porting to Python 3 13, Next: Regression Test Changes, Prev: Build Changes, Up: What’s New In Python 3 13
1.1.12 Porting to Python 3.13
-----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in the Python API::
* Changes in the C API::
File: python.info, Node: Changes in the Python API, Next: Changes in the C API, Up: Porting to Python 3 13
1.1.12.1 Changes in the Python API
..................................
* *note PEP 667: 142. introduces several changes to the semantics of
*note locals(): 141. and *note f_locals: 182.:
* Calling *note locals(): 141. in an *note optimized scope: 17e.
now produces an independent snapshot on each call, and hence
no longer implicitly updates previously returned references.
Obtaining the legacy CPython behavior now requires explicit
calls to update the initially returned dictionary with the
results of subsequent calls to ‘locals()’. Code execution
functions that implicitly target ‘locals()’ (such as ‘exec’
and ‘eval’) must be passed an explicit namespace to access
their results in an optimized scope. (Changed as part of PEP
667(1).)
* Calling *note locals(): 141. from a comprehension at module or
class scope (including via ‘exec’ or ‘eval’) once more behaves
as if the comprehension were running as an independent nested
function (i.e. the local variables from the containing scope
are not included). In Python 3.12, this had changed to
include the local variables from the containing scope when
implementing PEP 709(2). (Changed as part of PEP 667(3).)
* Accessing *note FrameType.f_locals: 182. in an *note optimized
scope: 17e. now returns a write-through proxy rather than a
snapshot that gets updated at ill-specified times. If a
snapshot is desired, it must be created explicitly with ‘dict’
or the proxy’s ‘.copy()’ method. (Changed as part of PEP
667(4).)
* *note functools.partial: 410. now emits a *note FutureWarning: 411.
when used as a method. The behavior will change in future Python
versions. Wrap it in *note staticmethod(): 412. if you want to
preserve the old behavior. (Contributed by Serhiy Storchaka in
gh-121027(5).)
* An *note OSError: 413. is now raised by *note getpass.getuser():
414. for any failure to retrieve a username, instead of *note
ImportError: 415. on non-Unix platforms or *note KeyError: 33f. on
Unix platforms where the password database is empty.
* The value of the ‘mode’ attribute of *note gzip.GzipFile: 416. is
now a string (‘'rb'’ or ‘'wb'’) instead of an integer (‘1’ or ‘2’).
The value of the ‘mode’ attribute of the readable file-like object
returned by *note zipfile.ZipFile.open(): 417. is now ‘'rb'’
instead of ‘'r'’. (Contributed by Serhiy Storchaka in
gh-115961(6).)
* *note mailbox.Maildir: 418. now ignores files with a leading dot
(‘.’). (Contributed by Zackery Spytz in gh-65559(7).)
* *note pathlib.Path.glob(): 22f. and *note rglob(): 230. now return
both files and directories if a pattern that ends with “‘**’” is
given, rather than directories only. Add a trailing slash to keep
the previous behavior and only match directories.
* The *note threading: ed. module now expects the ‘_thread’ module to
have an ‘_is_main_interpreter()’ function. This function takes no
arguments and returns ‘True’ if the current interpreter is the main
interpreter.
Any library or application that provides a custom ‘_thread’ module
must provide ‘_is_main_interpreter()’, just like the module’s other
“private” attributes. (gh-112826(8).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0667/
(2) https://peps.python.org/pep-0709/
(3) https://peps.python.org/pep-0667/
(4) https://peps.python.org/pep-0667/
(5) https://github.com/python/cpython/issues/121027
(6) https://github.com/python/cpython/issues/115961
(7) https://github.com/python/cpython/issues/65559
(8) https://github.com/python/cpython/issues/112826
File: python.info, Node: Changes in the C API, Prev: Changes in the Python API, Up: Porting to Python 3 13
1.1.12.2 Changes in the C API
.............................
* ‘Python.h’ no longer includes the ‘’ standard header. It
was included for the ‘finite()’ function which is now provided by
the ‘’ header. It should now be included explicitly if
needed. Remove also the ‘HAVE_IEEEFP_H’ macro. (Contributed by
Victor Stinner in gh-108765(1).)
* ‘Python.h’ no longer includes these standard header files:
‘’, ‘’ and ‘’. If needed, they
should now be included explicitly. For example, ‘’
provides the ‘clock()’ and ‘gmtime()’ functions, ‘’
provides the ‘select()’ function, and ‘’ provides the
‘futimes()’, ‘gettimeofday()’ and ‘setitimer()’ functions.
(Contributed by Victor Stinner in gh-108765(2).)
* On Windows, ‘Python.h’ no longer includes the ‘’ standard
header file. If needed, it should now be included explicitly. For
example, it provides ‘offsetof()’ function, and ‘size_t’ and
‘ptrdiff_t’ types. Including ‘’ explicitly was already
needed by all other platforms, the ‘HAVE_STDDEF_H’ macro is only
defined on Windows. (Contributed by Victor Stinner in
gh-108765(3).)
* If the *note Py_LIMITED_API: 41a. macro is defined,
‘Py_BUILD_CORE’, ‘Py_BUILD_CORE_BUILTIN’ and ‘Py_BUILD_CORE_MODULE’
macros are now undefined by ‘’. (Contributed by Victor
Stinner in gh-85283(4).)
* The old trashcan macros ‘Py_TRASHCAN_SAFE_BEGIN’ and
‘Py_TRASHCAN_SAFE_END’ were removed. They should be replaced by
the new macros ‘Py_TRASHCAN_BEGIN’ and ‘Py_TRASHCAN_END’.
A ‘tp_dealloc’ function that has the old macros, such as:
static void
mytype_dealloc(mytype *p)
{
PyObject_GC_UnTrack(p);
Py_TRASHCAN_SAFE_BEGIN(p);
...
Py_TRASHCAN_SAFE_END
}
should migrate to the new macros as follows:
static void
mytype_dealloc(mytype *p)
{
PyObject_GC_UnTrack(p);
Py_TRASHCAN_BEGIN(p, mytype_dealloc)
...
Py_TRASHCAN_END
}
Note that ‘Py_TRASHCAN_BEGIN’ has a second argument which should be
the deallocation function it is in. The new macros were added in
Python 3.8 and the old macros were deprecated in Python 3.11.
(Contributed by Irit Katriel in gh-105111(5).)
* *note PEP 667: 142. introduces several changes to frame-related
functions:
* The effects of mutating the dictionary returned from *note
PyEval_GetLocals(): 350. in an *note optimized scope: 17e.
have changed. New dict entries added this way will now 'only'
be visible to subsequent *note PyEval_GetLocals(): 350. calls
in that frame, as *note PyFrame_GetLocals(): 41b, *note
locals(): 141, and *note FrameType.f_locals: 182. no longer
access the same underlying cached dictionary. Changes made to
entries for actual variable names and names added via the
write-through proxy interfaces will be overwritten on
subsequent calls to *note PyEval_GetLocals(): 350. in that
frame. The recommended code update depends on how the
function was being used, so refer to the deprecation notice on
the function for details.
* Calling *note PyFrame_GetLocals(): 41b. in an *note optimized
scope: 17e. now returns a write-through proxy rather than a
snapshot that gets updated at ill-specified times. If a
snapshot is desired, it must be created explicitly (e.g. with
*note PyDict_Copy(): 41c.), or by calling the new *note
PyEval_GetFrameLocals(): 34f. API.
* ‘PyFrame_FastToLocals()’ and ‘PyFrame_FastToLocalsWithError()’
no longer have any effect. Calling these functions has been
redundant since Python 3.11, when *note PyFrame_GetLocals():
41b. was first introduced.
* ‘PyFrame_LocalsToFast()’ no longer has any effect. Calling
this function is redundant now that *note PyFrame_GetLocals():
41b. returns a write-through proxy for *note optimized scopes:
17e.
* Python 3.13 removed many private functions. Some of them can be
replaced using these alternatives:
* ‘_PyDict_Pop()’: *note PyDict_Pop(): 33c. or *note
PyDict_PopString(): 33d.;
* ‘_PyDict_GetItemWithError()’: *note PyDict_GetItemRef(): 335.;
* ‘_PyErr_WriteUnraisableMsg()’: *note PyErr_FormatUnraisable():
349.;
* ‘_PyEval_SetTrace()’: *note PyEval_SetTrace(): 41d. or *note
PyEval_SetTraceAllThreads(): 41e.;
* ‘_PyList_Extend()’: *note PyList_Extend(): 358.;
* ‘_PyLong_AsInt()’: *note PyLong_AsInt(): 35a.;
* ‘_PyMem_RawStrdup()’: ‘strdup()’;
* ‘_PyMem_Strdup()’: ‘strdup()’;
* ‘_PyObject_ClearManagedDict()’: *note
PyObject_ClearManagedDict(): 365.;
* ‘_PyObject_VisitManagedDict()’: *note
PyObject_VisitManagedDict(): 364.;
* ‘_PyThreadState_UncheckedGet()’: *note
PyThreadState_GetUnchecked(): 36c.;
* ‘_PyTime_AsSecondsDouble()’: *note PyTime_AsSecondsDouble():
32b.;
* ‘_PyTime_GetMonotonicClock()’: *note PyTime_Monotonic(): 32c.
or *note PyTime_MonotonicRaw(): 32d.;
* ‘_PyTime_GetPerfCounter()’: *note PyTime_PerfCounter(): 32e.
or *note PyTime_PerfCounterRaw(): 32f.;
* ‘_PyTime_GetSystemClock()’: *note PyTime_Time(): 330. or *note
PyTime_TimeRaw(): 331.;
* ‘_PyTime_MAX’: *note PyTime_MAX: 32a.;
* ‘_PyTime_MIN’: *note PyTime_MIN: 329.;
* ‘_PyTime_t’: *note PyTime_t: 328.;
* ‘_Py_HashPointer()’: *note Py_HashPointer(): 363.;
* ‘_Py_IsFinalizing()’: *note Py_IsFinalizing(): 355.
The pythoncapi-compat project(6) can be used to get most of these
new functions on Python 3.12 and older.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/108765
(2) https://github.com/python/cpython/issues/108765
(3) https://github.com/python/cpython/issues/108765
(4) https://github.com/python/cpython/issues/85283
(5) https://github.com/python/cpython/issues/105111
(6) https://github.com/python/pythoncapi-compat/
File: python.info, Node: Regression Test Changes, Next: Notable changes in 3 13 1, Prev: Porting to Python 3 13, Up: What’s New In Python 3 13
1.1.13 Regression Test Changes
------------------------------
* Python built with ‘configure’ *note -with-pydebug: 420. now
supports a *note -X presite=package.module: 176. command-line
option. If used, it specifies a module that should be imported
early in the lifecycle of the interpreter, before ‘site.py’ is
executed. (Contributed by Łukasz Langa in gh-110769(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/110769
File: python.info, Node: Notable changes in 3 13 1, Next: Notable changes in 3 13 4, Prev: Regression Test Changes, Up: What’s New In Python 3 13
1.1.14 Notable changes in 3.13.1
--------------------------------
* Menu:
* sys: sys<2>.
File: python.info, Node: sys<2>, Up: Notable changes in 3 13 1
1.1.14.1 sys
............
* The previously undocumented special function *note
sys.getobjects(): 423, which only exists in specialized builds of
Python, may now return objects from other interpreters than the one
it’s called in.
File: python.info, Node: Notable changes in 3 13 4, Prev: Notable changes in 3 13 1, Up: What’s New In Python 3 13
1.1.15 Notable changes in 3.13.4
--------------------------------
* Menu:
* os.path: os path<2>.
* tarfile::
File: python.info, Node: os path<2>, Next: tarfile, Up: Notable changes in 3 13 4
1.1.15.1 os.path
................
* The 'strict' parameter to *note os.path.realpath(): 227. accepts a
new value, *note os.path.ALLOW_MISSING: 426. If used, errors other
than *note FileNotFoundError: 427. will be re-raised; the resulting
path can be missing but it will be free of symlinks. (Contributed
by Petr Viktorin for CVE 2025-4517(1).)
---------- Footnotes ----------
(1) https://www.cve.org/CVERecord?id=CVE-2025-4517
File: python.info, Node: tarfile, Prev: os path<2>, Up: Notable changes in 3 13 4
1.1.15.2 tarfile
................
* *note data_filter(): 429. now normalizes symbolic link targets in
order to avoid path traversal attacks.Add commentMore actions
(Contributed by Petr Viktorin in gh-127987(1) and CVE
2025-4138(2).)
* *note extractall(): 42a. now skips fixing up directory attributes
when a directory was removed or replaced by another kind of file.
(Contributed by Petr Viktorin in gh-127987(3) and CVE
2024-12718(4).)
* *note extract(): 42b. and *note extractall(): 42a. now (re-)apply
the extraction filter when substituting a link (hard or symbolic)
with a copy of another archive member, and when fixing up directory
attributes. The former raises a new exception, *note
LinkFallbackError: 42c. (Contributed by Petr Viktorin for CVE
2025-4330(5) and CVE 2024-12718(6).)
* *note extract(): 42b. and *note extractall(): 42a. no longer
extract rejected members when *note errorlevel(): 42d. is zero.
(Contributed by Matt Prodani and Petr Viktorin in gh-112887(7) and
CVE 2025-4435(8).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/127987
(2) https://www.cve.org/CVERecord?id=CVE-2025-4138
(3) https://github.com/python/cpython/issues/127987
(4) https://www.cve.org/CVERecord?id=CVE-2024-12718
(5) https://www.cve.org/CVERecord?id=CVE-2025-4330
(6) https://www.cve.org/CVERecord?id=CVE-2024-12718
(7) https://github.com/python/cpython/issues/112887
(8) https://www.cve.org/CVERecord?id=CVE-2025-4435
File: python.info, Node: What’s New In Python 3 12, Next: What’s New In Python 3 11, Prev: What’s New In Python 3 13, Up: What’s New in Python
1.2 What’s New In Python 3.12
=============================
Editor: Adam Turner
This article explains the new features in Python 3.12, compared to 3.11.
Python 3.12 was released on October 2, 2023. For full details, see the
*note changelog: 13c.
See also
........
PEP 693(1) – Python 3.12 Release Schedule
* Menu:
* Summary – Release highlights::
* New Features: New Features<3>.
* New Features Related to Type Hints::
* Other Language Changes: Other Language Changes<2>.
* New Modules: New Modules<2>.
* Improved Modules: Improved Modules<2>.
* Optimizations: Optimizations<2>.
* CPython bytecode changes::
* Demos and Tools::
* Deprecated::
* Removed::
* Porting to Python 3.12: Porting to Python 3 12.
* Build Changes: Build Changes<2>.
* C API Changes: C API Changes<2>.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0693/
File: python.info, Node: Summary – Release highlights, Next: New Features<3>, Up: What’s New In Python 3 12
1.2.1 Summary – Release highlights
----------------------------------
Python 3.12 is a stable release of the Python programming language, with
a mix of changes to the language and the standard library. The library
changes focus on cleaning up deprecated APIs, usability, and
correctness. Of note, the ‘distutils’ package has been removed from the
standard library. Filesystem support in *note os: a1. and *note
pathlib: a4. has seen a number of improvements, and several modules have
better performance.
The language changes focus on usability, as *note f-strings: 431. have
had many limitations removed and ‘Did you mean …’ suggestions continue
to improve. The new *note type parameter syntax: 432. and *note type:
433. statement improve ergonomics for using *note generic types: 434.
and *note type aliases: 435. with static type checkers.
This article doesn’t attempt to provide a complete specification of all
new features, but instead gives a convenient overview. For full
details, you should refer to the documentation, such as the *note
Library Reference: 144. and *note Language Reference: 145. If you want
to understand the complete implementation and design rationale for a
change, refer to the PEP for a particular new feature; but note that
PEPs usually are not kept up-to-date once a feature has been fully
implemented.
__________________________________________________________________
New syntax features:
* *note PEP 695: 432, type parameter syntax and the *note type: 433.
statement
New grammar features:
* *note PEP 701: 436, *note f-strings: 431. in the grammar
Interpreter improvements:
* *note PEP 684: 437, a unique per-interpreter *note GIL: 148.
* *note PEP 669: 438, low impact monitoring
* *note Improved 'Did you mean ...' suggestions: 439.for *note
NameError: 43a, *note ImportError: 415, and *note SyntaxError: 18d.
exceptions
Python data model improvements:
* *note PEP 688: 43b, using the *note buffer protocol: 393. from
Python
Significant improvements in the standard library:
* The *note pathlib.Path: 22b. class now supports subclassing
* The *note os: a1. module received several improvements for Windows
support
* A *note command-line interface: 43c. has been added to the *note
sqlite3: cf. module
* *note isinstance(): 43d. checks against *note runtime-checkable
protocols: 43e. enjoy a speed up of between two and 20 times
* The *note asyncio: a. package has had a number of performance
improvements, with some benchmarks showing a 75% speed up.
* A *note command-line interface: 43f. has been added to the *note
uuid: 110. module
* Due to the changes in *note PEP 701: 436, producing tokens via the
*note tokenize: fb. module is up to 64% faster.
Security improvements:
* Replace the builtin *note hashlib: 68. implementations of SHA1,
SHA3, SHA2-384, SHA2-512, and MD5 with formally verified code from
the HACL*(1) project. These builtin implementations remain as
fallbacks that are only used when OpenSSL does not provide them.
C API improvements:
* *note PEP 697: 440, unstable C API tier
* *note PEP 683: 441, immortal objects
CPython implementation improvements:
* *note PEP 709: 442, comprehension inlining
* *note CPython support: 18f. for the Linux ‘perf’ profiler
* Implement stack overflow protection on supported platforms
New typing features:
* *note PEP 692: 443, using *note TypedDict: 162. to annotate *note
**kwargs: 444.
* *note PEP 698: 445, *note typing.override(): 446. decorator
Important deprecations, removals or restrictions:
* PEP 623(2): Remove ‘wstr’ from Unicode objects in Python’s C API,
reducing the size of every *note str: 447. object by at least 8
bytes.
* PEP 632(3): Remove the ‘distutils’ package. See the migration
guide(4) for advice replacing the APIs it provided. The
third-party Setuptools(5) package continues to provide ‘distutils’,
if you still require it in Python 3.12 and beyond.
* gh-95299(6): Do not pre-install ‘setuptools’ in virtual
environments created with *note venv: 111. This means that
‘distutils’, ‘setuptools’, ‘pkg_resources’, and ‘easy_install’ will
no longer available by default; to access these run ‘pip install
setuptools’ in the *note activated: 448. virtual environment.
* The ‘asynchat’, ‘asyncore’, and ‘imp’ modules have been removed,
along with several *note unittest.TestCase: 449. *note method
aliases: 44a..
---------- Footnotes ----------
(1) https://github.com/hacl-star/hacl-star/
(2) https://peps.python.org/pep-0623/
(3) https://peps.python.org/pep-0632/
(4) https://peps.python.org/pep-0632/#migration-advice
(5)
https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html
(6) https://github.com/python/cpython/issues/95299
File: python.info, Node: New Features<3>, Next: New Features Related to Type Hints, Prev: Summary – Release highlights, Up: What’s New In Python 3 12
1.2.2 New Features
------------------
* Menu:
* PEP 695; Type Parameter Syntax: PEP 695 Type Parameter Syntax.
* PEP 701; Syntactic formalization of f-strings: PEP 701 Syntactic formalization of f-strings.
* PEP 684; A Per-Interpreter GIL: PEP 684 A Per-Interpreter GIL.
* PEP 669; Low impact monitoring for CPython: PEP 669 Low impact monitoring for CPython.
* PEP 688; Making the buffer protocol accessible in Python: PEP 688 Making the buffer protocol accessible in Python.
* PEP 709; Comprehension inlining: PEP 709 Comprehension inlining.
* Improved Error Messages::
File: python.info, Node: PEP 695 Type Parameter Syntax, Next: PEP 701 Syntactic formalization of f-strings, Up: New Features<3>
1.2.2.1 PEP 695: Type Parameter Syntax
......................................
Generic classes and functions under PEP 484(1) were declared using a
verbose syntax that left the scope of type parameters unclear and
required explicit declarations of variance.
PEP 695(2) introduces a new, more compact and explicit way to create
*note generic classes: 44d. and *note functions: 44e.:
def max[T](args: Iterable[T]) -> T:
...
class list[T]:
def __getitem__(self, index: int, /) -> T:
...
def append(self, element: T) -> None:
...
In addition, the PEP introduces a new way to declare *note type aliases:
44f. using the *note type: 433. statement, which creates an instance of
*note TypeAliasType: 450.:
type Point = tuple[float, float]
Type aliases can also be *note generic: 451.:
type Point[T] = tuple[T, T]
The new syntax allows declaring *note TypeVarTuple: 15f. and *note
ParamSpec: 15e. parameters, as well as *note TypeVar: 15d. parameters
with bounds or constraints:
type IntFunc[**P] = Callable[P, int] # ParamSpec
type LabeledTuple[*Ts] = tuple[str, *Ts] # TypeVarTuple
type HashableSequence[T: Hashable] = Sequence[T] # TypeVar with bound
type IntOrStrSequence[T: (int, str)] = Sequence[T] # TypeVar with constraints
The value of type aliases and the bound and constraints of type
variables created through this syntax are evaluated only on demand (see
*note lazy evaluation: 452.). This means type aliases are able to refer
to other types defined later in the file.
Type parameters declared through a type parameter list are visible
within the scope of the declaration and any nested scopes, but not in
the outer scope. For example, they can be used in the type annotations
for the methods of a generic class or in the class body. However, they
cannot be used in the module scope after the class is defined. See
*note Type parameter lists: 2b5. for a detailed description of the
runtime semantics of type parameters.
In order to support these scoping semantics, a new kind of scope is
introduced, the *note annotation scope: 188. Annotation scopes behave
for the most part like function scopes, but interact differently with
enclosing class scopes. In Python 3.13, *note annotations: 453. will
also be evaluated in annotation scopes.
See PEP 695(3) for more details.
(PEP written by Eric Traut. Implementation by Jelle Zijlstra, Eric
Traut, and others in gh-103764(4).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0695/
(3) https://peps.python.org/pep-0695/
(4) https://github.com/python/cpython/issues/103764
File: python.info, Node: PEP 701 Syntactic formalization of f-strings, Next: PEP 684 A Per-Interpreter GIL, Prev: PEP 695 Type Parameter Syntax, Up: New Features<3>
1.2.2.2 PEP 701: Syntactic formalization of f-strings
.....................................................
PEP 701(1) lifts some restrictions on the usage of *note f-strings: 431.
Expression components inside f-strings can now be any valid Python
expression, including strings reusing the same quote as the containing
f-string, multi-line expressions, comments, backslashes, and unicode
escape sequences. Let’s cover these in detail:
* Quote reuse: in Python 3.11, reusing the same quotes as the
enclosing f-string raises a *note SyntaxError: 18d, forcing the
user to either use other available quotes (like using double quotes
or triple quotes if the f-string uses single quotes). In Python
3.12, you can now do things like this:
>>> songs = ['Take me back to Eden', 'Alkaline', 'Ascensionism']
>>> f"This is the playlist: {", ".join(songs)}"
'This is the playlist: Take me back to Eden, Alkaline, Ascensionism'
Note that before this change there was no explicit limit in how
f-strings can be nested, but the fact that string quotes cannot be
reused inside the expression component of f-strings made it
impossible to nest f-strings arbitrarily. In fact, this is the
most nested f-string that could be written:
>>> f"""{f'''{f'{f"{1+1}"}'}'''}"""
'2'
As now f-strings can contain any valid Python expression inside
expression components, it is now possible to nest f-strings
arbitrarily:
>>> f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}"
'2'
* Multi-line expressions and comments: In Python 3.11, f-string
expressions must be defined in a single line, even if the
expression within the f-string could normally span multiple lines
(like literal lists being defined over multiple lines), making them
harder to read. In Python 3.12 you can now define f-strings
spanning multiple lines, and add inline comments:
>>> f"This is the playlist: {", ".join([
... 'Take me back to Eden', # My, my, those eyes like fire
... 'Alkaline', # Not acid nor alkaline
... 'Ascensionism' # Take to the broken skies at last
... ])}"
'This is the playlist: Take me back to Eden, Alkaline, Ascensionism'
* Backslashes and unicode characters: before Python 3.12 f-string
expressions couldn’t contain any ‘\’ character. This also affected
unicode *note escape sequences: 455. (such as ‘\N{snowman}’) as
these contain the ‘\N’ part that previously could not be part of
expression components of f-strings. Now, you can define
expressions like this:
>>> print(f"This is the playlist: {"\n".join(songs)}")
This is the playlist: Take me back to Eden
Alkaline
Ascensionism
>>> print(f"This is the playlist: {"\N{BLACK HEART SUIT}".join(songs)}")
This is the playlist: Take me back to Eden♥Alkaline♥Ascensionism
See PEP 701(2) for more details.
As a positive side-effect of how this feature has been implemented (by
parsing f-strings with the PEG parser(3)), now error messages for
f-strings are more precise and include the exact location of the error.
For example, in Python 3.11, the following f-string raises a *note
SyntaxError: 18d.:
>>> my_string = f"{x z y}" + f"{1 + 1}"
File "", line 1
(x z y)
^^^
SyntaxError: f-string: invalid syntax. Perhaps you forgot a comma?
but the error message doesn’t include the exact location of the error
within the line and also has the expression artificially surrounded by
parentheses. In Python 3.12, as f-strings are parsed with the PEG
parser, error messages can be more precise and show the entire line:
>>> my_string = f"{x z y}" + f"{1 + 1}"
File "", line 1
my_string = f"{x z y}" + f"{1 + 1}"
^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?
(Contributed by Pablo Galindo, Batuhan Taskaya, Lysandros Nikolaou,
Cristián Maureira-Fredes and Marta Gómez in gh-102856(4). PEP written
by Pablo Galindo, Batuhan Taskaya, Lysandros Nikolaou and Marta Gómez).
---------- Footnotes ----------
(1) https://peps.python.org/pep-0701/
(2) https://peps.python.org/pep-0701/
(3) https://peps.python.org/pep-0617/
(4) https://github.com/python/cpython/issues/102856
File: python.info, Node: PEP 684 A Per-Interpreter GIL, Next: PEP 669 Low impact monitoring for CPython, Prev: PEP 701 Syntactic formalization of f-strings, Up: New Features<3>
1.2.2.3 PEP 684: A Per-Interpreter GIL
......................................
PEP 684(1) introduces a per-interpreter *note GIL: 148, so that
sub-interpreters may now be created with a unique GIL per interpreter.
This allows Python programs to take full advantage of multiple CPU
cores. This is currently only available through the C-API, though a
Python API is anticipated for 3.13(2).
Use the new *note Py_NewInterpreterFromConfig(): 457. function to create
an interpreter with its own GIL:
PyInterpreterConfig config = {
.check_multi_interp_extensions = 1,
.gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
if (PyStatus_Exception(status)) {
return -1;
}
/* The new interpreter is now active in the current thread. */
For further examples how to use the C-API for sub-interpreters with a
per-interpreter GIL, see ‘Modules/_xxsubinterpretersmodule.c’.
(Contributed by Eric Snow in gh-104210(3), etc.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0684/
(2) https://peps.python.org/pep-0554/
(3) https://github.com/python/cpython/issues/104210
File: python.info, Node: PEP 669 Low impact monitoring for CPython, Next: PEP 688 Making the buffer protocol accessible in Python, Prev: PEP 684 A Per-Interpreter GIL, Up: New Features<3>
1.2.2.4 PEP 669: Low impact monitoring for CPython
..................................................
PEP 669(1) defines a new *note API: da. for profilers, debuggers, and
other tools to monitor events in CPython. It covers a wide range of
events, including calls, returns, lines, exceptions, jumps, and more.
This means that you only pay for what you use, providing support for
near-zero overhead debuggers and coverage tools. See *note
sys.monitoring: da. for details.
(Contributed by Mark Shannon in gh-103082(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0669/
(2) https://github.com/python/cpython/issues/103082
File: python.info, Node: PEP 688 Making the buffer protocol accessible in Python, Next: PEP 709 Comprehension inlining, Prev: PEP 669 Low impact monitoring for CPython, Up: New Features<3>
1.2.2.5 PEP 688: Making the buffer protocol accessible in Python
................................................................
PEP 688(1) introduces a way to use the *note buffer protocol: 393. from
Python code. Classes that implement the *note __buffer__(): 45a. method
are now usable as buffer types.
The new *note collections.abc.Buffer: 2c6. ABC provides a standard way
to represent buffer objects, for example in type annotations. The new
*note inspect.BufferFlags: 45b. enum represents the flags that can be
used to customize buffer creation. (Contributed by Jelle Zijlstra in
gh-102500(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0688/
(2) https://github.com/python/cpython/issues/102500
File: python.info, Node: PEP 709 Comprehension inlining, Next: Improved Error Messages, Prev: PEP 688 Making the buffer protocol accessible in Python, Up: New Features<3>
1.2.2.6 PEP 709: Comprehension inlining
.......................................
Dictionary, list, and set comprehensions are now inlined, rather than
creating a new single-use function object for each execution of the
comprehension. This speeds up execution of a comprehension by up to two
times. See PEP 709(1) for further details.
Comprehension iteration variables remain isolated and don’t overwrite a
variable of the same name in the outer scope, nor are they visible after
the comprehension. Inlining does result in a few visible behavior
changes:
* There is no longer a separate frame for the comprehension in
tracebacks, and tracing/profiling no longer shows the comprehension
as a function call.
* The *note symtable: d8. module will no longer produce child symbol
tables for each comprehension; instead, the comprehension’s locals
will be included in the parent function’s symbol table.
* Calling *note locals(): 141. inside a comprehension now includes
variables from outside the comprehension, and no longer includes
the synthetic ‘.0’ variable for the comprehension “argument”.
* A comprehension iterating directly over ‘locals()’ (e.g. ‘[k for k
in locals()]’) may see “RuntimeError: dictionary changed size
during iteration” when run under tracing (e.g. code coverage
measurement). This is the same behavior already seen in e.g. ‘for
k in locals():’. To avoid the error, first create a list of keys
to iterate over: ‘keys = list(locals()); [k for k in keys]’.
(Contributed by Carl Meyer and Vladimir Matveev in PEP 709(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0709/
(2) https://peps.python.org/pep-0709/
File: python.info, Node: Improved Error Messages, Prev: PEP 709 Comprehension inlining, Up: New Features<3>
1.2.2.7 Improved Error Messages
...............................
* Modules from the standard library are now potentially suggested as
part of the error messages displayed by the interpreter when a
*note NameError: 43a. is raised to the top level. (Contributed by
Pablo Galindo in gh-98254(1).)
>>> sys.version_info
Traceback (most recent call last):
File "", line 1, in
NameError: name 'sys' is not defined. Did you forget to import 'sys'?
* Improve the error suggestion for *note NameError: 43a. exceptions
for instances. Now if a *note NameError: 43a. is raised in a
method and the instance has an attribute that’s exactly equal to
the name in the exception, the suggestion will include
‘self.’ instead of the closest match in the method scope.
(Contributed by Pablo Galindo in gh-99139(2).)
>>> class A:
... def __init__(self):
... self.blech = 1
...
... def foo(self):
... somethin = blech
...
>>> A().foo()
Traceback (most recent call last):
File "", line 1
somethin = blech
^^^^^
NameError: name 'blech' is not defined. Did you mean: 'self.blech'?
* Improve the *note SyntaxError: 18d. error message when the user
types ‘import x from y’ instead of ‘from y import x’. (Contributed
by Pablo Galindo in gh-98931(3).)
>>> import a.y.z from b.y.z
Traceback (most recent call last):
File "", line 1
import a.y.z from b.y.z
^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: Did you mean to use 'from ... import ...' instead?
* *note ImportError: 415. exceptions raised from failed ‘from
import ’ statements now include suggestions for the
value of ‘’ based on the available names in ‘’.
(Contributed by Pablo Galindo in gh-91058(4).)
>>> from collections import chainmap
Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'chainmap' from 'collections'. Did you mean: 'ChainMap'?
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/98254
(2) https://github.com/python/cpython/issues/99139
(3) https://github.com/python/cpython/issues/98931
(4) https://github.com/python/cpython/issues/91058
File: python.info, Node: New Features Related to Type Hints, Next: Other Language Changes<2>, Prev: New Features<3>, Up: What’s New In Python 3 12
1.2.3 New Features Related to Type Hints
----------------------------------------
This section covers major changes affecting type hints(1) and the *note
typing: 104. module.
* Menu:
* PEP 692; Using TypedDict for more precise **kwargs typing: PEP 692 Using TypedDict for more precise **kwargs typing.
* PEP 698; Override Decorator for Static Typing: PEP 698 Override Decorator for Static Typing.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
File: python.info, Node: PEP 692 Using TypedDict for more precise **kwargs typing, Next: PEP 698 Override Decorator for Static Typing, Up: New Features Related to Type Hints
1.2.3.1 PEP 692: Using ‘TypedDict’ for more precise ‘**kwargs’ typing
.....................................................................
Typing ‘**kwargs’ in a function signature as introduced by PEP 484(1)
allowed for valid annotations only in cases where all of the ‘**kwargs’
were of the same type.
PEP 692(2) specifies a more precise way of typing ‘**kwargs’ by relying
on typed dictionaries:
from typing import TypedDict, Unpack
class Movie(TypedDict):
name: str
year: int
def foo(**kwargs: Unpack[Movie]): ...
See PEP 692(3) for more details.
(Contributed by Franek Magiera in gh-103629(4).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0692/
(3) https://peps.python.org/pep-0692/
(4) https://github.com/python/cpython/issues/103629
File: python.info, Node: PEP 698 Override Decorator for Static Typing, Prev: PEP 692 Using TypedDict for more precise **kwargs typing, Up: New Features Related to Type Hints
1.2.3.2 PEP 698: Override Decorator for Static Typing
.....................................................
A new decorator *note typing.override(): 446. has been added to the
*note typing: 104. module. It indicates to type checkers that the
method is intended to override a method in a superclass. This allows
type checkers to catch mistakes where a method that is intended to
override something in a base class does not in fact do so.
Example:
from typing import override
class Base:
def get_color(self) -> str:
return "blue"
class GoodChild(Base):
@override # ok: overrides Base.get_color
def get_color(self) -> str:
return "yellow"
class BadChild(Base):
@override # type checker error: does not override Base.get_color
def get_colour(self) -> str:
return "red"
See PEP 698(1) for more details.
(Contributed by Steven Troxler in gh-101561(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0698/
(2) https://github.com/python/cpython/issues/101561
File: python.info, Node: Other Language Changes<2>, Next: New Modules<2>, Prev: New Features Related to Type Hints, Up: What’s New In Python 3 12
1.2.4 Other Language Changes
----------------------------
* The parser now raises *note SyntaxError: 18d. when parsing source
code containing null bytes. (Contributed by Pablo Galindo in
gh-96670(1).)
* A backslash-character pair that is not a valid escape sequence now
generates a *note SyntaxWarning: 461, instead of *note
DeprecationWarning: 1a5. For example, ‘re.compile("\d+\.\d+")’ now
emits a *note SyntaxWarning: 461. (‘"\d"’ is an invalid escape
sequence, use raw strings for regular expression:
‘re.compile(r"\d+\.\d+")’). In a future Python version, *note
SyntaxError: 18d. will eventually be raised, instead of *note
SyntaxWarning: 461. (Contributed by Victor Stinner in
gh-98401(2).)
* Octal escapes with value larger than ‘0o377’ (ex: ‘"\477"’),
deprecated in Python 3.11, now produce a *note SyntaxWarning: 461,
instead of *note DeprecationWarning: 1a5. In a future Python
version they will be eventually a *note SyntaxError: 18d.
(Contributed by Victor Stinner in gh-98401(3).)
* Variables used in the target part of comprehensions that are not
stored to can now be used in assignment expressions (‘:=’). For
example, in ‘[(b := 1) for a, b.prop in some_iter]’, the assignment
to ‘b’ is now allowed. Note that assigning to variables stored to
in the target part of comprehensions (like ‘a’) is still
disallowed, as per PEP 572(4). (Contributed by Nikita Sobolev in
gh-100581(5).)
* Exceptions raised in a class or type’s ‘__set_name__’ method are no
longer wrapped by a *note RuntimeError: 195. Context information
is added to the exception as a PEP 678(6) note. (Contributed by
Irit Katriel in gh-77757(7).)
* When a ‘try-except*’ construct handles the entire *note
ExceptionGroup: 1b6. and raises one other exception, that exception
is no longer wrapped in an *note ExceptionGroup: 1b6. Also changed
in version 3.11.4. (Contributed by Irit Katriel in gh-103590(8).)
* The Garbage Collector now runs only on the eval breaker mechanism
of the Python bytecode evaluation loop instead of object
allocations. The GC can also run when *note PyErr_CheckSignals():
462. is called so C extensions that need to run for a long time
without executing any Python code also have a chance to execute the
GC periodically. (Contributed by Pablo Galindo in gh-97922(9).)
* All builtin and extension callables expecting boolean parameters
now accept arguments of any type instead of just *note bool: 463.
and *note int: 259. (Contributed by Serhiy Storchaka in
gh-60203(10).)
* *note memoryview: 464. now supports the half-float type (the “e”
format code). (Contributed by Donghee Na and Antoine Pitrou in
gh-90751(11).)
* *note slice: 465. objects are now hashable, allowing them to be
used as dict keys and set items. (Contributed by Will Bradshaw,
Furkan Onder, and Raymond Hettinger in gh-101264(12).)
* *note sum(): 466. now uses Neumaier summation to improve accuracy
and commutativity when summing floats or mixed ints and floats.
(Contributed by Raymond Hettinger in gh-100425(13).)
* *note ast.parse(): 1a8. now raises *note SyntaxError: 18d. instead
of *note ValueError: 204. when parsing source code containing null
bytes. (Contributed by Pablo Galindo in gh-96670(14).)
* The extraction methods in *note tarfile: de, and *note
shutil.unpack_archive(): 467, have a new a 'filter' argument that
allows limiting tar features than may be surprising or dangerous,
such as creating files outside the destination directory. See
*note tarfile extraction filters: 468. for details. In Python
3.14, the default will switch to ‘'data'’. (Contributed by Petr
Viktorin in PEP 706(15).)
* *note types.MappingProxyType: 469. instances are now hashable if
the underlying mapping is hashable. (Contributed by Serhiy
Storchaka in gh-87995(16).)
* Add *note support for the perf profiler: 18f. through the new
environment variable *note PYTHONPERFSUPPORT: 46a. and command-line
option *note -X perf: 176, as well as the new *note
sys.activate_stack_trampoline(): 46b, *note
sys.deactivate_stack_trampoline(): 46c, and *note
sys.is_stack_trampoline_active(): 46d. functions. (Design by Pablo
Galindo. Contributed by Pablo Galindo and Christian Heimes with
contributions from Gregory P. Smith [Google] and Mark Shannon in
gh-96123(17).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/96670
(2) https://github.com/python/cpython/issues/98401
(3) https://github.com/python/cpython/issues/98401
(4) https://peps.python.org/pep-0572/
(5) https://github.com/python/cpython/issues/100581
(6) https://peps.python.org/pep-0678/
(7) https://github.com/python/cpython/issues/77757
(8) https://github.com/python/cpython/issues/103590
(9) https://github.com/python/cpython/issues/97922
(10) https://github.com/python/cpython/issues/60203
(11) https://github.com/python/cpython/issues/90751
(12) https://github.com/python/cpython/issues/101264
(13) https://github.com/python/cpython/issues/100425
(14) https://github.com/python/cpython/issues/96670
(15) https://peps.python.org/pep-0706/
(16) https://github.com/python/cpython/issues/87995
(17) https://github.com/python/cpython/issues/96123
File: python.info, Node: New Modules<2>, Next: Improved Modules<2>, Prev: Other Language Changes<2>, Up: What’s New In Python 3 12
1.2.5 New Modules
-----------------
* None.
File: python.info, Node: Improved Modules<2>, Next: Optimizations<2>, Prev: New Modules<2>, Up: What’s New In Python 3 12
1.2.6 Improved Modules
----------------------
* Menu:
* array: array<2>.
* asyncio: asyncio<2>.
* calendar::
* csv::
* dis: dis<2>.
* fractions: fractions<2>.
* importlib.resources: importlib resources.
* inspect::
* itertools: itertools<2>.
* math: math<2>.
* os: os<2>.
* os.path: os path<3>.
* pathlib: pathlib<3>.
* platform::
* pdb: pdb<2>.
* random: random<2>.
* shutil: shutil<2>.
* sqlite3: sqlite3<2>.
* statistics: statistics<2>.
* sys: sys<3>.
* tempfile: tempfile<2>.
* threading::
* tkinter: tkinter<2>.
* tokenize::
* types: types<2>.
* typing: typing<3>.
* unicodedata: unicodedata<2>.
* unittest: unittest<2>.
* uuid::
File: python.info, Node: array<2>, Next: asyncio<2>, Up: Improved Modules<2>
1.2.6.1 array
.............
* The *note array.array: 1a0. class now supports subscripting, making
it a *note generic type: 434. (Contributed by Jelle Zijlstra in
gh-98658(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/98658
File: python.info, Node: asyncio<2>, Next: calendar, Prev: array<2>, Up: Improved Modules<2>
1.2.6.2 asyncio
...............
* The performance of writing to sockets in *note asyncio: a. has been
significantly improved. ‘asyncio’ now avoids unnecessary copying
when writing to sockets and uses *note sendmsg(): 472. if the
platform supports it. (Contributed by Kumar Aditya in
gh-91166(1).)
* Add *note asyncio.eager_task_factory(): 473. and *note
asyncio.create_eager_task_factory(): 474. functions to allow opting
an event loop in to eager task execution, making some use-cases 2x
to 5x faster. (Contributed by Jacob Bower & Itamar Oren in
gh-102853(2), gh-104140(3), and gh-104138(4))
* On Linux, *note asyncio: a. uses *note asyncio.PidfdChildWatcher:
475. by default if *note os.pidfd_open(): 476. is available and
functional instead of *note asyncio.ThreadedChildWatcher: 477.
(Contributed by Kumar Aditya in gh-98024(5).)
* The event loop now uses the best available child watcher for each
platform (*note asyncio.PidfdChildWatcher: 475. if supported and
*note asyncio.ThreadedChildWatcher: 477. otherwise), so manually
configuring a child watcher is not recommended. (Contributed by
Kumar Aditya in gh-94597(6).)
* Add 'loop_factory' parameter to *note asyncio.run(): 478. to allow
specifying a custom event loop factory. (Contributed by Kumar
Aditya in gh-99388(7).)
* Add C implementation of *note asyncio.current_task(): 479. for
4x-6x speedup. (Contributed by Itamar Oren and Pranav Thulasiram
Bhat in gh-100344(8).)
* *note asyncio.iscoroutine(): 47a. now returns ‘False’ for
generators as *note asyncio: a. does not support legacy
generator-based coroutines. (Contributed by Kumar Aditya in
gh-102748(9).)
* *note asyncio.wait(): 47b. and *note asyncio.as_completed(): 1aa.
now accepts generators yielding tasks. (Contributed by Kumar
Aditya in gh-78530(10).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/91166
(2) https://github.com/python/cpython/issues/102853
(3) https://github.com/python/cpython/issues/104140
(4) https://github.com/python/cpython/issues/104138
(5) https://github.com/python/cpython/issues/98024
(6) https://github.com/python/cpython/issues/94597
(7) https://github.com/python/cpython/issues/99388
(8) https://github.com/python/cpython/issues/100344
(9) https://github.com/python/cpython/issues/102748
(10) https://github.com/python/cpython/issues/78530
File: python.info, Node: calendar, Next: csv, Prev: asyncio<2>, Up: Improved Modules<2>
1.2.6.3 calendar
................
* Add enums *note calendar.Month: 47d. and *note calendar.Day: 47e.
defining months of the year and days of the week. (Contributed by
Prince Roshan in gh-103636(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/103636
File: python.info, Node: csv, Next: dis<2>, Prev: calendar, Up: Improved Modules<2>
1.2.6.4 csv
...........
* Add *note csv.QUOTE_NOTNULL: 480. and *note csv.QUOTE_STRINGS: 481.
flags to provide finer grained control of ‘None’ and empty strings
by *note reader: 482. and *note writer: 483. objects.
File: python.info, Node: dis<2>, Next: fractions<2>, Prev: csv, Up: Improved Modules<2>
1.2.6.5 dis
...........
* Pseudo instruction opcodes (which are used by the compiler but do
not appear in executable bytecode) are now exposed in the *note
dis: 38. module. *note HAVE_ARGUMENT: 485. is still relevant to
real opcodes, but it is not useful for pseudo instructions. Use
the new *note dis.hasarg: 2a0. collection instead. (Contributed by
Irit Katriel in gh-94216(1).)
* Add the *note dis.hasexc: 486. collection to signify instructions
that set an exception handler. (Contributed by Irit Katriel in
gh-94216(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/94216
(2) https://github.com/python/cpython/issues/94216
File: python.info, Node: fractions<2>, Next: importlib resources, Prev: dis<2>, Up: Improved Modules<2>
1.2.6.6 fractions
.................
* Objects of type *note fractions.Fraction: 1e9. now support
float-style formatting. (Contributed by Mark Dickinson in
gh-100161(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/100161
File: python.info, Node: importlib resources, Next: inspect, Prev: fractions<2>, Up: Improved Modules<2>
1.2.6.7 importlib.resources
...........................
* *note importlib.resources.as_file(): 489. now supports resource
directories. (Contributed by Jason R. Coombs in gh-97930(1).)
* Rename first parameter of *note importlib.resources.files(): 48a.
to 'anchor'. (Contributed by Jason R. Coombs in gh-100598(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/97930
(2) https://github.com/python/cpython/issues/100598
File: python.info, Node: inspect, Next: itertools<2>, Prev: importlib resources, Up: Improved Modules<2>
1.2.6.8 inspect
...............
* Add *note inspect.markcoroutinefunction(): 48c. to mark sync
functions that return a *note coroutine: 48d. for use with *note
inspect.iscoroutinefunction(): 2e8. (Contributed by Carlton Gibson
in gh-99247(1).)
* Add *note inspect.getasyncgenstate(): 48e. and *note
inspect.getasyncgenlocals(): 48f. for determining the current state
of asynchronous generators. (Contributed by Thomas Krennwallner in
gh-79940(2).)
* The performance of *note inspect.getattr_static(): 490. has been
considerably improved. Most calls to the function should be at
least 2x faster than they were in Python 3.11. (Contributed by
Alex Waygood in gh-103193(3).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/99247
(2) https://github.com/python/cpython/issues/79940
(3) https://github.com/python/cpython/issues/103193
File: python.info, Node: itertools<2>, Next: math<2>, Prev: inspect, Up: Improved Modules<2>
1.2.6.9 itertools
.................
* Add *note itertools.batched(): 203. for collecting into even-sized
tuples where the last batch may be shorter than the rest.
(Contributed by Raymond Hettinger in gh-98363(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/98363
File: python.info, Node: math<2>, Next: os<2>, Prev: itertools<2>, Up: Improved Modules<2>
1.2.6.10 math
.............
* Add *note math.sumprod(): 493. for computing a sum of products.
(Contributed by Raymond Hettinger in gh-100485(1).)
* Extend *note math.nextafter(): 494. to include a 'steps' argument
for moving up or down multiple steps at a time. (Contributed by
Matthias Goergens, Mark Dickinson, and Raymond Hettinger in
gh-94906(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/100485
(2) https://github.com/python/cpython/issues/94906
File: python.info, Node: os<2>, Next: os path<3>, Prev: math<2>, Up: Improved Modules<2>
1.2.6.11 os
...........
* Add *note os.PIDFD_NONBLOCK: 496. to open a file descriptor for a
process with *note os.pidfd_open(): 476. in non-blocking mode.
(Contributed by Kumar Aditya in gh-93312(1).)
* *note os.DirEntry: 497. now includes an *note
os.DirEntry.is_junction(): 498. method to check if the entry is a
junction. (Contributed by Charles Machalow in gh-99547(2).)
* Add *note os.listdrives(): 499, *note os.listvolumes(): 49a. and
*note os.listmounts(): 49b. functions on Windows for enumerating
drives, volumes and mount points. (Contributed by Steve Dower in
gh-102519(3).)
* *note os.stat(): 49c. and *note os.lstat(): 49d. are now more
accurate on Windows. The ‘st_birthtime’ field will now be filled
with the creation time of the file, and ‘st_ctime’ is deprecated
but still contains the creation time (but in the future will return
the last metadata change, for consistency with other platforms).
‘st_dev’ may be up to 64 bits and ‘st_ino’ up to 128 bits depending
on your file system, and ‘st_rdev’ is always set to zero rather
than incorrect values. Both functions may be significantly faster
on newer releases of Windows. (Contributed by Steve Dower in
gh-99726(4).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/93312
(2) https://github.com/python/cpython/issues/99547
(3) https://github.com/python/cpython/issues/102519
(4) https://github.com/python/cpython/issues/99726
File: python.info, Node: os path<3>, Next: pathlib<3>, Prev: os<2>, Up: Improved Modules<2>
1.2.6.12 os.path
................
* Add *note os.path.isjunction(): 49f. to check if a given path is a
junction. (Contributed by Charles Machalow in gh-99547(1).)
* Add *note os.path.splitroot(): 4a0. to split a path into a triad
‘(drive, root, tail)’. (Contributed by Barney Gale in
gh-101000(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/99547
(2) https://github.com/python/cpython/issues/101000
File: python.info, Node: pathlib<3>, Next: platform, Prev: os path<3>, Up: Improved Modules<2>
1.2.6.13 pathlib
................
* Add support for subclassing *note pathlib.PurePath: 4a2. and *note
pathlib.Path: 22b, plus their Posix- and Windows-specific variants.
Subclasses may override the *note pathlib.PurePath.with_segments():
4a3. method to pass information between path instances.
* Add *note pathlib.Path.walk(): 4a4. for walking the directory trees
and generating all file or directory names within them, similar to
*note os.walk(): 4a5. (Contributed by Stanislav Zmiev in
gh-90385(1).)
* Add 'walk_up' optional parameter to *note
pathlib.PurePath.relative_to(): 2cd. to allow the insertion of ‘..’
entries in the result; this behavior is more consistent with *note
os.path.relpath(): 4a6. (Contributed by Domenico Ragusa in
gh-84538(2).)
* Add *note pathlib.Path.is_junction(): 4a7. as a proxy to *note
os.path.isjunction(): 49f. (Contributed by Charles Machalow in
gh-99547(3).)
* Add 'case_sensitive' optional parameter to *note
pathlib.Path.glob(): 22f, *note pathlib.Path.rglob(): 230. and
*note pathlib.PurePath.match(): 4a8. for matching the path’s case
sensitivity, allowing for more precise control over the matching
process.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/90385
(2) https://github.com/python/cpython/issues/84538
(3) https://github.com/python/cpython/issues/99547
File: python.info, Node: platform, Next: pdb<2>, Prev: pathlib<3>, Up: Improved Modules<2>
1.2.6.14 platform
.................
* Add support for detecting Windows 11 and Windows Server releases
past 2012. Previously, lookups on Windows Server platforms newer
than Windows Server 2012 and on Windows 11 would return
‘Windows-10’. (Contributed by Steve Dower in gh-89545(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/89545
File: python.info, Node: pdb<2>, Next: random<2>, Prev: platform, Up: Improved Modules<2>
1.2.6.15 pdb
............
* Add convenience variables to hold values temporarily for debug
session and provide quick access to values like the current frame
or the return value. (Contributed by Tian Gao in gh-103693(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/103693
File: python.info, Node: random<2>, Next: shutil<2>, Prev: pdb<2>, Up: Improved Modules<2>
1.2.6.16 random
...............
* Add *note random.binomialvariate(): 4ac. (Contributed by Raymond
Hettinger in gh-81620(1).)
* Add a default of ‘lambd=1.0’ to *note random.expovariate(): 4ad.
(Contributed by Raymond Hettinger in gh-100234(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/81620
(2) https://github.com/python/cpython/issues/100234
File: python.info, Node: shutil<2>, Next: sqlite3<2>, Prev: random<2>, Up: Improved Modules<2>
1.2.6.17 shutil
...............
* *note shutil.make_archive(): 4af. now passes the 'root_dir'
argument to custom archivers which support it. In this case it no
longer temporarily changes the current working directory of the
process to 'root_dir' to perform archiving. (Contributed by Serhiy
Storchaka in gh-74696(1).)
* *note shutil.rmtree(): 2fb. now accepts a new argument 'onexc'
which is an error handler like 'onerror' but which expects an
exception instance rather than a '(typ, val, tb)' triplet.
'onerror' is deprecated. (Contributed by Irit Katriel in
gh-102828(2).)
* *note shutil.which(): 4b0. now consults the 'PATHEXT' environment
variable to find matches within 'PATH' on Windows even when the
given 'cmd' includes a directory component. (Contributed by
Charles Machalow in gh-103179(3).)
*note shutil.which(): 4b0. will call
‘NeedCurrentDirectoryForExePathW’ when querying for executables on
Windows to determine if the current working directory should be
prepended to the search path. (Contributed by Charles Machalow in
gh-103179(4).)
*note shutil.which(): 4b0. will return a path matching the 'cmd'
with a component from ‘PATHEXT’ prior to a direct match elsewhere
in the search path on Windows. (Contributed by Charles Machalow in
gh-103179(5).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/74696
(2) https://github.com/python/cpython/issues/102828
(3) https://github.com/python/cpython/issues/103179
(4) https://github.com/python/cpython/issues/103179
(5) https://github.com/python/cpython/issues/103179
File: python.info, Node: sqlite3<2>, Next: statistics<2>, Prev: shutil<2>, Up: Improved Modules<2>
1.2.6.18 sqlite3
................
* Add a *note command-line interface: 43c. (Contributed by Erlend E.
Aasland in gh-77617(1).)
* Add the *note sqlite3.Connection.autocommit: 4b2. attribute to
*note sqlite3.Connection: 247. and the 'autocommit' parameter to
*note sqlite3.connect(): 2aa. to control PEP 249(2)-compliant *note
transaction handling: 4b3. (Contributed by Erlend E. Aasland in
gh-83638(3).)
* Add 'entrypoint' keyword-only parameter to *note
sqlite3.Connection.load_extension(): 4b4, for overriding the SQLite
extension entry point. (Contributed by Erlend E. Aasland in
gh-103015(4).)
* Add *note sqlite3.Connection.getconfig(): 4b5. and *note
sqlite3.Connection.setconfig(): 4b6. to *note sqlite3.Connection:
247. to make configuration changes to a database connection.
(Contributed by Erlend E. Aasland in gh-103489(5).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/77617
(2) https://peps.python.org/pep-0249/
(3) https://github.com/python/cpython/issues/83638
(4) https://github.com/python/cpython/issues/103015
(5) https://github.com/python/cpython/issues/103489
File: python.info, Node: statistics<2>, Next: sys<3>, Prev: sqlite3<2>, Up: Improved Modules<2>
1.2.6.19 statistics
...................
* Extend *note statistics.correlation(): 4b8. to include as a
‘ranked’ method for computing the Spearman correlation of ranked
data. (Contributed by Raymond Hettinger in gh-95861(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/95861
File: python.info, Node: sys<3>, Next: tempfile<2>, Prev: statistics<2>, Up: Improved Modules<2>
1.2.6.20 sys
............
* Add the *note sys.monitoring: da. namespace to expose the new *note
PEP 669: 438. monitoring API. (Contributed by Mark Shannon in
gh-103082(1).)
* Add *note sys.activate_stack_trampoline(): 46b. and *note
sys.deactivate_stack_trampoline(): 46c. for activating and
deactivating stack profiler trampolines, and *note
sys.is_stack_trampoline_active(): 46d. for querying if stack
profiler trampolines are active. (Contributed by Pablo Galindo and
Christian Heimes with contributions from Gregory P. Smith [Google]
and Mark Shannon in gh-96123(2).)
* Add *note sys.last_exc: 4ba. which holds the last unhandled
exception that was raised (for post-mortem debugging use cases).
Deprecate the three fields that have the same information in its
legacy form: *note sys.last_type: 4bb, *note sys.last_value: 4bc.
and *note sys.last_traceback: 4bd. (Contributed by Irit Katriel in
gh-102778(3).)
* *note sys._current_exceptions(): 4be. now returns a mapping from
thread-id to an exception instance, rather than to a ‘(typ, exc,
tb)’ tuple. (Contributed by Irit Katriel in gh-103176(4).)
* *note sys.setrecursionlimit(): 4bf. and *note
sys.getrecursionlimit(): 4c0. The recursion limit now applies only
to Python code. Builtin functions do not use the recursion limit,
but are protected by a different mechanism that prevents recursion
from causing a virtual machine crash.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/103082
(2) https://github.com/python/cpython/issues/96123
(3) https://github.com/python/cpython/issues/102778
(4) https://github.com/python/cpython/issues/103176
File: python.info, Node: tempfile<2>, Next: threading, Prev: sys<3>, Up: Improved Modules<2>
1.2.6.21 tempfile
.................
* The *note tempfile.NamedTemporaryFile: 4c2. function has a new
optional parameter 'delete_on_close' (Contributed by Evgeny Zorin
in gh-58451(1).)
* *note tempfile.mkdtemp(): 221. now always returns an absolute path,
even if the argument provided to the 'dir' parameter is a relative
path.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/58451
File: python.info, Node: threading, Next: tkinter<2>, Prev: tempfile<2>, Up: Improved Modules<2>
1.2.6.22 threading
..................
* Add *note threading.settrace_all_threads(): 4c4. and *note
threading.setprofile_all_threads(): 4c5. that allow to set tracing
and profiling functions in all running threads in addition to the
calling one. (Contributed by Pablo Galindo in gh-93503(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/93503
File: python.info, Node: tkinter<2>, Next: tokenize, Prev: threading, Up: Improved Modules<2>
1.2.6.23 tkinter
................
* ‘tkinter.Canvas.coords()’ now flattens its arguments. It now
accepts not only coordinates as separate arguments (‘x1, y1, x2,
y2, ...’) and a sequence of coordinates (‘[x1, y1, x2, y2, ...]’),
but also coordinates grouped in pairs (‘(x1, y1), (x2, y2), ...’
and ‘[(x1, y1), (x2, y2), ...]’), like ‘create_*()’ methods.
(Contributed by Serhiy Storchaka in gh-94473(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/94473
File: python.info, Node: tokenize, Next: types<2>, Prev: tkinter<2>, Up: Improved Modules<2>
1.2.6.24 tokenize
.................
* The *note tokenize: fb. module includes the changes introduced in
PEP 701(1). (Contributed by Marta Gómez Macías and Pablo Galindo
in gh-102856(2).) See *note Porting to Python 3.12: 4c8. for more
information on the changes to the *note tokenize: fb. module.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0701/
(2) https://github.com/python/cpython/issues/102856
File: python.info, Node: types<2>, Next: typing<3>, Prev: tokenize, Up: Improved Modules<2>
1.2.6.25 types
..............
* Add *note types.get_original_bases(): 4ca. to allow for further
introspection of *note User-defined generic types: 4cb. when
subclassed. (Contributed by James Hilton-Balfe and Alex Waygood in
gh-101827(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/101827
File: python.info, Node: typing<3>, Next: unicodedata<2>, Prev: types<2>, Up: Improved Modules<2>
1.2.6.26 typing
...............
* *note isinstance(): 43d. checks against *note runtime-checkable
protocols: 43e. now use *note inspect.getattr_static(): 490. rather
than *note hasattr(): 4ce. to lookup whether attributes exist.
This means that descriptors and *note __getattr__(): 4cf. methods
are no longer unexpectedly evaluated during ‘isinstance()’ checks
against runtime-checkable protocols. However, it may also mean
that some objects which used to be considered instances of a
runtime-checkable protocol may no longer be considered instances of
that protocol on Python 3.12+, and vice versa. Most users are
unlikely to be affected by this change. (Contributed by Alex
Waygood in gh-102433(1).)
* The members of a runtime-checkable protocol are now considered
“frozen” at runtime as soon as the class has been created.
Monkey-patching attributes onto a runtime-checkable protocol will
still work, but will have no impact on *note isinstance(): 43d.
checks comparing objects to the protocol. For example:
>>> from typing import Protocol, runtime_checkable
>>> @runtime_checkable
... class HasX(Protocol):
... x = 1
...
>>> class Foo: ...
...
>>> f = Foo()
>>> isinstance(f, HasX)
False
>>> f.x = 1
>>> isinstance(f, HasX)
True
>>> HasX.y = 2
>>> isinstance(f, HasX) # unchanged, even though HasX now also has a "y" attribute
True
This change was made in order to speed up ‘isinstance()’ checks
against runtime-checkable protocols.
* The performance profile of *note isinstance(): 43d. checks against
*note runtime-checkable protocols: 43e. has changed significantly.
Most ‘isinstance()’ checks against protocols with only a few
members should be at least 2x faster than in 3.11, and some may be
20x faster or more. However, ‘isinstance()’ checks against
protocols with many members may be slower than in Python 3.11.
(Contributed by Alex Waygood in gh-74690(2) and gh-103193(3).)
* All *note typing.TypedDict: 162. and *note typing.NamedTuple: 2b2.
classes now have the ‘__orig_bases__’ attribute. (Contributed by
Adrian Garcia Badaracco in gh-103699(4).)
* Add ‘frozen_default’ parameter to *note
typing.dataclass_transform(): 4d0. (Contributed by Erik De Bonte
in gh-99957(5).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/102433
(2) https://github.com/python/cpython/issues/74690
(3) https://github.com/python/cpython/issues/103193
(4) https://github.com/python/cpython/issues/103699
(5) https://github.com/python/cpython/issues/99957
File: python.info, Node: unicodedata<2>, Next: unittest<2>, Prev: typing<3>, Up: Improved Modules<2>
1.2.6.27 unicodedata
....................
* The Unicode database has been updated to version 15.0.0.
(Contributed by Benjamin Peterson in gh-96734(1)).
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/96734
File: python.info, Node: unittest<2>, Next: uuid, Prev: unicodedata<2>, Up: Improved Modules<2>
1.2.6.28 unittest
.................
Add a ‘--durations’ command line option, showing the N slowest test
cases:
python3 -m unittest --durations=3 lib.tests.test_threading
.....
Slowest test durations
----------------------------------------------------------------------
1.210s test_timeout (Lib.test.test_threading.BarrierTests)
1.003s test_default_timeout (Lib.test.test_threading.BarrierTests)
0.518s test_timeout (Lib.test.test_threading.EventTests)
(0.000 durations hidden. Use -v to show these durations.)
----------------------------------------------------------------------
Ran 158 tests in 9.869s
OK (skipped=3)
(Contributed by Giampaolo Rodola in gh-48330(1))
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/48330
File: python.info, Node: uuid, Prev: unittest<2>, Up: Improved Modules<2>
1.2.6.29 uuid
.............
* Add a *note command-line interface: 43f. (Contributed by Adam
Chhina in gh-88597(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/88597
File: python.info, Node: Optimizations<2>, Next: CPython bytecode changes, Prev: Improved Modules<2>, Up: What’s New In Python 3 12
1.2.7 Optimizations
-------------------
* Remove ‘wstr’ and ‘wstr_length’ members from Unicode objects. It
reduces object size by 8 or 16 bytes on 64bit platform. ( PEP
623(1)) (Contributed by Inada Naoki in gh-92536(2).)
* Add experimental support for using the BOLT binary optimizer in the
build process, which improves performance by 1-5%. (Contributed by
Kevin Modzelewski in gh-90536(3) and tuned by Donghee Na in
gh-101525(4))
* Speed up the regular expression substitution (functions *note
re.sub(): 2a5. and *note re.subn(): 2a6. and corresponding
‘re.Pattern’ methods) for replacement strings containing group
references by 2–3 times. (Contributed by Serhiy Storchaka in
gh-91524(5).)
* Speed up *note asyncio.Task: 1be. creation by deferring expensive
string formatting. (Contributed by Itamar Oren in gh-103793(6).)
* The *note tokenize.tokenize(): 4d5. and *note
tokenize.generate_tokens(): 4d6. functions are up to 64% faster as
a side effect of the changes required to cover PEP 701(7) in the
*note tokenize: fb. module. (Contributed by Marta Gómez Macías and
Pablo Galindo in gh-102856(8).)
* Speed up *note super(): 4d7. method calls and attribute loads via
the new *note LOAD_SUPER_ATTR: 4d8. instruction. (Contributed by
Carl Meyer and Vladimir Matveev in gh-103497(9).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0623/
(2) https://github.com/python/cpython/issues/92536
(3) https://github.com/python/cpython/issues/90536
(4) https://github.com/python/cpython/issues/101525
(5) https://github.com/python/cpython/issues/91524
(6) https://github.com/python/cpython/issues/103793
(7) https://peps.python.org/pep-0701/
(8) https://github.com/python/cpython/issues/102856
(9) https://github.com/python/cpython/issues/103497
File: python.info, Node: CPython bytecode changes, Next: Demos and Tools, Prev: Optimizations<2>, Up: What’s New In Python 3 12
1.2.8 CPython bytecode changes
------------------------------
* Remove the ‘LOAD_METHOD’ instruction. It has been merged into
*note LOAD_ATTR: 4da. *note LOAD_ATTR: 4da. will now behave like
the old ‘LOAD_METHOD’ instruction if the low bit of its oparg is
set. (Contributed by Ken Jin in gh-93429(1).)
* Remove the ‘JUMP_IF_FALSE_OR_POP’ and ‘JUMP_IF_TRUE_OR_POP’
instructions. (Contributed by Irit Katriel in gh-102859(2).)
* Remove the ‘PRECALL’ instruction. (Contributed by Mark Shannon in
gh-92925(3).)
* Add the *note BINARY_SLICE: 4db. and *note STORE_SLICE: 4dc.
instructions. (Contributed by Mark Shannon in gh-94163(4).)
* Add the *note CALL_INTRINSIC_1: 4dd. instructions. (Contributed by
Mark Shannon in gh-99005(5).)
* Add the *note CALL_INTRINSIC_2: 4de. instruction. (Contributed by
Irit Katriel in gh-101799(6).)
* Add the *note CLEANUP_THROW: 4df. instruction. (Contributed by
Brandt Bucher in gh-90997(7).)
* Add the ‘END_SEND’ instruction. (Contributed by Mark Shannon in
gh-103082(8).)
* Add the *note LOAD_FAST_AND_CLEAR: 4e0. instruction as part of the
implementation of PEP 709(9). (Contributed by Carl Meyer in
gh-101441(10).)
* Add the *note LOAD_FAST_CHECK: 4e1. instruction. (Contributed by
Dennis Sweeney in gh-93143(11).)
* Add the *note LOAD_FROM_DICT_OR_DEREF: 4e2, *note
LOAD_FROM_DICT_OR_GLOBALS: 4e3, and *note LOAD_LOCALS: 4e4. opcodes
as part of the implementation of PEP 695(12). Remove the
‘LOAD_CLASSDEREF’ opcode, which can be replaced with *note
LOAD_LOCALS: 4e4. plus *note LOAD_FROM_DICT_OR_DEREF: 4e2.
(Contributed by Jelle Zijlstra in gh-103764(13).)
* Add the *note LOAD_SUPER_ATTR: 4d8. instruction. (Contributed by
Carl Meyer and Vladimir Matveev in gh-103497(14).)
* Add the *note RETURN_CONST: 4e5. instruction. (Contributed by
Wenyang Wang in gh-101632(15).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/93429
(2) https://github.com/python/cpython/issues/102859
(3) https://github.com/python/cpython/issues/92925
(4) https://github.com/python/cpython/issues/94163
(5) https://github.com/python/cpython/issues/99005
(6) https://github.com/python/cpython/issues/101799
(7) https://github.com/python/cpython/issues/90997
(8) https://github.com/python/cpython/issues/103082
(9) https://peps.python.org/pep-0709/
(10) https://github.com/python/cpython/issues/101441
(11) https://github.com/python/cpython/issues/93143
(12) https://peps.python.org/pep-0695/
(13) https://github.com/python/cpython/issues/103764
(14) https://github.com/python/cpython/issues/103497
(15) https://github.com/python/cpython/issues/101632
File: python.info, Node: Demos and Tools, Next: Deprecated, Prev: CPython bytecode changes, Up: What’s New In Python 3 12
1.2.9 Demos and Tools
---------------------
* Remove the ‘Tools/demo/’ directory which contained old demo
scripts. A copy can be found in the old-demos project(1).
(Contributed by Victor Stinner in gh-97681(2).)
* Remove outdated example scripts of the ‘Tools/scripts/’ directory.
A copy can be found in the old-demos project(3). (Contributed by
Victor Stinner in gh-97669(4).)
---------- Footnotes ----------
(1) https://github.com/gvanrossum/old-demos
(2) https://github.com/python/cpython/issues/97681
(3) https://github.com/gvanrossum/old-demos
(4) https://github.com/python/cpython/issues/97669
File: python.info, Node: Deprecated, Next: Removed, Prev: Demos and Tools, Up: What’s New In Python 3 12
1.2.10 Deprecated
-----------------
* *note argparse: 6.: The 'type', 'choices', and 'metavar' parameters
of ‘argparse.BooleanOptionalAction’ are deprecated and will be
removed in 3.14. (Contributed by Nikita Sobolev in gh-92248(1).)
* *note ast: 8.: The following *note ast: 8. features have been
deprecated in documentation since Python 3.8, now cause a *note
DeprecationWarning: 1a5. to be emitted at runtime when they are
accessed or used, and will be removed in Python 3.14:
* ‘ast.Num’
* ‘ast.Str’
* ‘ast.Bytes’
* ‘ast.NameConstant’
* ‘ast.Ellipsis’
Use *note ast.Constant: 2bb. instead. (Contributed by Serhiy
Storchaka in gh-90953(2).)
* *note asyncio: a.:
* The child watcher classes *note asyncio.MultiLoopChildWatcher:
2bc, *note asyncio.FastChildWatcher: 2bd, *note
asyncio.AbstractChildWatcher: 2be. and *note
asyncio.SafeChildWatcher: 2bf. are deprecated and will be
removed in Python 3.14. (Contributed by Kumar Aditya in
gh-94597(3).)
* *note asyncio.set_child_watcher(): 2c0, *note
asyncio.get_child_watcher(): 2c1, *note
asyncio.AbstractEventLoopPolicy.set_child_watcher(): 2c2. and
*note asyncio.AbstractEventLoopPolicy.get_child_watcher():
2c3. are deprecated and will be removed in Python 3.14.
(Contributed by Kumar Aditya in gh-94597(4).)
* The *note get_event_loop(): 2c4. method of the default event
loop policy now emits a *note DeprecationWarning: 1a5. if
there is no current event loop set and it decides to create
one. (Contributed by Serhiy Storchaka and Guido van Rossum in
gh-100160(5).)
* *note calendar: 14.: ‘calendar.January’ and ‘calendar.February’
constants are deprecated and replaced by *note calendar.JANUARY:
2f3. and *note calendar.FEBRUARY: 2f4. (Contributed by Prince
Roshan in gh-103636(6).)
* *note collections.abc: 1e.: Deprecated *note
collections.abc.ByteString: 2c5. Prefer ‘Sequence’ or *note
collections.abc.Buffer: 2c6. For use in typing, prefer a union,
like ‘bytes | bytearray’, or *note collections.abc.Buffer: 2c6.
(Contributed by Shantanu Jain in gh-91896(7).)
* *note datetime: 30.: *note datetime.datetime: 1cc.’s *note
utcnow(): 2f6. and *note utcfromtimestamp(): 2f7. are deprecated
and will be removed in a future version. Instead, use
timezone-aware objects to represent datetimes in UTC: respectively,
call *note now(): 4e8. and *note fromtimestamp(): 4e9. with the
'tz' parameter set to *note datetime.UTC: 4ea. (Contributed by
Paul Ganssle in gh-103857(8).)
* *note email: 3b.: Deprecate the 'isdst' parameter in *note
email.utils.localtime(): 2c7. (Contributed by Alan Williams in
gh-72346(9).)
* *note importlib.abc: 78.: Deprecated the following classes,
scheduled for removal in Python 3.14:
* ‘importlib.abc.ResourceReader’
* ‘importlib.abc.Traversable’
* ‘importlib.abc.TraversableResources’
Use *note importlib.resources.abc: 7c. classes instead:
* *note importlib.resources.abc.Traversable: 1f5.
* *note importlib.resources.abc.TraversableResources: 2c8.
(Contributed by Jason R. Coombs and Hugo van Kemenade in
gh-93963(10).)
* *note itertools: 81.: Deprecate the support for copy, deepcopy, and
pickle operations, which is undocumented, inefficient, historically
buggy, and inconsistent. This will be removed in 3.14 for a
significant reduction in code volume and maintenance burden.
(Contributed by Raymond Hettinger in gh-101588(11).)
* *note multiprocessing: 94.: In Python 3.14, the default *note
multiprocessing: 94. start method will change to a safer one on
Linux, BSDs, and other non-macOS POSIX platforms where ‘'fork'’ is
currently the default (gh-84559(12)). Adding a runtime warning
about this was deemed too disruptive as the majority of code is not
expected to care. Use the *note get_context(): 2c9. or *note
set_start_method(): 2ca. APIs to explicitly specify when your code
'requires' ‘'fork'’. See *note contexts and start methods: 2cb.
* *note pkgutil: a9.: *note pkgutil.find_loader(): 2ce. and *note
pkgutil.get_loader(): 2cf. are deprecated and will be removed in
Python 3.14; use *note importlib.util.find_spec(): 2d0. instead.
(Contributed by Nikita Sobolev in gh-97850(13).)
* *note pty: b1.: The module has two undocumented ‘master_open()’ and
‘slave_open()’ functions that have been deprecated since Python 2
but only gained a proper *note DeprecationWarning: 1a5. in 3.12.
Remove them in 3.14. (Contributed by Soumendra Ganguly and Gregory
P. Smith in gh-85984(14).)
* *note os: a1.:
* The ‘st_ctime’ fields return by *note os.stat(): 49c. and
*note os.lstat(): 49d. on Windows are deprecated. In a future
release, they will contain the last metadata change time,
consistent with other platforms. For now, they still contain
the creation time, which is also available in the new
‘st_birthtime’ field. (Contributed by Steve Dower in
gh-99726(15).)
* On POSIX platforms, *note os.fork(): 197. can now raise a
*note DeprecationWarning: 1a5. when it can detect being called
from a multithreaded process. There has always been a
fundamental incompatibility with the POSIX platform when doing
so. Even if such code 'appeared' to work. We added the
warning to raise awareness as issues encountered by code doing
this are becoming more frequent. See the *note os.fork():
197. documentation for more details along with this discussion
on fork being incompatible with threads(16) for 'why' we’re
now surfacing this longstanding platform compatibility problem
to developers.
When this warning appears due to usage of *note multiprocessing:
94. or *note concurrent.futures: 21. the fix is to use a different
*note multiprocessing: 94. start method such as ‘"spawn"’ or
‘"forkserver"’.
* *note shutil: c5.: The 'onerror' argument of *note shutil.rmtree():
2fb. is deprecated; use 'onexc' instead. (Contributed by Irit
Katriel in gh-102828(17).)
* *note sqlite3: cf.:
* *note default adapters and converters: 4eb. are now
deprecated. Instead, use the *note Adapter and converter
recipes: 4ec. and tailor them to your needs. (Contributed by
Erlend E. Aasland in gh-90016(18).)
* In *note execute(): 2d4, *note DeprecationWarning: 1a5. is now
emitted when *note named placeholders: 2d6. are used together
with parameters supplied as a *note sequence: 4ed. instead of
as a *note dict: 258. Starting from Python 3.14, using named
placeholders with parameters supplied as a sequence will raise
a *note ProgrammingError: 4ee. (Contributed by Erlend E.
Aasland in gh-101698(19).)
* *note sys: d9.: The *note sys.last_type: 4bb, *note sys.last_value:
4bc. and *note sys.last_traceback: 4bd. fields are deprecated. Use
*note sys.last_exc: 4ba. instead. (Contributed by Irit Katriel in
gh-102778(20).)
* *note tarfile: de.: Extracting tar archives without specifying
'filter' is deprecated until Python 3.14, when ‘'data'’ filter will
become the default. See *note Extraction filters: 468. for
details.
* *note typing: 104.:
* *note typing.Hashable: 4ef. and *note typing.Sized: 4f0,
aliases for *note collections.abc.Hashable: 4f1. and *note
collections.abc.Sized: 4f2. respectively, are deprecated.
(gh-94309(21).)
* *note typing.ByteString: 2d7, deprecated since Python 3.9, now
causes a *note DeprecationWarning: 1a5. to be emitted when it
is used. (Contributed by Alex Waygood in gh-91896(22).)
* *note xml.etree.ElementTree: 125.: The module now emits *note
DeprecationWarning: 1a5. when testing the truth value of an *note
xml.etree.ElementTree.Element: 30a. Before, the Python
implementation emitted *note FutureWarning: 411, and the C
implementation emitted nothing. (Contributed by Jacob Walls in
gh-83122(23).)
* The 3-arg signatures (type, value, traceback) of *note coroutine
throw(): 4f3, *note generator throw(): 4f4. and *note async
generator throw(): 4f5. are deprecated and may be removed in a
future version of Python. Use the single-arg versions of these
functions instead. (Contributed by Ofey Chan in gh-89874(24).)
* *note DeprecationWarning: 1a5. is now raised when *note
__package__: 2db. on a module differs from *note __spec__.parent:
2dc. (previously it was *note ImportWarning: 4f6.). (Contributed
by Brett Cannon in gh-65961(25).)
* Setting *note __package__: 2db. or *note __cached__: 2d9. on a
module is deprecated, and will cease to be set or taken into
consideration by the import system in Python 3.14. (Contributed by
Brett Cannon in gh-65961(26).)
* The bitwise inversion operator (‘~’) on bool is deprecated. It
will throw an error in Python 3.16. Use ‘not’ for logical negation
of bools instead. In the rare case that you really need the
bitwise inversion of the underlying ‘int’, convert to int
explicitly: ‘~int(x)’. (Contributed by Tim Hoffmann in
gh-103487(27).)
* Accessing *note co_lnotab: 2e4. on code objects was deprecated in
Python 3.10 via PEP 626(28), but it only got a proper *note
DeprecationWarning: 1a5. in 3.12. May be removed in 3.15.
(Contributed by Nikita Sobolev in gh-101866(29).)
* Menu:
* Pending Removal in Python 3.13: Pending Removal in Python 3 13.
* Pending Removal in Python 3.14: Pending Removal in Python 3 14<3>.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15<3>.
* Pending removal in Python 3.16: Pending removal in Python 3 16<3>.
* Pending Removal in Future Versions: Pending Removal in Future Versions<3>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/92248
(2) https://github.com/python/cpython/issues/90953
(3) https://github.com/python/cpython/issues/94597
(4) https://github.com/python/cpython/issues/94597
(5) https://github.com/python/cpython/issues/100160
(6) https://github.com/python/cpython/issues/103636
(7) https://github.com/python/cpython/issues/91896
(8) https://github.com/python/cpython/issues/103857
(9) https://github.com/python/cpython/issues/72346
(10) https://github.com/python/cpython/issues/93963
(11) https://github.com/python/cpython/issues/101588
(12) https://github.com/python/cpython/issues/84559
(13) https://github.com/python/cpython/issues/97850
(14) https://github.com/python/cpython/issues/85984
(15) https://github.com/python/cpython/issues/99726
(16)
https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555
(17) https://github.com/python/cpython/issues/102828
(18) https://github.com/python/cpython/issues/90016
(19) https://github.com/python/cpython/issues/101698
(20) https://github.com/python/cpython/issues/102778
(21) https://github.com/python/cpython/issues/94309
(22) https://github.com/python/cpython/issues/91896
(23) https://github.com/python/cpython/issues/83122
(24) https://github.com/python/cpython/issues/89874
(25) https://github.com/python/cpython/issues/65961
(26) https://github.com/python/cpython/issues/65961
(27) https://github.com/python/cpython/issues/103487
(28) https://peps.python.org/pep-0626/
(29) https://github.com/python/cpython/issues/101866
File: python.info, Node: Pending Removal in Python 3 13, Next: Pending Removal in Python 3 14<3>, Up: Deprecated
1.2.10.1 Pending Removal in Python 3.13
.......................................
Modules (see PEP 594(1)):
* ‘aifc’
* ‘audioop’
* ‘cgi’
* ‘cgitb’
* ‘chunk’
* ‘crypt’
* ‘imghdr’
* ‘mailcap’
* ‘msilib’
* ‘nis’
* ‘nntplib’
* ‘ossaudiodev’
* ‘pipes’
* ‘sndhdr’
* ‘spwd’
* ‘sunau’
* ‘telnetlib’
* ‘uu’
* ‘xdrlib’
Other modules:
* ‘lib2to3’, and the ‘2to3’ program (gh-84540(2))
APIs:
* ‘configparser.LegacyInterpolation’ (gh-90765(3))
* ‘locale.resetlocale()’ (gh-90817(4))
* ‘turtle.RawTurtle.settiltangle()’ (gh-50096(5))
* ‘unittest.findTestCases()’ (gh-50096(6))
* ‘unittest.getTestCaseNames()’ (gh-50096(7))
* ‘unittest.makeSuite()’ (gh-50096(8))
* ‘unittest.TestProgram.usageExit()’ (gh-67048(9))
* ‘webbrowser.MacOSX’ (gh-86421(10))
* *note classmethod: 166. descriptor chaining (gh-89519(11))
* *note importlib.resources: 7b. deprecated methods:
* ‘contents()’
* ‘is_resource()’
* ‘open_binary()’
* ‘open_text()’
* ‘path()’
* ‘read_binary()’
* ‘read_text()’
Use *note importlib.resources.files(): 48a. instead. Refer to
importlib-resources: Migrating from Legacy(12) (gh-106531(13))
---------- Footnotes ----------
(1) https://peps.python.org/pep-0594/
(2) https://github.com/python/cpython/issues/84540
(3) https://github.com/python/cpython/issues/90765
(4) https://github.com/python/cpython/issues/90817
(5) https://github.com/python/cpython/issues/50096
(6) https://github.com/python/cpython/issues/50096
(7) https://github.com/python/cpython/issues/50096
(8) https://github.com/python/cpython/issues/50096
(9) https://github.com/python/cpython/issues/67048
(10) https://github.com/python/cpython/issues/86421
(11) https://github.com/python/cpython/issues/89519
(12)
https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy
(13) https://github.com/python/cpython/issues/106531
File: python.info, Node: Pending Removal in Python 3 14<3>, Next: Pending Removal in Python 3 15<3>, Prev: Pending Removal in Python 3 13, Up: Deprecated
1.2.10.2 Pending Removal in Python 3.14
.......................................
* *note argparse: 6.: The 'type', 'choices', and 'metavar' parameters
of ‘argparse.BooleanOptionalAction’ are deprecated and will be
removed in 3.14. (Contributed by Nikita Sobolev in gh-92248(1).)
* *note ast: 8.: The following features have been deprecated in
documentation since Python 3.8, now cause a *note
DeprecationWarning: 1a5. to be emitted at runtime when they are
accessed or used, and will be removed in Python 3.14:
* ‘ast.Num’
* ‘ast.Str’
* ‘ast.Bytes’
* ‘ast.NameConstant’
* ‘ast.Ellipsis’
Use *note ast.Constant: 2bb. instead. (Contributed by Serhiy
Storchaka in gh-90953(2).)
* *note asyncio: a.:
* The child watcher classes *note MultiLoopChildWatcher: 2bc,
*note FastChildWatcher: 2bd, *note AbstractChildWatcher: 2be.
and *note SafeChildWatcher: 2bf. are deprecated and will be
removed in Python 3.14. (Contributed by Kumar Aditya in
gh-94597(3).)
* *note asyncio.set_child_watcher(): 2c0, *note
asyncio.get_child_watcher(): 2c1, *note
asyncio.AbstractEventLoopPolicy.set_child_watcher(): 2c2. and
*note asyncio.AbstractEventLoopPolicy.get_child_watcher():
2c3. are deprecated and will be removed in Python 3.14.
(Contributed by Kumar Aditya in gh-94597(4).)
* The *note get_event_loop(): 2c4. method of the default event
loop policy now emits a *note DeprecationWarning: 1a5. if
there is no current event loop set and it decides to create
one. (Contributed by Serhiy Storchaka and Guido van Rossum in
gh-100160(5).)
* *note collections.abc: 1e.: Deprecated *note ByteString: 2c5.
Prefer ‘Sequence’ or *note Buffer: 2c6. For use in typing, prefer
a union, like ‘bytes | bytearray’, or *note collections.abc.Buffer:
2c6. (Contributed by Shantanu Jain in gh-91896(6).)
* *note email: 3b.: Deprecated the 'isdst' parameter in *note
email.utils.localtime(): 2c7. (Contributed by Alan Williams in
gh-72346(7).)
* *note importlib.abc: 78. deprecated classes:
* ‘importlib.abc.ResourceReader’
* ‘importlib.abc.Traversable’
* ‘importlib.abc.TraversableResources’
Use *note importlib.resources.abc: 7c. classes instead:
* *note importlib.resources.abc.Traversable: 1f5.
* *note importlib.resources.abc.TraversableResources: 2c8.
(Contributed by Jason R. Coombs and Hugo van Kemenade in
gh-93963(8).)
* *note itertools: 81. had undocumented, inefficient, historically
buggy, and inconsistent support for copy, deepcopy, and pickle
operations. This will be removed in 3.14 for a significant
reduction in code volume and maintenance burden. (Contributed by
Raymond Hettinger in gh-101588(9).)
* *note multiprocessing: 94.: The default start method will change to
a safer one on Linux, BSDs, and other non-macOS POSIX platforms
where ‘'fork'’ is currently the default (gh-84559(10)). Adding a
runtime warning about this was deemed too disruptive as the
majority of code is not expected to care. Use the *note
get_context(): 2c9. or *note set_start_method(): 2ca. APIs to
explicitly specify when your code 'requires' ‘'fork'’. See *note
Contexts and start methods: 2cb.
* *note pathlib: a4.: *note is_relative_to(): 2cc. and *note
relative_to(): 2cd.: passing additional arguments is deprecated.
* *note pkgutil: a9.: *note find_loader(): 2ce. and *note
get_loader(): 2cf. now raise *note DeprecationWarning: 1a5.; use
*note importlib.util.find_spec(): 2d0. instead. (Contributed by
Nikita Sobolev in gh-97850(11).)
* *note pty: b1.:
* ‘master_open()’: use *note pty.openpty(): 2d1.
* ‘slave_open()’: use *note pty.openpty(): 2d1.
* *note sqlite3: cf.:
* *note version: 2d2. and *note version_info: 2d3.
* *note execute(): 2d4. and *note executemany(): 2d5. if *note
named placeholders: 2d6. are used and 'parameters' is a
sequence instead of a *note dict: 258.
* *note typing: 104.: *note ByteString: 2d7, deprecated since Python
3.9, now causes a *note DeprecationWarning: 1a5. to be emitted when
it is used.
* *note urllib: 108.: ‘urllib.parse.Quoter’ is deprecated: it was not
intended to be a public API. (Contributed by Gregory P. Smith in
gh-88168(12).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/92248
(2) https://github.com/python/cpython/issues/90953
(3) https://github.com/python/cpython/issues/94597
(4) https://github.com/python/cpython/issues/94597
(5) https://github.com/python/cpython/issues/100160
(6) https://github.com/python/cpython/issues/91896
(7) https://github.com/python/cpython/issues/72346
(8) https://github.com/python/cpython/issues/93963
(9) https://github.com/python/cpython/issues/101588
(10) https://github.com/python/cpython/issues/84559
(11) https://github.com/python/cpython/issues/97850
(12) https://github.com/python/cpython/issues/88168
File: python.info, Node: Pending Removal in Python 3 15<3>, Next: Pending removal in Python 3 16<3>, Prev: Pending Removal in Python 3 14<3>, Up: Deprecated
1.2.10.3 Pending Removal in Python 3.15
.......................................
* The import system:
* Setting *note __cached__: 2d9. on a module while failing to
set *note __spec__.cached: 2da. is deprecated. In Python
3.15, ‘__cached__’ will cease to be set or take into
consideration by the import system or standard library.
(gh-97879(1))
* Setting *note __package__: 2db. on a module while failing to
set *note __spec__.parent: 2dc. is deprecated. In Python
3.15, ‘__package__’ will cease to be set or take into
consideration by the import system or standard library.
(gh-97879(2))
* *note ctypes: 2a.:
* The undocumented ‘ctypes.SetPointerType()’ function has been
deprecated since Python 3.13.
* *note http.server: 72.:
* The obsolete and rarely used *note CGIHTTPRequestHandler: 2a3.
has been deprecated since Python 3.13. No direct replacement
exists. 'Anything' is better than CGI to interface a web
server with a request handler.
* The ‘--cgi’ flag to the ‘python -m http.server’ command-line
interface has been deprecated since Python 3.13.
* *note importlib: 77.:
* ‘load_module()’ method: use ‘exec_module()’ instead.
* *note locale: 86.:
* The *note getdefaultlocale(): 2dd. function has been
deprecated since Python 3.11. Its removal was originally
planned for Python 3.13 (gh-90817(3)), but has been postponed
to Python 3.15. Use *note getlocale(): 2de, *note
setlocale(): 2df, and *note getencoding(): 2e0. instead.
(Contributed by Hugo van Kemenade in gh-111187(4).)
* *note pathlib: a4.:
* *note PurePath.is_reserved(): 2a8. has been deprecated since
Python 3.13. Use *note os.path.isreserved(): 225. to detect
reserved paths on Windows.
* *note platform: aa.:
* *note java_ver(): 2a9. has been deprecated since Python 3.13.
This function is only useful for Jython support, has a
confusing API, and is largely untested.
* *note sysconfig: db.:
* The 'check_home' argument of *note
sysconfig.is_python_build(): 2e1. has been deprecated since
Python 3.12.
* *note threading: ed.:
* *note RLock(): 2e2. will take no arguments in Python 3.15.
Passing any arguments has been deprecated since Python 3.14,
as the Python version does not permit any arguments, but the C
version allows any number of positional or keyword arguments,
ignoring every argument.
* *note types: 103.:
* *note types.CodeType: 2e3.: Accessing *note co_lnotab: 2e4.
was deprecated in PEP 626(5) since 3.10 and was planned to be
removed in 3.12, but it only got a proper *note
DeprecationWarning: 1a5. in 3.12. May be removed in 3.15.
(Contributed by Nikita Sobolev in gh-101866(6).)
* *note typing: 104.:
* The undocumented keyword argument syntax for creating *note
NamedTuple: 2b2. classes (e.g. ‘Point = NamedTuple("Point",
x=int, y=int)’) has been deprecated since Python 3.13. Use
the class-based syntax or the functional syntax instead.
* When using the functional syntax of *note TypedDict: 162.s,
failing to pass a value to the 'fields' parameter (‘TD =
TypedDict("TD")’) or passing ‘None’ (‘TD = TypedDict("TD",
None)’) has been deprecated since Python 3.13. Use ‘class
TD(TypedDict): pass’ or ‘TD = TypedDict("TD", {})’ to create a
TypedDict with zero field.
* The *note typing.no_type_check_decorator(): 2b3. decorator
function has been deprecated since Python 3.13. After eight
years in the *note typing: 104. module, it has yet to be
supported by any major type checker.
* *note wave: 113.:
* The *note getmark(): 2b6, ‘setmark()’, and *note getmarkers():
2b7. methods of the *note Wave_read: 2b8. and *note
Wave_write: 2b9. classes have been deprecated since Python
3.13.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/97879
(2) https://github.com/python/cpython/issues/97879
(3) https://github.com/python/cpython/issues/90817
(4) https://github.com/python/cpython/issues/111187
(5) https://peps.python.org/pep-0626/
(6) https://github.com/python/cpython/issues/101866
File: python.info, Node: Pending removal in Python 3 16<3>, Next: Pending Removal in Future Versions<3>, Prev: Pending Removal in Python 3 15<3>, Up: Deprecated
1.2.10.4 Pending removal in Python 3.16
.......................................
* The import system:
* Setting *note __loader__: 2e6. on a module while failing to
set *note __spec__.loader: 2e7. is deprecated. In Python
3.16, ‘__loader__’ will cease to be set or taken into
consideration by the import system or the standard library.
* *note array: 7.:
* The ‘'u'’ format code (‘wchar_t’) has been deprecated in
documentation since Python 3.3 and at runtime since Python
3.13. Use the ‘'w'’ format code (*note Py_UCS4: 29d.) for
Unicode characters instead.
* *note asyncio: a.:
* ‘asyncio.iscoroutinefunction()’ is deprecated and will be
removed in Python 3.16, use *note
inspect.iscoroutinefunction(): 2e8. instead. (Contributed by
Jiahao Li and Kumar Aditya in gh-122875(1).)
* *note builtins: 12.:
* Bitwise inversion on boolean types, ‘~True’ or ‘~False’ has
been deprecated since Python 3.12, as it produces surprising
and unintuitive results (‘-2’ and ‘-1’). Use ‘not x’ instead
for the logical negation of a Boolean. In the rare case that
you need the bitwise inversion of the underlying integer,
convert to ‘int’ explicitly (‘~int(x)’).
* *note shutil: c5.:
* The ‘ExecError’ exception has been deprecated since Python
3.14. It has not been used by any function in ‘shutil’ since
Python 3.4, and is now an alias of *note RuntimeError: 195.
* *note symtable: d8.:
* The *note Class.get_methods: 2e9. method has been deprecated
since Python 3.14.
* *note sys: d9.:
* The *note _enablelegacywindowsfsencoding(): 2b0. function has
been deprecated since Python 3.13. Use the *note
PYTHONLEGACYWINDOWSFSENCODING: 2b1. environment variable
instead.
* *note tarfile: de.:
* The undocumented and unused ‘TarFile.tarfile’ attribute has
been deprecated since Python 3.13.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/122875
File: python.info, Node: Pending Removal in Future Versions<3>, Prev: Pending removal in Python 3 16<3>, Up: Deprecated
1.2.10.5 Pending Removal in Future Versions
...........................................
The following APIs will be removed in the future, although there is
currently no date scheduled for their removal.
* *note argparse: 6.: Nesting argument groups and nesting mutually
exclusive groups are deprecated.
* *note builtins: 12.:
* ‘bool(NotImplemented)’.
* Generators: ‘throw(type, exc, tb)’ and ‘athrow(type, exc, tb)’
signature is deprecated: use ‘throw(exc)’ and ‘athrow(exc)’
instead, the single argument signature.
* Currently Python accepts numeric literals immediately followed
by keywords, for example ‘0in x’, ‘1or x’, ‘0if 1else 2’. It
allows confusing and ambiguous expressions like ‘[0x1for x in
y]’ (which can be interpreted as ‘[0x1 for x in y]’ or ‘[0x1f
or x in y]’). A syntax warning is raised if the numeric
literal is immediately followed by one of keywords *note and:
2eb, *note else: 18c, *note for: 2ec, *note if: 2ed, *note in:
2ee, *note is: 2ef. and *note or: 2f0. In a future release it
will be changed to a syntax error. (gh-87999(1))
* Support for ‘__index__()’ and ‘__int__()’ method returning
non-int type: these methods will be required to return an
instance of a strict subclass of *note int: 259.
* Support for ‘__float__()’ method returning a strict subclass
of *note float: 2f1.: these methods will be required to return
an instance of *note float: 2f1.
* Support for ‘__complex__()’ method returning a strict subclass
of *note complex: 2f2.: these methods will be required to
return an instance of *note complex: 2f2.
* Delegation of ‘int()’ to ‘__trunc__()’ method.
* Passing a complex number as the 'real' or 'imag' argument in
the *note complex(): 2f2. constructor is now deprecated; it
should only be passed as a single positional argument.
(Contributed by Serhiy Storchaka in gh-109218(2).)
* *note calendar: 14.: ‘calendar.January’ and ‘calendar.February’
constants are deprecated and replaced by *note calendar.JANUARY:
2f3. and *note calendar.FEBRUARY: 2f4. (Contributed by Prince
Roshan in gh-103636(3).)
* *note codeobject.co_lnotab: 2e4.: use the *note
codeobject.co_lines(): 2f5. method instead.
* *note datetime: 30.:
* *note utcnow(): 2f6.: use
‘datetime.datetime.now(tz=datetime.UTC)’.
* *note utcfromtimestamp(): 2f7.: use
‘datetime.datetime.fromtimestamp(timestamp, tz=datetime.UTC)’.
* *note gettext: 63.: Plural value must be an integer.
* *note importlib: 77.:
* *note cache_from_source(): 2f8. 'debug_override' parameter is
deprecated: use the 'optimization' parameter instead.
* *note importlib.metadata: 7a.:
* ‘EntryPoints’ tuple interface.
* Implicit ‘None’ on return values.
* *note logging: 87.: the ‘warn()’ method has been deprecated since
Python 3.3, use *note warning(): 2f9. instead.
* *note mailbox: 8b.: Use of StringIO input and text mode is
deprecated, use BytesIO and binary mode instead.
* *note os: a1.: Calling *note os.register_at_fork(): 2fa. in
multi-threaded process.
* ‘pydoc.ErrorDuringImport’: A tuple value for 'exc_info' parameter
is deprecated, use an exception instance.
* *note re: b9.: More strict rules are now applied for numerical
group references and group names in regular expressions. Only
sequence of ASCII digits is now accepted as a numerical reference.
The group name in bytes patterns and replacement strings can now
only contain ASCII letters and digits and underscore. (Contributed
by Serhiy Storchaka in gh-91760(4).)
* ‘sre_compile’, ‘sre_constants’ and ‘sre_parse’ modules.
* *note shutil: c5.: *note rmtree(): 2fb.’s 'onerror' parameter is
deprecated in Python 3.12; use the 'onexc' parameter instead.
* *note ssl: d0. options and protocols:
* *note ssl.SSLContext: 296. without protocol argument is
deprecated.
* *note ssl.SSLContext: 296.: *note set_npn_protocols(): 2fc.
and ‘selected_npn_protocol()’ are deprecated: use ALPN
instead.
* ‘ssl.OP_NO_SSL*’ options
* ‘ssl.OP_NO_TLS*’ options
* ‘ssl.PROTOCOL_SSLv3’
* ‘ssl.PROTOCOL_TLS’
* ‘ssl.PROTOCOL_TLSv1’
* ‘ssl.PROTOCOL_TLSv1_1’
* ‘ssl.PROTOCOL_TLSv1_2’
* ‘ssl.TLSVersion.SSLv3’
* ‘ssl.TLSVersion.TLSv1’
* ‘ssl.TLSVersion.TLSv1_1’
* *note threading: ed. methods:
* ‘threading.Condition.notifyAll()’: use *note notify_all():
2fd.
* ‘threading.Event.isSet()’: use *note is_set(): 2fe.
* ‘threading.Thread.isDaemon()’, *note
threading.Thread.setDaemon(): 2ff.: use *note
threading.Thread.daemon: 300. attribute.
* ‘threading.Thread.getName()’, *note
threading.Thread.setName(): 301.: use *note
threading.Thread.name: 302. attribute.
* ‘threading.currentThread()’: use *note
threading.current_thread(): 303.
* ‘threading.activeCount()’: use *note threading.active_count():
304.
* *note typing.Text: 305. (gh-92332(5)).
* *note unittest.IsolatedAsyncioTestCase: 306.: it is deprecated to
return a value that is not ‘None’ from a test case.
* *note urllib.parse: 10a. deprecated functions: *note urlparse():
307. instead
* ‘splitattr()’
* ‘splithost()’
* ‘splitnport()’
* ‘splitpasswd()’
* ‘splitport()’
* ‘splitquery()’
* ‘splittag()’
* ‘splittype()’
* ‘splituser()’
* ‘splitvalue()’
* ‘to_bytes()’
* *note urllib.request: 10b.: *note URLopener: 308. and *note
FancyURLopener: 309. style of invoking requests is deprecated. Use
newer *note urlopen(): 295. functions and methods.
* *note wsgiref: 118.: ‘SimpleHandler.stdout.write()’ should not do
partial writes.
* *note xml.etree.ElementTree: 125.: Testing the truth value of an
*note Element: 30a. is deprecated. In a future release it will
always return ‘True’. Prefer explicit ‘len(elem)’ or ‘elem is not
None’ tests instead.
* *note zipimport.zipimporter.load_module(): 30b. is deprecated: use
*note exec_module(): 30c. instead.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/87999
(2) https://github.com/python/cpython/issues/109218
(3) https://github.com/python/cpython/issues/103636
(4) https://github.com/python/cpython/issues/91760
(5) https://github.com/python/cpython/issues/92332
File: python.info, Node: Removed, Next: Porting to Python 3 12, Prev: Deprecated, Up: What’s New In Python 3 12
1.2.11 Removed
--------------
* Menu:
* asynchat and asyncore::
* configparser: configparser<3>.
* distutils::
* ensurepip::
* enum: enum<2>.
* ftplib::
* gzip::
* hashlib::
* importlib: importlib<2>.
* imp::
* io: io<2>.
* locale: locale<2>.
* smtpd::
* sqlite3: sqlite3<3>.
* ssl: ssl<2>.
* unittest: unittest<3>.
* webbrowser: webbrowser<2>.
* xml.etree.ElementTree: xml etree ElementTree.
* zipimport: zipimport<2>.
* Others::
File: python.info, Node: asynchat and asyncore, Next: configparser<3>, Up: Removed
1.2.11.1 asynchat and asyncore
..............................
* These two modules have been removed according to the schedule in
PEP 594(1), having been deprecated in Python 3.6. Use *note
asyncio: a. instead. (Contributed by Nikita Sobolev in
gh-96580(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0594/
(2) https://github.com/python/cpython/issues/96580
File: python.info, Node: configparser<3>, Next: distutils, Prev: asynchat and asyncore, Up: Removed
1.2.11.2 configparser
.....................
* Several names deprecated in the *note configparser: 22. way back in
3.2 have been removed per gh-89336(1):
* *note configparser.ParsingError: 500. no longer has a
‘filename’ attribute or argument. Use the ‘source’ attribute
and argument instead.
* *note configparser: 22. no longer has a ‘SafeConfigParser’
class. Use the shorter *note ConfigParser: 1c8. name instead.
* *note configparser.ConfigParser: 1c8. no longer has a ‘readfp’
method. Use *note read_file(): 501. instead.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/89336
File: python.info, Node: distutils, Next: ensurepip, Prev: configparser<3>, Up: Removed
1.2.11.3 distutils
..................
* Remove the ‘distutils’ package. It was deprecated in Python 3.10
by PEP 632(1) “Deprecate distutils module”. For projects still
using ‘distutils’ and cannot be updated to something else, the
‘setuptools’ project can be installed: it still provides
‘distutils’. (Contributed by Victor Stinner in gh-92584(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0632/
(2) https://github.com/python/cpython/issues/92584
File: python.info, Node: ensurepip, Next: enum<2>, Prev: distutils, Up: Removed
1.2.11.4 ensurepip
..................
* Remove the bundled setuptools wheel from *note ensurepip: 55, and
stop installing setuptools in environments created by *note venv:
111.
‘pip (>= 22.1)’ does not require setuptools to be installed in the
environment. ‘setuptools’-based (and ‘distutils’-based) packages
can still be used with ‘pip install’, since pip will provide
‘setuptools’ in the build environment it uses for building a
package.
‘easy_install’, ‘pkg_resources’, ‘setuptools’ and ‘distutils’ are
no longer provided by default in environments created with ‘venv’
or bootstrapped with ‘ensurepip’, since they are part of the
‘setuptools’ package. For projects relying on these at runtime,
the ‘setuptools’ project should be declared as a dependency and
installed separately (typically, using pip).
(Contributed by Pradyun Gedam in gh-95299(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/95299
File: python.info, Node: enum<2>, Next: ftplib, Prev: ensurepip, Up: Removed
1.2.11.5 enum
.............
* Remove *note enum: 56.’s ‘EnumMeta.__getattr__’, which is no longer
needed for enum attribute access. (Contributed by Ethan Furman in
gh-95083(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/95083
File: python.info, Node: ftplib, Next: gzip, Prev: enum<2>, Up: Removed
1.2.11.6 ftplib
...............
* Remove *note ftplib: 5e.’s ‘FTP_TLS.ssl_version’ class attribute:
use the 'context' parameter instead. (Contributed by Victor
Stinner in gh-94172(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/94172
File: python.info, Node: gzip, Next: hashlib, Prev: ftplib, Up: Removed
1.2.11.7 gzip
.............
* Remove the ‘filename’ attribute of *note gzip: 67.’s *note
gzip.GzipFile: 416, deprecated since Python 2.6, use the *note
name: 508. attribute instead. In write mode, the ‘filename’
attribute added ‘'.gz'’ file extension if it was not present.
(Contributed by Victor Stinner in gh-94196(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/94196
File: python.info, Node: hashlib, Next: importlib<2>, Prev: gzip, Up: Removed
1.2.11.8 hashlib
................
* Remove the pure Python implementation of *note hashlib: 68.’s *note
hashlib.pbkdf2_hmac(): 50a, deprecated in Python 3.10. Python 3.10
and newer requires OpenSSL 1.1.1 ( PEP 644(1)): this OpenSSL
version provides a C implementation of *note pbkdf2_hmac(): 50a.
which is faster. (Contributed by Victor Stinner in gh-94199(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0644/
(2) https://github.com/python/cpython/issues/94199
File: python.info, Node: importlib<2>, Next: imp, Prev: hashlib, Up: Removed
1.2.11.9 importlib
..................
* Many previously deprecated cleanups in *note importlib: 77. have
now been completed:
* References to, and support for ‘module_repr()’ has been
removed. (Contributed by Barry Warsaw in gh-97850(1).)
* ‘importlib.util.set_package’, ‘importlib.util.set_loader’ and
‘importlib.util.module_for_loader’ have all been removed.
(Contributed by Brett Cannon and Nikita Sobolev in gh-65961(2)
and gh-97850(3).)
* Support for ‘find_loader()’ and ‘find_module()’ APIs have been
removed. (Contributed by Barry Warsaw in gh-98040(4).)
* ‘importlib.abc.Finder’, ‘pkgutil.ImpImporter’, and
‘pkgutil.ImpLoader’ have been removed. (Contributed by Barry
Warsaw in gh-98040(5).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/97850
(2) https://github.com/python/cpython/issues/65961
(3) https://github.com/python/cpython/issues/97850
(4) https://github.com/python/cpython/issues/98040
(5) https://github.com/python/cpython/issues/98040
File: python.info, Node: imp, Next: io<2>, Prev: importlib<2>, Up: Removed
1.2.11.10 imp
.............
* The ‘imp’ module has been removed. (Contributed by Barry Warsaw in
gh-98040(1).)
To migrate, consult the following correspondence table:
imp importlib
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
‘imp.NullImporter’ Insert ‘None’ into ‘sys.path_importer_cache’
‘imp.cache_from_source()’ *note importlib.util.cache_from_source(): 2f8.
‘imp.find_module()’ *note importlib.util.find_spec(): 2d0.
‘imp.get_magic()’ *note importlib.util.MAGIC_NUMBER: 50e.
‘imp.get_suffixes()’ *note importlib.machinery.SOURCE_SUFFIXES: 50f, *note importlib.machinery.EXTENSION_SUFFIXES: 510, and
*note importlib.machinery.BYTECODE_SUFFIXES: 511.
‘imp.get_tag()’ *note sys.implementation.cache_tag: 512.
‘imp.load_module()’ *note importlib.import_module(): 513.
‘imp.new_module(name)’ ‘types.ModuleType(name)’
‘imp.reload()’ *note importlib.reload(): 514.
‘imp.source_from_cache()’ *note importlib.util.source_from_cache(): 515.
‘imp.load_source()’ 'See below'
Replace ‘imp.load_source()’ with:
import importlib.util
import importlib.machinery
def load_source(modname, filename):
loader = importlib.machinery.SourceFileLoader(modname, filename)
spec = importlib.util.spec_from_file_location(modname, filename, loader=loader)
module = importlib.util.module_from_spec(spec)
# The module is always executed and not cached in sys.modules.
# Uncomment the following line to cache the module.
# sys.modules[module.__name__] = module
loader.exec_module(module)
return module
* Remove ‘imp’ functions and attributes with no replacements:
* Undocumented functions:
* ‘imp.init_builtin()’
* ‘imp.load_compiled()’
* ‘imp.load_dynamic()’
* ‘imp.load_package()’
* ‘imp.lock_held()’, ‘imp.acquire_lock()’, ‘imp.release_lock()’:
the locking scheme has changed in Python 3.3 to per-module
locks.
* ‘imp.find_module()’ constants: ‘SEARCH_ERROR’, ‘PY_SOURCE’,
‘PY_COMPILED’, ‘C_EXTENSION’, ‘PY_RESOURCE’, ‘PKG_DIRECTORY’,
‘C_BUILTIN’, ‘PY_FROZEN’, ‘PY_CODERESOURCE’, ‘IMP_HOOK’.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/98040
File: python.info, Node: io<2>, Next: locale<2>, Prev: imp, Up: Removed
1.2.11.11 io
............
* Remove *note io: 7f.’s ‘io.OpenWrapper’ and ‘_pyio.OpenWrapper’,
deprecated in Python 3.10: just use *note open(): 517. instead.
The *note open(): 517. (*note io.open(): 518.) function is a
built-in function. Since Python 3.10, ‘_pyio.open()’ is also a
static method. (Contributed by Victor Stinner in gh-94169(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/94169
File: python.info, Node: locale<2>, Next: smtpd, Prev: io<2>, Up: Removed
1.2.11.12 locale
................
* Remove *note locale: 86.’s ‘locale.format()’ function, deprecated
in Python 3.7: use *note locale.format_string(): 51a. instead.
(Contributed by Victor Stinner in gh-94226(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/94226
File: python.info, Node: smtpd, Next: sqlite3<3>, Prev: locale<2>, Up: Removed
1.2.11.13 smtpd
...............
* The ‘smtpd’ module has been removed according to the schedule in
PEP 594(1), having been deprecated in Python 3.4.7 and 3.5.4. Use
the aiosmtpd(2) PyPI module or any other *note asyncio: a.-based
server instead. (Contributed by Oleg Iarygin in gh-93243(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0594/
(2) https://pypi.org/project/aiosmtpd/
(3) https://github.com/python/cpython/issues/93243
File: python.info, Node: sqlite3<3>, Next: ssl<2>, Prev: smtpd, Up: Removed
1.2.11.14 sqlite3
.................
* The following undocumented *note sqlite3: cf. features, deprecated
in Python 3.10, are now removed:
* ‘sqlite3.enable_shared_cache()’
* ‘sqlite3.OptimizedUnicode’
If a shared cache must be used, open the database in URI mode using
the ‘cache=shared’ query parameter.
The ‘sqlite3.OptimizedUnicode’ text factory has been an alias for
*note str: 447. since Python 3.3. Code that previously set the
text factory to ‘OptimizedUnicode’ can either use ‘str’ explicitly,
or rely on the default value which is also ‘str’.
(Contributed by Erlend E. Aasland in gh-92548(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/92548
File: python.info, Node: ssl<2>, Next: unittest<3>, Prev: sqlite3<3>, Up: Removed
1.2.11.15 ssl
.............
* Remove *note ssl: d0.’s ‘ssl.RAND_pseudo_bytes()’ function,
deprecated in Python 3.6: use *note os.urandom(): 51e. or *note
ssl.RAND_bytes(): 51f. instead. (Contributed by Victor Stinner in
gh-94199(1).)
* Remove the ‘ssl.match_hostname()’ function. It was deprecated in
Python 3.7. OpenSSL performs hostname matching since Python 3.7,
Python no longer uses the ‘ssl.match_hostname()’ function.
(Contributed by Victor Stinner in gh-94199(2).)
* Remove the ‘ssl.wrap_socket()’ function, deprecated in Python 3.7:
instead, create a *note ssl.SSLContext: 296. object and call its
*note ssl.SSLContext.wrap_socket: 520. method. Any package that
still uses ‘ssl.wrap_socket()’ is broken and insecure. The
function neither sends a SNI TLS extension nor validates the server
hostname. Code is subject to CWE 295(3) (Improper Certificate
Validation). (Contributed by Victor Stinner in gh-94199(4).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/94199
(2) https://github.com/python/cpython/issues/94199
(3) https://cwe.mitre.org/data/definitions/295.html
(4) https://github.com/python/cpython/issues/94199
File: python.info, Node: unittest<3>, Next: webbrowser<2>, Prev: ssl<2>, Up: Removed
1.2.11.16 unittest
..................
* Remove many long-deprecated *note unittest: 106. features:
* A number of *note TestCase: 449. method aliases:
Deprecated alias Method Name Deprecated in
-----------------------------------------------------------------------------------------
‘failUnless’ *note assertTrue(): 522. 3.1
‘failIf’ *note assertFalse(): 523. 3.1
‘failUnlessEqual’ *note assertEqual(): 524. 3.1
‘failIfEqual’ *note assertNotEqual(): 525. 3.1
‘failUnlessAlmostEqual’ *note assertAlmostEqual(): 526. 3.1
‘failIfAlmostEqual’ *note assertNotAlmostEqual(): 527. 3.1
‘failUnlessRaises’ *note assertRaises(): 528. 3.1
‘assert_’ *note assertTrue(): 522. 3.2
‘assertEquals’ *note assertEqual(): 524. 3.2
‘assertNotEquals’ *note assertNotEqual(): 525. 3.2
‘assertAlmostEquals’ *note assertAlmostEqual(): 526. 3.2
‘assertNotAlmostEquals’ *note assertNotAlmostEqual(): 527. 3.2
‘assertRegexpMatches’ *note assertRegex(): 529. 3.2
‘assertRaisesRegexp’ *note assertRaisesRegex(): 52a. 3.2
‘assertNotRegexpMatches’ *note assertNotRegex(): 52b. 3.5
You can use ‘https://github.com/isidentical/teyit’ to
automatically modernise your unit tests.
* Undocumented and broken *note TestCase: 449. method
‘assertDictContainsSubset’ (deprecated in Python 3.2).
* Undocumented *note TestLoader.loadTestsFromModule: 291.
parameter 'use_load_tests' (deprecated and ignored since
Python 3.5).
* An alias of the *note TextTestResult: 52c. class:
‘_TextTestResult’ (deprecated in Python 3.2).
(Contributed by Serhiy Storchaka in gh-89325(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/89325
File: python.info, Node: webbrowser<2>, Next: xml etree ElementTree, Prev: unittest<3>, Up: Removed
1.2.11.17 webbrowser
....................
* Remove support for obsolete browsers from *note webbrowser: 115.
The removed browsers include: Grail, Mosaic, Netscape, Galeon,
Skipstone, Iceape, Firebird, and Firefox versions 35 and below
(gh-102871(1)).
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/102871
File: python.info, Node: xml etree ElementTree, Next: zipimport<2>, Prev: webbrowser<2>, Up: Removed
1.2.11.18 xml.etree.ElementTree
...............................
* Remove the ‘ElementTree.Element.copy()’ method of the pure Python
implementation, deprecated in Python 3.10, use the *note
copy.copy(): 52f. function instead. The C implementation of *note
xml.etree.ElementTree: 125. has no ‘copy()’ method, only a
‘__copy__()’ method. (Contributed by Victor Stinner in
gh-94383(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/94383
File: python.info, Node: zipimport<2>, Next: Others, Prev: xml etree ElementTree, Up: Removed
1.2.11.19 zipimport
...................
* Remove *note zipimport: 132.’s ‘find_loader()’ and ‘find_module()’
methods, deprecated in Python 3.10: use the ‘find_spec()’ method
instead. See PEP 451(1) for the rationale. (Contributed by Victor
Stinner in gh-94379(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0451/
(2) https://github.com/python/cpython/issues/94379
File: python.info, Node: Others, Prev: zipimport<2>, Up: Removed
1.2.11.20 Others
................
* Remove the ‘suspicious’ rule from the documentation ‘Makefile’ and
‘Doc/tools/rstlint.py’, both in favor of sphinx-lint(1).
(Contributed by Julien Palard in gh-98179(2).)
* Remove the 'keyfile' and 'certfile' parameters from the *note
ftplib: 5e, *note imaplib: 74, *note poplib: ac. and *note smtplib:
ca. modules, and the 'key_file', 'cert_file' and 'check_hostname'
parameters from the *note http.client: 6f. module, all deprecated
since Python 3.6. Use the 'context' parameter ('ssl_context' in
*note imaplib: 74.) instead. (Contributed by Victor Stinner in
gh-94172(3).)
* Remove ‘Jython’ compatibility hacks from several stdlib modules and
tests. (Contributed by Nikita Sobolev in gh-99482(4).)
* Remove ‘_use_broken_old_ctypes_structure_semantics_’ flag from
*note ctypes: 2a. module. (Contributed by Nikita Sobolev in
gh-99285(5).)
---------- Footnotes ----------
(1) https://github.com/sphinx-contrib/sphinx-lint
(2) https://github.com/python/cpython/issues/98179
(3) https://github.com/python/cpython/issues/94172
(4) https://github.com/python/cpython/issues/99482
(5) https://github.com/python/cpython/issues/99285
File: python.info, Node: Porting to Python 3 12, Next: Build Changes<2>, Prev: Removed, Up: What’s New In Python 3 12
1.2.12 Porting to Python 3.12
-----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in the Python API: Changes in the Python API<2>.
File: python.info, Node: Changes in the Python API<2>, Up: Porting to Python 3 12
1.2.12.1 Changes in the Python API
..................................
* More strict rules are now applied for numerical group references
and group names in regular expressions. Only sequence of ASCII
digits is now accepted as a numerical reference. The group name in
bytes patterns and replacement strings can now only contain ASCII
letters and digits and underscore. (Contributed by Serhiy
Storchaka in gh-91760(1).)
* Remove ‘randrange()’ functionality deprecated since Python 3.10.
Formerly, ‘randrange(10.0)’ losslessly converted to
‘randrange(10)’. Now, it raises a *note TypeError: 534. Also, the
exception raised for non-integer values such as ‘randrange(10.5)’
or ‘randrange('10')’ has been changed from *note ValueError: 204.
to *note TypeError: 534. This also prevents bugs where
‘randrange(1e25)’ would silently select from a larger range than
‘randrange(10**25)’. (Originally suggested by Serhiy Storchaka
gh-86388(2).)
* *note argparse.ArgumentParser: 535. changed encoding and error
handler for reading arguments from file (e.g.
‘fromfile_prefix_chars’ option) from default text encoding (e.g.
*note locale.getpreferredencoding(False): 536.) to *note filesystem
encoding and error handler: 537. Argument files should be encoded
in UTF-8 instead of ANSI Codepage on Windows.
* Remove the ‘asyncore’-based ‘smtpd’ module deprecated in Python
3.4.7 and 3.5.4. A recommended replacement is the *note asyncio:
a.-based aiosmtpd(3) PyPI module.
* *note shlex.split(): 538.: Passing ‘None’ for 's' argument now
raises an exception, rather than reading *note sys.stdin: 539. The
feature was deprecated in Python 3.9. (Contributed by Victor
Stinner in gh-94352(4).)
* The *note os: a1. module no longer accepts bytes-like paths, like
*note bytearray: 53a. and *note memoryview: 464. types: only the
exact *note bytes: 1c2. type is accepted for bytes strings.
(Contributed by Victor Stinner in gh-98393(5).)
* *note syslog.openlog(): 53b. and *note syslog.closelog(): 53c. now
fail if used in subinterpreters. *note syslog.syslog(): 53d. may
still be used in subinterpreters, but now only if *note
syslog.openlog(): 53b. has already been called in the main
interpreter. These new restrictions do not apply to the main
interpreter, so only a very small set of users might be affected.
This change helps with interpreter isolation. Furthermore, *note
syslog: dc. is a wrapper around process-global resources, which are
best managed from the main interpreter. (Contributed by Donghee Na
in gh-99127(6).)
* The undocumented locking behavior of *note cached_property(): 53e.
is removed, because it locked across all instances of the class,
leading to high lock contention. This means that a cached property
getter function could now run more than once for a single instance,
if two threads race. For most simple cached properties (e.g.
those that are idempotent and simply calculate a value based on
other attributes of the instance) this will be fine. If
synchronization is needed, implement locking within the cached
property getter function or around multi-threaded access points.
* *note sys._current_exceptions(): 4be. now returns a mapping from
thread-id to an exception instance, rather than to a ‘(typ, exc,
tb)’ tuple. (Contributed by Irit Katriel in gh-103176(7).)
* When extracting tar files using *note tarfile: de. or *note
shutil.unpack_archive(): 467, pass the 'filter' argument to limit
features that may be surprising or dangerous. See *note Extraction
filters: 468. for details.
* The output of the *note tokenize.tokenize(): 4d5. and *note
tokenize.generate_tokens(): 4d6. functions is now changed due to
the changes introduced in PEP 701(8). This means that ‘STRING’
tokens are not emitted any more for f-strings and the tokens
described in PEP 701(9) are now produced instead: ‘FSTRING_START’,
‘FSTRING_MIDDLE’ and ‘FSTRING_END’ are now emitted for f-string
“string” parts in addition to the appropriate tokens for the
tokenization in the expression components. For example for the
f-string ‘f"start {1+1} end"’ the old version of the tokenizer
emitted:
1,0-1,18: STRING 'f"start {1+1} end"'
while the new version emits:
1,0-1,2: FSTRING_START 'f"'
1,2-1,8: FSTRING_MIDDLE 'start '
1,8-1,9: OP '{'
1,9-1,10: NUMBER '1'
1,10-1,11: OP '+'
1,11-1,12: NUMBER '1'
1,12-1,13: OP '}'
1,13-1,17: FSTRING_MIDDLE ' end'
1,17-1,18: FSTRING_END '"'
Additionally, there may be some minor behavioral changes as a
consequence of the changes required to support PEP 701(10). Some
of these changes include:
* The ‘type’ attribute of the tokens emitted when tokenizing
some invalid Python characters such as ‘!’ has changed from
‘ERRORTOKEN’ to ‘OP’.
* Incomplete single-line strings now also raise *note
tokenize.TokenError: 53f. as incomplete multiline strings do.
* Some incomplete or invalid Python code now raises *note
tokenize.TokenError: 53f. instead of returning arbitrary
‘ERRORTOKEN’ tokens when tokenizing it.
* Mixing tabs and spaces as indentation in the same file is not
supported anymore and will raise a *note TabError: 540.
* The *note threading: ed. module now expects the ‘_thread’ module to
have an ‘_is_main_interpreter’ attribute. It is a function with no
arguments that returns ‘True’ if the current interpreter is the
main interpreter.
Any library or application that provides a custom ‘_thread’ module
should provide ‘_is_main_interpreter()’. (See gh-112826(11).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/91760
(2) https://github.com/python/cpython/issues/86388
(3) https://pypi.org/project/aiosmtpd/
(4) https://github.com/python/cpython/issues/94352
(5) https://github.com/python/cpython/issues/98393
(6) https://github.com/python/cpython/issues/99127
(7) https://github.com/python/cpython/issues/103176
(8) https://peps.python.org/pep-0701/
(9) https://peps.python.org/pep-0701/
(10) https://peps.python.org/pep-0701/
(11) https://github.com/python/cpython/issues/112826
File: python.info, Node: Build Changes<2>, Next: C API Changes<2>, Prev: Porting to Python 3 12, Up: What’s New In Python 3 12
1.2.13 Build Changes
--------------------
* Python no longer uses ‘setup.py’ to build shared C extension
modules. Build parameters like headers and libraries are detected
in ‘configure’ script. Extensions are built by ‘Makefile’. Most
extensions use ‘pkg-config’ and fall back to manual detection.
(Contributed by Christian Heimes in gh-93939(1).)
* ‘va_start()’ with two parameters, like ‘va_start(args, format),’ is
now required to build Python. ‘va_start()’ is no longer called
with a single parameter. (Contributed by Kumar Aditya in
gh-93207(2).)
* CPython now uses the ThinLTO option as the default link time
optimization policy if the Clang compiler accepts the flag.
(Contributed by Donghee Na in gh-89536(3).)
* Add ‘COMPILEALL_OPTS’ variable in ‘Makefile’ to override *note
compileall: 20. options (default: ‘-j0’) in ‘make install’. Also
merged the 3 ‘compileall’ commands into a single command to build
.pyc files for all optimization levels (0, 1, 2) at once.
(Contributed by Victor Stinner in gh-99289(4).)
* Add platform triplets for 64-bit LoongArch:
* loongarch64-linux-gnusf
* loongarch64-linux-gnuf32
* loongarch64-linux-gnu
(Contributed by Zhang Na in gh-90656(5).)
* ‘PYTHON_FOR_REGEN’ now require Python 3.10 or newer.
* Autoconf 2.71 and aclocal 1.16.4 is now required to regenerate
‘!configure’. (Contributed by Christian Heimes in gh-89886(6).)
* Windows builds and macOS installers from python.org now use OpenSSL
3.0.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/93939
(2) https://github.com/python/cpython/issues/93207
(3) https://github.com/python/cpython/issues/89536
(4) https://github.com/python/cpython/issues/99289
(5) https://github.com/python/cpython/issues/90656
(6) https://github.com/python/cpython/issues/89886
File: python.info, Node: C API Changes<2>, Prev: Build Changes<2>, Up: What’s New In Python 3 12
1.2.14 C API Changes
--------------------
* Menu:
* New Features: New Features<4>.
* Porting to Python 3.12: Porting to Python 3 12<2>.
* Deprecated: Deprecated<2>.
* Removed: Removed<2>.
File: python.info, Node: New Features<4>, Next: Porting to Python 3 12<2>, Up: C API Changes<2>
1.2.14.1 New Features
.....................
* PEP 697(1): Introduce the *note Unstable C API tier: 544, intended
for low-level tools like debuggers and JIT compilers. This API may
change in each minor release of CPython without deprecation
warnings. Its contents are marked by the ‘PyUnstable_’ prefix in
names.
Code object constructors:
- ‘PyUnstable_Code_New()’ (renamed from ‘PyCode_New’)
- ‘PyUnstable_Code_NewWithPosOnlyArgs()’ (renamed from
‘PyCode_NewWithPosOnlyArgs’)
Extra storage for code objects ( PEP 523(2)):
- ‘PyUnstable_Eval_RequestCodeExtraIndex()’ (renamed from
‘_PyEval_RequestCodeExtraIndex’)
- ‘PyUnstable_Code_GetExtra()’ (renamed from ‘_PyCode_GetExtra’)
- ‘PyUnstable_Code_SetExtra()’ (renamed from ‘_PyCode_SetExtra’)
The original names will continue to be available until the
respective API changes.
(Contributed by Petr Viktorin in gh-101101(3).)
* PEP 697(4): Add an API for extending types whose instance memory
layout is opaque:
- *note PyType_Spec.basicsize: 545. can be zero or negative to
specify inheriting or extending the base class size.
- *note PyObject_GetTypeData(): 546. and *note
PyType_GetTypeDataSize(): 547. added to allow access to
subclass-specific instance data.
- *note Py_TPFLAGS_ITEMS_AT_END: 548. and *note
PyObject_GetItemData(): 549. added to allow safely extending
certain variable-sized types, including *note PyType_Type:
54a.
- *note Py_RELATIVE_OFFSET: 54b. added to allow defining *note
members: 54c. in terms of a subclass-specific struct.
(Contributed by Petr Viktorin in gh-103509(5).)
* Add the new *note limited C API: 391. function *note
PyType_FromMetaclass(): 54d, which generalizes the existing *note
PyType_FromModuleAndSpec(): 54e. using an additional metaclass
argument. (Contributed by Wenzel Jakob in gh-93012(6).)
* API for creating objects that can be called using *note the
vectorcall protocol: 54f. was added to the *note Limited API: 550.:
* *note Py_TPFLAGS_HAVE_VECTORCALL: 551.
* *note PyVectorcall_NARGS(): 552.
* *note PyVectorcall_Call(): 553.
* *note vectorcallfunc: 554.
The *note Py_TPFLAGS_HAVE_VECTORCALL: 551. flag is now removed from
a class when the class’s *note __call__(): 555. method is
reassigned. This makes vectorcall safe to use with mutable types
(i.e. heap types without the immutable flag, *note
Py_TPFLAGS_IMMUTABLETYPE: 3be.). Mutable types that do not
override *note tp_call: 556. now inherit the
‘Py_TPFLAGS_HAVE_VECTORCALL’ flag. (Contributed by Petr Viktorin
in gh-93274(7).)
The *note Py_TPFLAGS_MANAGED_DICT: 366. and *note
Py_TPFLAGS_MANAGED_WEAKREF: 557. flags have been added. This
allows extensions classes to support object *note __dict__: 558.
and weakrefs with less bookkeeping, using less memory and with
faster access.
* API for performing calls using *note the vectorcall protocol: 54f.
was added to the *note Limited API: 550.:
* *note PyObject_Vectorcall(): 559.
* *note PyObject_VectorcallMethod(): 55a.
* *note PY_VECTORCALL_ARGUMENTS_OFFSET: 55b.
This means that both the incoming and outgoing ends of the vector
call protocol are now available in the *note Limited API: 550.
(Contributed by Wenzel Jakob in gh-98586(8).)
* Add two new public functions, *note PyEval_SetProfileAllThreads():
55c. and *note PyEval_SetTraceAllThreads(): 41e, that allow to set
tracing and profiling functions in all running threads in addition
to the calling one. (Contributed by Pablo Galindo in gh-93503(9).)
* Add new function *note PyFunction_SetVectorcall(): 55d. to the C
API which sets the vectorcall field of a given *note
PyFunctionObject: 55e. (Contributed by Andrew Frost in
gh-92257(10).)
* The C API now permits registering callbacks via *note
PyDict_AddWatcher(): 55f, *note PyDict_Watch(): 560. and related
APIs to be called whenever a dictionary is modified. This is
intended for use by optimizing interpreters, JIT compilers, or
debuggers. (Contributed by Carl Meyer in gh-91052(11).)
* Add *note PyType_AddWatcher(): 561. and *note PyType_Watch(): 562.
API to register callbacks to receive notification on changes to a
type. (Contributed by Carl Meyer in gh-91051(12).)
* Add *note PyCode_AddWatcher(): 563. and *note
PyCode_ClearWatcher(): 564. APIs to register callbacks to receive
notification on creation and destruction of code objects.
(Contributed by Itamar Oren in gh-91054(13).)
* Add *note PyFrame_GetVar(): 565. and *note PyFrame_GetVarString():
566. functions to get a frame variable by its name. (Contributed
by Victor Stinner in gh-91248(14).)
* Add *note PyErr_GetRaisedException(): 3f0. and *note
PyErr_SetRaisedException(): 3f3. for saving and restoring the
current exception. These functions return and accept a single
exception object, rather than the triple arguments of the
now-deprecated *note PyErr_Fetch(): 3ef. and *note PyErr_Restore():
3f2. This is less error prone and a bit more efficient.
(Contributed by Mark Shannon in gh-101578(15).)
* Add ‘_PyErr_ChainExceptions1’, which takes an exception instance,
to replace the legacy-API ‘_PyErr_ChainExceptions’, which is now
deprecated. (Contributed by Mark Shannon in gh-101578(16).)
* Add *note PyException_GetArgs(): 567. and *note
PyException_SetArgs(): 568. as convenience functions for retrieving
and modifying the *note args: 569. passed to the exception’s
constructor. (Contributed by Mark Shannon in gh-101578(17).)
* Add *note PyErr_DisplayException(): 3fe, which takes an exception
instance, to replace the legacy-api ‘PyErr_Display()’.
(Contributed by Irit Katriel in gh-102755(18)).
* PEP 683(19): Introduce 'Immortal Objects', which allows objects to
bypass reference counts, and related changes to the C-API:
-
‘_Py_IMMORTAL_REFCNT’: The reference count that defines an object
as immortal.
- ‘_Py_IsImmortal’ Checks if an object has the immortal
reference count.
-
‘PyObject_HEAD_INIT’ This will now initialize reference count to
‘_Py_IMMORTAL_REFCNT’ when used with ‘Py_BUILD_CORE’.
-
‘SSTATE_INTERNED_IMMORTAL’ An identifier for interned unicode objects
that are immortal.
-
‘SSTATE_INTERNED_IMMORTAL_STATIC’ An identifier for interned unicode
objects that are immortal and static
-
‘sys.getunicodeinternedsize’ This returns the total number of unicode
objects that have been interned. This is now needed for
‘refleak.py’ to correctly track reference counts and
allocated blocks
(Contributed by Eddie Elizondo in gh-84436(20).)
* PEP 684(21): Add the new *note Py_NewInterpreterFromConfig(): 457.
function and *note PyInterpreterConfig: 56a, which may be used to
create sub-interpreters with their own GILs. (See *note PEP 684; A
Per-Interpreter GIL: 437. for more info.) (Contributed by Eric
Snow in gh-104110(22).)
* In the limited C API version 3.12, *note Py_INCREF(): 56b. and
*note Py_DECREF(): 56c. functions are now implemented as opaque
function calls to hide implementation details. (Contributed by
Victor Stinner in gh-105387(23).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0697/
(2) https://peps.python.org/pep-0523/
(3) https://github.com/python/cpython/issues/101101
(4) https://peps.python.org/pep-0697/
(5) https://github.com/python/cpython/issues/103509
(6) https://github.com/python/cpython/issues/93012
(7) https://github.com/python/cpython/issues/93274
(8) https://github.com/python/cpython/issues/98586
(9) https://github.com/python/cpython/issues/93503
(10) https://github.com/python/cpython/issues/92257
(11) https://github.com/python/cpython/issues/91052
(12) https://github.com/python/cpython/issues/91051
(13) https://github.com/python/cpython/issues/91054
(14) https://github.com/python/cpython/issues/91248
(15) https://github.com/python/cpython/issues/101578
(16) https://github.com/python/cpython/issues/101578
(17) https://github.com/python/cpython/issues/101578
(18) https://github.com/python/cpython/issues/102755
(19) https://peps.python.org/pep-0683/
(20) https://github.com/python/cpython/issues/84436
(21) https://peps.python.org/pep-0684/
(22) https://github.com/python/cpython/issues/104110
(23) https://github.com/python/cpython/issues/105387
File: python.info, Node: Porting to Python 3 12<2>, Next: Deprecated<2>, Prev: New Features<4>, Up: C API Changes<2>
1.2.14.2 Porting to Python 3.12
...............................
* Legacy Unicode APIs based on ‘Py_UNICODE*’ representation has been
removed. Please migrate to APIs based on UTF-8 or ‘wchar_t*’.
* Argument parsing functions like *note PyArg_ParseTuple(): 56e.
doesn’t support ‘Py_UNICODE*’ based format (e.g. ‘u’, ‘Z’)
anymore. Please migrate to other formats for Unicode like ‘s’,
‘z’, ‘es’, and ‘U’.
* ‘tp_weaklist’ for all static builtin types is always ‘NULL’. This
is an internal-only field on ‘PyTypeObject’ but we’re pointing out
the change in case someone happens to be accessing the field
directly anyway. To avoid breakage, consider using the existing
public C-API instead, or, if necessary, the (internal-only)
‘_PyObject_GET_WEAKREFS_LISTPTR()’ macro.
* This internal-only *note PyTypeObject.tp_subclasses: 56f. may now
not be a valid object pointer. Its type was changed to void* to
reflect this. We mention this in case someone happens to be
accessing the internal-only field directly.
To get a list of subclasses, call the Python method *note
__subclasses__(): 570. (using *note PyObject_CallMethod(): 39b, for
example).
* Add support of more formatting options (left aligning, octals,
uppercase hexadecimals, ‘intmax_t’, ‘ptrdiff_t’, ‘wchar_t’ C
strings, variable width and precision) in *note
PyUnicode_FromFormat(): 385. and *note PyUnicode_FromFormatV():
571. (Contributed by Serhiy Storchaka in gh-98836(1).)
* An unrecognized format character in *note PyUnicode_FromFormat():
385. and *note PyUnicode_FromFormatV(): 571. now sets a *note
SystemError: 572. In previous versions it caused all the rest of
the format string to be copied as-is to the result string, and any
extra arguments discarded. (Contributed by Serhiy Storchaka in
gh-95781(2).)
* Fix wrong sign placement in *note PyUnicode_FromFormat(): 385. and
*note PyUnicode_FromFormatV(): 571. (Contributed by Philip Georgi
in gh-95504(3).)
* Extension classes wanting to add a *note __dict__: 558. or weak
reference slot should use *note Py_TPFLAGS_MANAGED_DICT: 366. and
*note Py_TPFLAGS_MANAGED_WEAKREF: 557. instead of ‘tp_dictoffset’
and ‘tp_weaklistoffset’, respectively. The use of ‘tp_dictoffset’
and ‘tp_weaklistoffset’ is still supported, but does not fully
support multiple inheritance (gh-95589(4)), and performance may be
worse. Classes declaring *note Py_TPFLAGS_MANAGED_DICT: 366. must
call ‘_PyObject_VisitManagedDict()’ and
‘_PyObject_ClearManagedDict()’ to traverse and clear their
instance’s dictionaries. To clear weakrefs, call *note
PyObject_ClearWeakRefs(): 573, as before.
* The *note PyUnicode_FSDecoder(): 574. function no longer accepts
bytes-like paths, like *note bytearray: 53a. and *note memoryview:
464. types: only the exact *note bytes: 1c2. type is accepted for
bytes strings. (Contributed by Victor Stinner in gh-98393(5).)
* The *note Py_CLEAR: 575, *note Py_SETREF: 576. and *note
Py_XSETREF: 577. macros now only evaluate their arguments once. If
an argument has side effects, these side effects are no longer
duplicated. (Contributed by Victor Stinner in gh-98724(6).)
* The interpreter’s error indicator is now always normalized. This
means that *note PyErr_SetObject(): 578, *note PyErr_SetString():
579. and the other functions that set the error indicator now
normalize the exception before storing it. (Contributed by Mark
Shannon in gh-101578(7).)
* ‘_Py_RefTotal’ is no longer authoritative and only kept around for
ABI compatibility. Note that it is an internal global and only
available on debug builds. If you happen to be using it then
you’ll need to start using ‘_Py_GetGlobalRefTotal()’.
* The following functions now select an appropriate metaclass for the
newly created type:
* *note PyType_FromSpec(): 57a.
* *note PyType_FromSpecWithBases(): 57b.
* *note PyType_FromModuleAndSpec(): 54e.
Creating classes whose metaclass overrides *note tp_new: 57c. is
deprecated, and in Python 3.14+ it will be disallowed. Note that
these functions ignore ‘tp_new’ of the metaclass, possibly allowing
incomplete initialization.
Note that *note PyType_FromMetaclass(): 54d. (added in Python 3.12)
already disallows creating classes whose metaclass overrides
‘tp_new’ (*note __new__(): 57d. in Python).
Since ‘tp_new’ overrides almost everything ‘PyType_From*’ functions
do, the two are incompatible with each other. The existing
behavior – ignoring the metaclass for several steps of type
creation – is unsafe in general, since (meta)classes assume that
‘tp_new’ was called. There is no simple general workaround. One
of the following may work for you:
- If you control the metaclass, avoid using ‘tp_new’ in it:
- If initialization can be skipped, it can be done in *note
tp_init: 57e. instead.
- If the metaclass doesn’t need to be instantiated from
Python, set its ‘tp_new’ to ‘NULL’ using the *note
Py_TPFLAGS_DISALLOW_INSTANTIATION: 57f. flag. This makes
it acceptable for ‘PyType_From*’ functions.
- Avoid ‘PyType_From*’ functions: if you don’t need C-specific
features (slots or setting the instance size), create types by
*note calling: 580. the metaclass.
- If you 'know' the ‘tp_new’ can be skipped safely, filter the
deprecation warning out using *note warnings.catch_warnings():
581. from Python.
* *note PyOS_InputHook: 582. and *note PyOS_ReadlineFunctionPointer:
583. are no longer called in *note subinterpreters: 584. This is
because clients generally rely on process-wide global state (since
these callbacks have no way of recovering extension module state).
This also avoids situations where extensions may find themselves
running in a subinterpreter that they don’t support (or haven’t yet
been loaded in). See gh-104668(8) for more info.
* *note PyLongObject: 585. has had its internals changed for better
performance. Although the internals of *note PyLongObject: 585.
are private, they are used by some extension modules. The internal
fields should no longer be accessed directly, instead the API
functions beginning ‘PyLong_...’ should be used instead. Two new
'unstable' API functions are provided for efficient access to the
value of *note PyLongObject: 585.s which fit into a single machine
word:
* *note PyUnstable_Long_IsCompact(): 586.
* *note PyUnstable_Long_CompactValue(): 587.
* Custom allocators, set via *note PyMem_SetAllocator(): 588, are now
required to be thread-safe, regardless of memory domain.
Allocators that don’t have their own state, including “hooks”, are
not affected. If your custom allocator is not already thread-safe
and you need guidance then please create a new GitHub issue and CC
‘@ericsnowcurrently’.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/98836
(2) https://github.com/python/cpython/issues/95781
(3) https://github.com/python/cpython/issues/95504
(4) https://github.com/python/cpython/issues/95589
(5) https://github.com/python/cpython/issues/98393
(6) https://github.com/python/cpython/issues/98724
(7) https://github.com/python/cpython/issues/101578
(8) https://github.com/python/cpython/issues/104668
File: python.info, Node: Deprecated<2>, Next: Removed<2>, Prev: Porting to Python 3 12<2>, Up: C API Changes<2>
1.2.14.3 Deprecated
...................
* In accordance with PEP 699(1), the ‘ma_version_tag’ field in *note
PyDictObject: 3bd. is deprecated for extension modules. Accessing
this field will generate a compiler warning at compile time. This
field will be removed in Python 3.14. (Contributed by Ramvikrams
and Kumar Aditya in gh-101193(2). PEP by Ken Jin.)
* Deprecate global configuration variable:
* *note Py_DebugFlag: 3c2.: use *note PyConfig.parser_debug:
3c3.
* *note Py_VerboseFlag: 3c4.: use *note PyConfig.verbose: 3c5.
* *note Py_QuietFlag: 3c6.: use *note PyConfig.quiet: 3c7.
* *note Py_InteractiveFlag: 3c8.: use *note
PyConfig.interactive: 3c9.
* *note Py_InspectFlag: 3ca.: use *note PyConfig.inspect: 3cb.
* *note Py_OptimizeFlag: 3cc.: use *note
PyConfig.optimization_level: 3cd.
* *note Py_NoSiteFlag: 3ce.: use *note PyConfig.site_import:
3cf.
* *note Py_BytesWarningFlag: 3d0.: use *note
PyConfig.bytes_warning: 3d1.
* *note Py_FrozenFlag: 3d2.: use *note
PyConfig.pathconfig_warnings: 3d3.
* *note Py_IgnoreEnvironmentFlag: 3d4.: use *note
PyConfig.use_environment: 3d5.
* *note Py_DontWriteBytecodeFlag: 3d6.: use *note
PyConfig.write_bytecode: 3d7.
* *note Py_NoUserSiteDirectory: 3d8.: use *note
PyConfig.user_site_directory: 3d9.
* *note Py_UnbufferedStdioFlag: 3da.: use *note
PyConfig.buffered_stdio: 3db.
* *note Py_HashRandomizationFlag: 3dc.: use *note
PyConfig.use_hash_seed: 3dd. and *note PyConfig.hash_seed:
3de.
* *note Py_IsolatedFlag: 3df.: use *note PyConfig.isolated: 3e0.
* *note Py_LegacyWindowsFSEncodingFlag: 3e1.: use *note
PyPreConfig.legacy_windows_fs_encoding: 3e2.
* *note Py_LegacyWindowsStdioFlag: 3e3.: use *note
PyConfig.legacy_windows_stdio: 3a0.
* ‘Py_FileSystemDefaultEncoding’: use *note
PyConfig.filesystem_encoding: 3e4.
* ‘Py_HasFileSystemDefaultEncoding’: use *note
PyConfig.filesystem_encoding: 3e4.
* ‘Py_FileSystemDefaultEncodeErrors’: use *note
PyConfig.filesystem_errors: 3e5.
* ‘Py_UTF8Mode’: use *note PyPreConfig.utf8_mode: 3e6. (see
*note Py_PreInitialize(): 3e7.)
The *note Py_InitializeFromConfig(): 3c1. API should be used with
*note PyConfig: 3a2. instead. (Contributed by Victor Stinner in
gh-77782(3).)
* Creating *note immutable types: 3be. with mutable bases is
deprecated and will be disabled in Python 3.14. (gh-95388(4))
* The ‘structmember.h’ header is deprecated, though it continues to
be available and there are no plans to remove it.
Its contents are now available just by including ‘Python.h’, with a
‘Py’ prefix added if it was missing:
- *note PyMemberDef: 54c, *note PyMember_GetOne(): 58a. and
*note PyMember_SetOne(): 58b.
- Type macros like *note Py_T_INT: 58c, *note Py_T_DOUBLE: 58d,
etc. (previously ‘T_INT’, ‘T_DOUBLE’, etc.)
- The flags *note Py_READONLY: 58e. (previously ‘READONLY’) and
*note Py_AUDIT_READ: 58f. (previously all uppercase)
Several items are not exposed from ‘Python.h’:
- *note T_OBJECT: 590. (use *note Py_T_OBJECT_EX: 591.)
- *note T_NONE: 592. (previously undocumented, and pretty
quirky)
- The macro ‘WRITE_RESTRICTED’ which does nothing.
- The macros ‘RESTRICTED’ and ‘READ_RESTRICTED’, equivalents of
*note Py_AUDIT_READ: 58f.
- In some configurations, ‘’ is not included from
‘Python.h’. It should be included manually when using
‘offsetof()’.
The deprecated header continues to provide its original contents
under the original names. Your old code can stay unchanged, unless
the extra include and non-namespaced macros bother you greatly.
(Contributed in gh-47146(5) by Petr Viktorin, based on earlier work
by Alexander Belopolsky and Matthias Braun.)
* *note PyErr_Fetch(): 3ef. and *note PyErr_Restore(): 3f2. are
deprecated. Use *note PyErr_GetRaisedException(): 3f0. and *note
PyErr_SetRaisedException(): 3f3. instead. (Contributed by Mark
Shannon in gh-101578(6).)
* ‘PyErr_Display()’ is deprecated. Use *note
PyErr_DisplayException(): 3fe. instead. (Contributed by Irit
Katriel in gh-102755(7)).
* ‘_PyErr_ChainExceptions’ is deprecated. Use
‘_PyErr_ChainExceptions1’ instead. (Contributed by Irit Katriel in
gh-102192(8).)
* Using *note PyType_FromSpec(): 57a, *note
PyType_FromSpecWithBases(): 57b. or *note
PyType_FromModuleAndSpec(): 54e. to create a class whose metaclass
overrides *note tp_new: 57c. is deprecated. Call the metaclass
instead.
* Menu:
* Pending Removal in Python 3.14: Pending Removal in Python 3 14<4>.
* Pending Removal in Python 3.15: Pending Removal in Python 3 15<4>.
* Pending removal in Python 3.16: Pending removal in Python 3 16<4>.
* Pending Removal in Future Versions: Pending Removal in Future Versions<4>.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0699/
(2) https://github.com/python/cpython/issues/101193
(3) https://github.com/python/cpython/issues/77782
(4) https://github.com/python/cpython/issues/95388
(5) https://github.com/python/cpython/issues/47146
(6) https://github.com/python/cpython/issues/101578
(7) https://github.com/python/cpython/issues/102755
(8) https://github.com/python/cpython/issues/102192
File: python.info, Node: Pending Removal in Python 3 14<4>, Next: Pending Removal in Python 3 15<4>, Up: Deprecated<2>
1.2.14.4 Pending Removal in Python 3.14
.......................................
* The ‘ma_version_tag’ field in *note PyDictObject: 3bd. for
extension modules ( PEP 699(1); gh-101193(2)).
* Creating *note immutable types: 3be. with mutable bases
(gh-95388(3)).
* Functions to configure Python’s initialization, deprecated in
Python 3.11:
* ‘PySys_SetArgvEx()’: Set *note PyConfig.argv: 3bf. instead.
* ‘PySys_SetArgv()’: Set *note PyConfig.argv: 3bf. instead.
* ‘Py_SetProgramName()’: Set *note PyConfig.program_name: 3c0.
instead.
* ‘Py_SetPythonHome()’: Set *note PyConfig.home: 3b7. instead.
The *note Py_InitializeFromConfig(): 3c1. API should be used with
*note PyConfig: 3a2. instead.
* Global configuration variables:
* *note Py_DebugFlag: 3c2.: Use *note PyConfig.parser_debug:
3c3. instead.
* *note Py_VerboseFlag: 3c4.: Use *note PyConfig.verbose: 3c5.
instead.
* *note Py_QuietFlag: 3c6.: Use *note PyConfig.quiet: 3c7.
instead.
* *note Py_InteractiveFlag: 3c8.: Use *note
PyConfig.interactive: 3c9. instead.
* *note Py_InspectFlag: 3ca.: Use *note PyConfig.inspect: 3cb.
instead.
* *note Py_OptimizeFlag: 3cc.: Use *note
PyConfig.optimization_level: 3cd. instead.
* *note Py_NoSiteFlag: 3ce.: Use *note PyConfig.site_import:
3cf. instead.
* *note Py_BytesWarningFlag: 3d0.: Use *note
PyConfig.bytes_warning: 3d1. instead.
* *note Py_FrozenFlag: 3d2.: Use *note
PyConfig.pathconfig_warnings: 3d3. instead.
* *note Py_IgnoreEnvironmentFlag: 3d4.: Use *note
PyConfig.use_environment: 3d5. instead.
* *note Py_DontWriteBytecodeFlag: 3d6.: Use *note
PyConfig.write_bytecode: 3d7. instead.
* *note Py_NoUserSiteDirectory: 3d8.: Use *note
PyConfig.user_site_directory: 3d9. instead.
* *note Py_UnbufferedStdioFlag: 3da.: Use *note
PyConfig.buffered_stdio: 3db. instead.
* *note Py_HashRandomizationFlag: 3dc.: Use *note
PyConfig.use_hash_seed: 3dd. and *note PyConfig.hash_seed:
3de. instead.
* *note Py_IsolatedFlag: 3df.: Use *note PyConfig.isolated: 3e0.
instead.
* *note Py_LegacyWindowsFSEncodingFlag: 3e1.: Use *note
PyPreConfig.legacy_windows_fs_encoding: 3e2. instead.
* *note Py_LegacyWindowsStdioFlag: 3e3.: Use *note
PyConfig.legacy_windows_stdio: 3a0. instead.
* ‘Py_FileSystemDefaultEncoding’: Use *note
PyConfig.filesystem_encoding: 3e4. instead.
* ‘Py_HasFileSystemDefaultEncoding’: Use *note
PyConfig.filesystem_encoding: 3e4. instead.
* ‘Py_FileSystemDefaultEncodeErrors’: Use *note
PyConfig.filesystem_errors: 3e5. instead.
* ‘Py_UTF8Mode’: Use *note PyPreConfig.utf8_mode: 3e6. instead.
(see *note Py_PreInitialize(): 3e7.)
The *note Py_InitializeFromConfig(): 3c1. API should be used with
*note PyConfig: 3a2. instead.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0699/
(2) https://github.com/python/cpython/issues/101193
(3) https://github.com/python/cpython/issues/95388
File: python.info, Node: Pending Removal in Python 3 15<4>, Next: Pending removal in Python 3 16<4>, Prev: Pending Removal in Python 3 14<4>, Up: Deprecated<2>
1.2.14.5 Pending Removal in Python 3.15
.......................................
* The *note PyImport_ImportModuleNoBlock(): 3b9.: Use *note
PyImport_ImportModule(): 3ba. instead.
* *note PyWeakref_GetObject(): 374. and *note PyWeakref_GET_OBJECT():
3bb.: Use *note PyWeakref_GetRef(): 373. instead.
* *note Py_UNICODE: 3e9. type and the ‘Py_UNICODE_WIDE’ macro: Use
‘wchar_t’ instead.
* Python initialization functions:
* *note PySys_ResetWarnOptions(): 3ab.: Clear *note
sys.warnoptions: 3ac. and ‘warnings.filters’ instead.
* *note Py_GetExecPrefix(): 3ad.: Get *note
sys.base_exec_prefix: 3ea. and *note sys.exec_prefix: 3ae.
instead.
* *note Py_GetPath(): 3af.: Get *note sys.path: 3b0. instead.
* *note Py_GetPrefix(): 3b1.: Get *note sys.base_prefix: 3eb.
and *note sys.prefix: 3b2. instead.
* *note Py_GetProgramFullPath(): 3b3.: Get *note sys.executable:
3b4. instead.
* *note Py_GetProgramName(): 3b5.: Get *note sys.executable:
3b4. instead.
* *note Py_GetPythonHome(): 3b6.: Get *note PyConfig.home: 3b7.
or the *note PYTHONHOME: 3b8. environment variable instead.
File: python.info, Node: Pending removal in Python 3 16<4>, Next: Pending Removal in Future Versions<4>, Prev: Pending Removal in Python 3 15<4>, Up: Deprecated<2>
1.2.14.6 Pending removal in Python 3.16
.......................................
* The bundled copy of ‘libmpdec’.
File: python.info, Node: Pending Removal in Future Versions<4>, Prev: Pending removal in Python 3 16<4>, Up: Deprecated<2>
1.2.14.7 Pending Removal in Future Versions
...........................................
The following APIs are deprecated and will be removed, although there is
currently no date scheduled for their removal.
* *note Py_TPFLAGS_HAVE_FINALIZE: 3ee.: Unneeded since Python 3.8.
* *note PyErr_Fetch(): 3ef.: Use *note PyErr_GetRaisedException():
3f0. instead.
* *note PyErr_NormalizeException(): 3f1.: Use *note
PyErr_GetRaisedException(): 3f0. instead.
* *note PyErr_Restore(): 3f2.: Use *note PyErr_SetRaisedException():
3f3. instead.
* *note PyModule_GetFilename(): 3f4.: Use *note
PyModule_GetFilenameObject(): 3f5. instead.
* *note PyOS_AfterFork(): 3f6.: Use *note PyOS_AfterFork_Child():
3f7. instead.
* *note PySlice_GetIndicesEx(): 3f8.: Use *note PySlice_Unpack():
3f9. and *note PySlice_AdjustIndices(): 3fa. instead.
* ‘PyUnicode_AsDecodedObject()’: Use *note PyCodec_Decode(): 3fb.
instead.
* ‘PyUnicode_AsDecodedUnicode()’: Use *note PyCodec_Decode(): 3fb.
instead.
* ‘PyUnicode_AsEncodedObject()’: Use *note PyCodec_Encode(): 3fc.
instead.
* ‘PyUnicode_AsEncodedUnicode()’: Use *note PyCodec_Encode(): 3fc.
instead.
* *note PyUnicode_READY(): 3fd.: Unneeded since Python 3.12
* ‘PyErr_Display()’: Use *note PyErr_DisplayException(): 3fe.
instead.
* ‘_PyErr_ChainExceptions()’: Use ‘_PyErr_ChainExceptions1()’
instead.
* ‘PyBytesObject.ob_shash’ member: call *note PyObject_Hash(): 3ff.
instead.
* ‘PyDictObject.ma_version_tag’ member.
* Thread Local Storage (TLS) API:
* *note PyThread_create_key(): 400.: Use *note
PyThread_tss_alloc(): 401. instead.
* *note PyThread_delete_key(): 402.: Use *note
PyThread_tss_free(): 403. instead.
* *note PyThread_set_key_value(): 404.: Use *note
PyThread_tss_set(): 405. instead.
* *note PyThread_get_key_value(): 406.: Use *note
PyThread_tss_get(): 407. instead.
* *note PyThread_delete_key_value(): 408.: Use *note
PyThread_tss_delete(): 409. instead.
* *note PyThread_ReInitTLS(): 40a.: Unneeded since Python 3.7.
File: python.info, Node: Removed<2>, Prev: Deprecated<2>, Up: C API Changes<2>
1.2.14.8 Removed
................
* Remove the ‘token.h’ header file. There was never any public
tokenizer C API. The ‘token.h’ header file was only designed to be
used by Python internals. (Contributed by Victor Stinner in
gh-92651(1).)
* Legacy Unicode APIs have been removed. See PEP 623(2) for detail.
* ‘PyUnicode_WCHAR_KIND’
* ‘PyUnicode_AS_UNICODE()’
* ‘PyUnicode_AsUnicode()’
* ‘PyUnicode_AsUnicodeAndSize()’
* ‘PyUnicode_AS_DATA()’
* ‘PyUnicode_FromUnicode()’
* ‘PyUnicode_GET_SIZE()’
* ‘PyUnicode_GetSize()’
* ‘PyUnicode_GET_DATA_SIZE()’
* Remove the ‘PyUnicode_InternImmortal()’ function macro.
(Contributed by Victor Stinner in gh-85858(3).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/92651
(2) https://peps.python.org/pep-0623/
(3) https://github.com/python/cpython/issues/85858
File: python.info, Node: What’s New In Python 3 11, Next: What’s New In Python 3 10, Prev: What’s New In Python 3 12, Up: What’s New in Python
1.3 What’s New In Python 3.11
=============================
Editor: Pablo Galindo Salgado
This article explains the new features in Python 3.11, compared to 3.10.
Python 3.11 was released on October 24, 2022. For full details, see the
*note changelog: 13c.
* Menu:
* Summary – Release highlights: Summary – Release highlights<2>.
* New Features: New Features<5>.
* New Features Related to Type Hints: New Features Related to Type Hints<2>.
* Other Language Changes: Other Language Changes<3>.
* Other CPython Implementation Changes::
* New Modules: New Modules<3>.
* Improved Modules: Improved Modules<3>.
* Optimizations: Optimizations<3>.
* Faster CPython::
* CPython bytecode changes: CPython bytecode changes<2>.
* Deprecated: Deprecated<3>.
* Pending Removal in Python 3.12: Pending Removal in Python 3 12.
* Removed: Removed<3>.
* Porting to Python 3.11: Porting to Python 3 11.
* Build Changes: Build Changes<3>.
* C API Changes: C API Changes<3>.
* Notable changes in 3.11.4: Notable changes in 3 11 4.
* Notable changes in 3.11.5: Notable changes in 3 11 5.
File: python.info, Node: Summary – Release highlights<2>, Next: New Features<5>, Up: What’s New In Python 3 11
1.3.1 Summary – Release highlights
----------------------------------
* Python 3.11 is between 10-60% faster than Python 3.10. On average,
we measured a 1.25x speedup on the standard benchmark suite. See
*note Faster CPython: 59c. for details.
New syntax features:
* *note PEP 654; Exception Groups and except*: 59d.
New built-in features:
* *note PEP 678; Exceptions can be enriched with notes: 59e.
New standard library modules:
* PEP 680(1): *note tomllib: fc. — Support for parsing TOML(2) in the
Standard Library
Interpreter improvements:
* *note PEP 657; Fine-grained error locations in tracebacks: 59f.
* New *note -P: 5a0. command line option and *note PYTHONSAFEPATH:
5a1. environment variable to *note disable automatically prepending
potentially unsafe paths: 5a2. to *note sys.path: 3b0.
New typing features:
* *note PEP 646; Variadic generics: 5a3.
* *note PEP 655; Marking individual TypedDict items as required or
not-required: 5a4.
* *note PEP 673; Self type: 5a5.
* *note PEP 675; Arbitrary literal string type: 5a6.
* *note PEP 681; Data class transforms: 5a7.
Important deprecations, removals and restrictions:
* PEP 594(3): *note Many legacy standard library modules have been
deprecated: 5a8. and will be removed in Python 3.13
* PEP 624(4): *note Py_UNICODE encoder APIs have been removed: 5a9.
* PEP 670(5): *note Macros converted to static inline functions: 5aa.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0680/
(2) https://toml.io/
(3) https://peps.python.org/pep-0594/
(4) https://peps.python.org/pep-0624/
(5) https://peps.python.org/pep-0670/
File: python.info, Node: New Features<5>, Next: New Features Related to Type Hints<2>, Prev: Summary – Release highlights<2>, Up: What’s New In Python 3 11
1.3.2 New Features
------------------
* Menu:
* PEP 657; Fine-grained error locations in tracebacks: PEP 657 Fine-grained error locations in tracebacks.
* PEP 654; Exception Groups and except*: PEP 654 Exception Groups and except*.
* PEP 678; Exceptions can be enriched with notes: PEP 678 Exceptions can be enriched with notes.
* Windows py.exe launcher improvements: Windows py exe launcher improvements.
File: python.info, Node: PEP 657 Fine-grained error locations in tracebacks, Next: PEP 654 Exception Groups and except*, Up: New Features<5>
1.3.2.1 PEP 657: Fine-grained error locations in tracebacks
...........................................................
When printing tracebacks, the interpreter will now point to the exact
expression that caused the error, instead of just the line. For
example:
Traceback (most recent call last):
File "distance.py", line 11, in
print(manhattan_distance(p1, p2))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "distance.py", line 6, in manhattan_distance
return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y)
^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'x'
Previous versions of the interpreter would point to just the line,
making it ambiguous which object was ‘None’. These enhanced errors can
also be helpful when dealing with deeply nested *note dict: 258. objects
and multiple function calls:
Traceback (most recent call last):
File "query.py", line 37, in
magic_arithmetic('foo')
File "query.py", line 18, in magic_arithmetic
return add_counts(x) / 25
^^^^^^^^^^^^^
File "query.py", line 24, in add_counts
return 25 + query_user(user1) + query_user(user2)
^^^^^^^^^^^^^^^^^
File "query.py", line 32, in query_user
return 1 + query_count(db, response['a']['b']['c']['user'], retry=True)
~~~~~~~~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable
As well as complex arithmetic expressions:
Traceback (most recent call last):
File "calculation.py", line 54, in
result = (x / y / z) * (a / b / c)
~~~~~~^~~
ZeroDivisionError: division by zero
Additionally, the information used by the enhanced traceback feature is
made available via a general API, that can be used to correlate *note
bytecode: 187. *note instructions: 5ae. with source code location. This
information can be retrieved using:
- The *note codeobject.co_positions(): 5af. method in Python.
- The *note PyCode_Addr2Location(): 5b0. function in the C API.
See PEP 657(1) for more details. (Contributed by Pablo Galindo, Batuhan
Taskaya and Ammar Askar in bpo-43950(2).)
Note: This feature requires storing column positions in *note Code
Objects: 5b1, which may result in a small increase in interpreter
memory usage and disk usage for compiled Python files. To avoid
storing the extra information and deactivate printing the extra
traceback information, use the *note -X no_debug_ranges: 176.
command line option or the *note PYTHONNODEBUGRANGES: 5b2.
environment variable.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0657/
(2) https://bugs.python.org/issue?@action=redirect&bpo=43950
File: python.info, Node: PEP 654 Exception Groups and except*, Next: PEP 678 Exceptions can be enriched with notes, Prev: PEP 657 Fine-grained error locations in tracebacks, Up: New Features<5>
1.3.2.2 PEP 654: Exception Groups and ‘except*’
...............................................
PEP 654(1) introduces language features that enable a program to raise
and handle multiple unrelated exceptions simultaneously. The builtin
types *note ExceptionGroup: 1b6. and *note BaseExceptionGroup: 261. make
it possible to group exceptions and raise them together, and the new
*note except*: 5b4. syntax generalizes *note except: 18b. to match
subgroups of exception groups.
See PEP 654(2) for more details.
(Contributed by Irit Katriel in bpo-45292(3). PEP written by Irit
Katriel, Yury Selivanov and Guido van Rossum.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0654/
(2) https://peps.python.org/pep-0654/
(3) https://bugs.python.org/issue?@action=redirect&bpo=45292
File: python.info, Node: PEP 678 Exceptions can be enriched with notes, Next: Windows py exe launcher improvements, Prev: PEP 654 Exception Groups and except*, Up: New Features<5>
1.3.2.3 PEP 678: Exceptions can be enriched with notes
......................................................
The *note add_note(): 5b6. method is added to *note BaseException: 5b7.
It can be used to enrich exceptions with context information that is not
available at the time when the exception is raised. The added notes
appear in the default traceback.
See PEP 678(1) for more details.
(Contributed by Irit Katriel in bpo-45607(2). PEP written by Zac
Hatfield-Dodds.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0678/
(2) https://bugs.python.org/issue?@action=redirect&bpo=45607
File: python.info, Node: Windows py exe launcher improvements, Prev: PEP 678 Exceptions can be enriched with notes, Up: New Features<5>
1.3.2.4 Windows ‘py.exe’ launcher improvements
..............................................
The copy of the *note Python Launcher for Windows: 5ba. included with
Python 3.11 has been significantly updated. It now supports company/tag
syntax as defined in PEP 514(1) using the ‘-V:/’ argument
instead of the limited ‘-.’. This allows launching
distributions other than ‘PythonCore’, the one hosted on python.org(2).
When using ‘-V:’ selectors, either company or tag can be omitted, but
all installs will be searched. For example, ‘-V:OtherPython/’ will
select the “best” tag registered for ‘OtherPython’, while ‘-V:3.11’ or
‘-V:/3.11’ will select the “best” distribution with tag ‘3.11’.
When using the legacy ‘-’, ‘-.’,
‘--’ or ‘-.-’ arguments, all
existing behaviour should be preserved from past versions, and only
releases from ‘PythonCore’ will be selected. However, the ‘-64’ suffix
now implies “not 32-bit” (not necessarily x86-64), as there are multiple
supported 64-bit platforms. 32-bit runtimes are detected by checking
the runtime’s tag for a ‘-32’ suffix. All releases of Python since 3.5
have included this in their 32-bit builds.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0514/
(2) https://www.python.org
File: python.info, Node: New Features Related to Type Hints<2>, Next: Other Language Changes<3>, Prev: New Features<5>, Up: What’s New In Python 3 11
1.3.3 New Features Related to Type Hints
----------------------------------------
This section covers major changes affecting PEP 484(1) type hints and
the *note typing: 104. module.
* Menu:
* PEP 646; Variadic generics: PEP 646 Variadic generics.
* PEP 655; Marking individual TypedDict items as required or not-required: PEP 655 Marking individual TypedDict items as required or not-required.
* PEP 673; Self type: PEP 673 Self type.
* PEP 675; Arbitrary literal string type: PEP 675 Arbitrary literal string type.
* PEP 681; Data class transforms: PEP 681 Data class transforms.
* PEP 563 may not be the future::
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
File: python.info, Node: PEP 646 Variadic generics, Next: PEP 655 Marking individual TypedDict items as required or not-required, Up: New Features Related to Type Hints<2>
1.3.3.1 PEP 646: Variadic generics
..................................
PEP 484(1) previously introduced *note TypeVar: 15d, enabling creation
of generics parameterised with a single type. PEP 646(2) adds *note
TypeVarTuple: 15f, enabling parameterisation with an 'arbitrary' number
of types. In other words, a *note TypeVarTuple: 15f. is a 'variadic'
type variable, enabling 'variadic' generics.
This enables a wide variety of use cases. In particular, it allows the
type of array-like structures in numerical computing libraries such as
NumPy and TensorFlow to be parameterised with the array 'shape'. Static
type checkers will now be able to catch shape-related bugs in code that
uses these libraries.
See PEP 646(3) for more details.
(Contributed by Matthew Rahtz in bpo-43224(4), with contributions by
Serhiy Storchaka and Jelle Zijlstra. PEP written by Mark Mendoza,
Matthew Rahtz, Pradeep Kumar Srinivasan, and Vincent Siles.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0646/
(3) https://peps.python.org/pep-0646/
(4) https://bugs.python.org/issue?@action=redirect&bpo=43224
File: python.info, Node: PEP 655 Marking individual TypedDict items as required or not-required, Next: PEP 673 Self type, Prev: PEP 646 Variadic generics, Up: New Features Related to Type Hints<2>
1.3.3.2 PEP 655: Marking individual ‘TypedDict’ items as required or not-required
.................................................................................
*note Required: 5c0. and *note NotRequired: 5c1. provide a
straightforward way to mark whether individual items in a *note
TypedDict: 162. must be present. Previously, this was only possible
using inheritance.
All fields are still required by default, unless the 'total' parameter
is set to ‘False’, in which case all fields are still not-required by
default. For example, the following specifies a ‘TypedDict’ with one
required and one not-required key:
class Movie(TypedDict):
title: str
year: NotRequired[int]
m1: Movie = {"title": "Black Panther", "year": 2018} # OK
m2: Movie = {"title": "Star Wars"} # OK (year is not required)
m3: Movie = {"year": 2022} # ERROR (missing required field title)
The following definition is equivalent:
class Movie(TypedDict, total=False):
title: Required[str]
year: int
See PEP 655(1) for more details.
(Contributed by David Foster and Jelle Zijlstra in bpo-47087(2). PEP
written by David Foster.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0655/
(2) https://bugs.python.org/issue?@action=redirect&bpo=47087
File: python.info, Node: PEP 673 Self type, Next: PEP 675 Arbitrary literal string type, Prev: PEP 655 Marking individual TypedDict items as required or not-required, Up: New Features Related to Type Hints<2>
1.3.3.3 PEP 673: ‘Self’ type
............................
The new *note Self: 5c3. annotation provides a simple and intuitive way
to annotate methods that return an instance of their class. This
behaves the same as the *note TypeVar: 15d.-based approach specified in
PEP 484(1), but is more concise and easier to follow.
Common use cases include alternative constructors provided as *note
classmethod: 166.s, and *note __enter__(): 5c4. methods that return
‘self’:
class MyLock:
def __enter__(self) -> Self:
self.lock()
return self
...
class MyInt:
@classmethod
def fromhex(cls, s: str) -> Self:
return cls(int(s, 16))
...
*note Self: 5c3. can also be used to annotate method parameters or
attributes of the same type as their enclosing class.
See PEP 673(2) for more details.
(Contributed by James Hilton-Balfe in bpo-46534(3). PEP written by
Pradeep Kumar Srinivasan and James Hilton-Balfe.)
---------- Footnotes ----------
(1)
https://peps.python.org/pep-0484/#annotating-instance-and-class-methods
(2) https://peps.python.org/pep-0673/
(3) https://bugs.python.org/issue?@action=redirect&bpo=46534
File: python.info, Node: PEP 675 Arbitrary literal string type, Next: PEP 681 Data class transforms, Prev: PEP 673 Self type, Up: New Features Related to Type Hints<2>
1.3.3.4 PEP 675: Arbitrary literal string type
..............................................
The new *note LiteralString: 5c6. annotation may be used to indicate
that a function parameter can be of any literal string type. This
allows a function to accept arbitrary literal string types, as well as
strings created from other literal strings. Type checkers can then
enforce that sensitive functions, such as those that execute SQL
statements or shell commands, are called only with static arguments,
providing protection against injection attacks.
For example, a SQL query function could be annotated as follows:
def run_query(sql: LiteralString) -> ...
...
def caller(
arbitrary_string: str,
query_string: LiteralString,
table_name: LiteralString,
) -> None:
run_query("SELECT * FROM students") # ok
run_query(query_string) # ok
run_query("SELECT * FROM " + table_name) # ok
run_query(arbitrary_string) # type checker error
run_query( # type checker error
f"SELECT * FROM students WHERE name = {arbitrary_string}"
)
See PEP 675(1) for more details.
(Contributed by Jelle Zijlstra in bpo-47088(2). PEP written by Pradeep
Kumar Srinivasan and Graham Bleaney.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0675/
(2) https://bugs.python.org/issue?@action=redirect&bpo=47088
File: python.info, Node: PEP 681 Data class transforms, Next: PEP 563 may not be the future, Prev: PEP 675 Arbitrary literal string type, Up: New Features Related to Type Hints<2>
1.3.3.5 PEP 681: Data class transforms
......................................
*note dataclass_transform: 4d0. may be used to decorate a class,
metaclass, or a function that is itself a decorator. The presence of
‘@dataclass_transform()’ tells a static type checker that the decorated
object performs runtime “magic” that transforms a class, giving it *note
dataclass: 1cb.-like behaviors.
For example:
# The create_model decorator is defined by a library.
@typing.dataclass_transform()
def create_model(cls: Type[T]) -> Type[T]:
cls.__init__ = ...
cls.__eq__ = ...
cls.__ne__ = ...
return cls
# The create_model decorator can now be used to create new model classes:
@create_model
class CustomerModel:
id: int
name: str
c = CustomerModel(id=327, name="Eric Idle")
See PEP 681(1) for more details.
(Contributed by Jelle Zijlstra in gh-91860(2). PEP written by Erik De
Bonte and Eric Traut.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0681/
(2) https://github.com/python/cpython/issues/91860
File: python.info, Node: PEP 563 may not be the future, Prev: PEP 681 Data class transforms, Up: New Features Related to Type Hints<2>
1.3.3.6 PEP 563 may not be the future
.....................................
PEP 563(1) Postponed Evaluation of Annotations (the ‘from __future__
import annotations’ *note future statement: 189.) that was originally
planned for release in Python 3.10 has been put on hold indefinitely.
See this message from the Steering Council(2) for more information.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0563/
(2)
https://mail.python.org/archives/list/python-dev@python.org/message/VIZEBX5EYMSYIJNDBF6DMUMZOCWHARSO/
File: python.info, Node: Other Language Changes<3>, Next: Other CPython Implementation Changes, Prev: New Features Related to Type Hints<2>, Up: What’s New In Python 3 11
1.3.4 Other Language Changes
----------------------------
* Starred unpacking expressions can now be used in *note for: 2ec.
statements. (See bpo-46725(1) for more details.)
* Asynchronous *note comprehensions: 5cc. are now allowed inside
comprehensions in *note asynchronous functions: 5cd. Outer
comprehensions implicitly become asynchronous in this case.
(Contributed by Serhiy Storchaka in bpo-33346(2).)
* A *note TypeError: 534. is now raised instead of an *note
AttributeError: 348. in *note with: 5ce. statements and *note
contextlib.ExitStack.enter_context(): 5cf. for objects that do not
support the *note context manager: 5d0. protocol, and in *note
async with: 5d1. statements and *note
contextlib.AsyncExitStack.enter_async_context(): 5d2. for objects
not supporting the *note asynchronous context manager: 5d3.
protocol. (Contributed by Serhiy Storchaka in bpo-12022(3) and
bpo-44471(4).)
* Added *note object.__getstate__(): 5d4, which provides the default
implementation of the ‘__getstate__()’ method. *note copy: 25.ing
and *note pickle: a6.ing instances of subclasses of builtin types
*note bytearray: 53a, *note set: 5d5, *note frozenset: 5d6, *note
collections.OrderedDict: 5d7, *note collections.deque: 5d8, *note
weakref.WeakSet: 5d9, and *note datetime.tzinfo: 5da. now copies
and pickles instance attributes implemented as *note slots: 5db.
This change has an unintended side effect: It trips up a small
minority of existing Python projects not expecting *note
object.__getstate__(): 5d4. to exist. See the later comments on
gh-70766(5) for discussions of what workarounds such code may need.
(Contributed by Serhiy Storchaka in bpo-26579(6).)
* Added a *note -P: 5a0. command line option and a *note
PYTHONSAFEPATH: 5a1. environment variable, which disable the
automatic prepending to *note sys.path: 3b0. of the script’s
directory when running a script, or the current directory when
using *note -c: 5dc. and *note -m: 5dd. This ensures only stdlib
and installed modules are picked up by *note import: 5de, and
avoids unintentionally or maliciously shadowing modules with those
in a local (and typically user-writable) directory. (Contributed
by Victor Stinner in gh-57684(7).)
* A ‘"z"’ option was added to the *note Format Specification
Mini-Language: 1ea. that coerces negative to positive zero after
rounding to the format precision. See PEP 682(8) for more details.
(Contributed by John Belmonte in gh-90153(9).)
* Bytes are no longer accepted on *note sys.path: 3b0. Support broke
sometime between Python 3.2 and 3.6, with no one noticing until
after Python 3.10.0 was released. In addition, bringing back
support would be problematic due to interactions between *note -b:
5df. and *note sys.path_importer_cache: 5e0. when there is a
mixture of *note str: 447. and *note bytes: 1c2. keys.
(Contributed by Thomas Grainger in gh-91181(10).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=46725
(2) https://bugs.python.org/issue?@action=redirect&bpo=33346
(3) https://bugs.python.org/issue?@action=redirect&bpo=12022
(4) https://bugs.python.org/issue?@action=redirect&bpo=44471
(5) https://github.com/python/cpython/issues/70766
(6) https://bugs.python.org/issue?@action=redirect&bpo=26579
(7) https://github.com/python/cpython/issues/57684
(8) https://peps.python.org/pep-0682/
(9) https://github.com/python/cpython/issues/90153
(10) https://github.com/python/cpython/issues/91181
File: python.info, Node: Other CPython Implementation Changes, Next: New Modules<3>, Prev: Other Language Changes<3>, Up: What’s New In Python 3 11
1.3.5 Other CPython Implementation Changes
------------------------------------------
* The special methods *note __complex__(): 5e3. for *note complex:
2f2. and *note __bytes__(): 5e4. for *note bytes: 1c2. are
implemented to support the *note typing.SupportsComplex: 5e5. and
*note typing.SupportsBytes: 5e6. protocols. (Contributed by Mark
Dickinson and Donghee Na in bpo-24234(1).)
* ‘siphash13’ is added as a new internal hashing algorithm. It has
similar security properties as ‘siphash24’, but it is slightly
faster for long inputs. *note str: 447, *note bytes: 1c2, and some
other types now use it as the default algorithm for *note hash():
5e7. PEP 552(2) *note hash-based .pyc files: 5e8. now use
‘siphash13’ too. (Contributed by Inada Naoki in bpo-29410(3).)
* When an active exception is re-raised by a *note raise: 5e9.
statement with no parameters, the traceback attached to this
exception is now always ‘sys.exc_info()[1].__traceback__’. This
means that changes made to the traceback in the current *note
except: 18b. clause are reflected in the re-raised exception.
(Contributed by Irit Katriel in bpo-45711(4).)
* The interpreter state’s representation of handled exceptions (aka
‘exc_info’ or ‘_PyErr_StackItem’) now only has the ‘exc_value’
field; ‘exc_type’ and ‘exc_traceback’ have been removed, as they
can be derived from ‘exc_value’. (Contributed by Irit Katriel in
bpo-45711(5).)
* A new *note command line option: 5ea, ‘AppendPath’, has been added
for the Windows installer. It behaves similarly to ‘PrependPath’,
but appends the install and scripts directories instead of
prepending them. (Contributed by Bastian Neuburger in
bpo-44934(6).)
* The *note PyConfig.module_search_paths_set: 5eb. field must now be
set to ‘1’ for initialization to use *note
PyConfig.module_search_paths: 39e. to initialize *note sys.path:
3b0. Otherwise, initialization will recalculate the path and
replace any values added to ‘module_search_paths’.
* The output of the *note -help: 5ec. option now fits in 50 lines/80
columns. Information about *note Python environment variables:
5ed. and *note -X: 176. options is now available using the
respective *note -help-env: 5ee. and *note -help-xoptions: 5ef.
flags, and with the new *note -help-all: 5f0. (Contributed by Éric
Araujo in bpo-46142(7).)
* Converting between *note int: 259. and *note str: 447. in bases
other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such
as base 10 (decimal) now raises a *note ValueError: 204. if the
number of digits in string form is above a limit to avoid potential
denial of service attacks due to the algorithmic complexity. This
is a mitigation for CVE 2020-10735(8). This limit can be
configured or disabled by environment variable, command line flag,
or *note sys: d9. APIs. See the *note integer string conversion
length limitation: 5f1. documentation. The default limit is 4300
digits in string form.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24234
(2) https://peps.python.org/pep-0552/
(3) https://bugs.python.org/issue?@action=redirect&bpo=29410
(4) https://bugs.python.org/issue?@action=redirect&bpo=45711
(5) https://bugs.python.org/issue?@action=redirect&bpo=45711
(6) https://bugs.python.org/issue?@action=redirect&bpo=44934
(7) https://bugs.python.org/issue?@action=redirect&bpo=46142
(8) https://www.cve.org/CVERecord?id=CVE-2020-10735
File: python.info, Node: New Modules<3>, Next: Improved Modules<3>, Prev: Other CPython Implementation Changes, Up: What’s New In Python 3 11
1.3.6 New Modules
-----------------
* *note tomllib: fc.: For parsing TOML(1). See PEP 680(2) for more
details. (Contributed by Taneli Hukkinen in bpo-40059(3).)
* *note wsgiref.types: 11c.: WSGI(4)-specific types for static type
checking. (Contributed by Sebastian Rittau in bpo-42012(5).)
---------- Footnotes ----------
(1) https://toml.io/
(2) https://peps.python.org/pep-0680/
(3) https://bugs.python.org/issue?@action=redirect&bpo=40059
(4) https://peps.python.org/pep-3333/
(5) https://bugs.python.org/issue?@action=redirect&bpo=42012
File: python.info, Node: Improved Modules<3>, Next: Optimizations<3>, Prev: New Modules<3>, Up: What’s New In Python 3 11
1.3.7 Improved Modules
----------------------
* Menu:
* asyncio: asyncio<3>.
* contextlib::
* dataclasses::
* datetime::
* enum: enum<3>.
* fcntl::
* fractions: fractions<3>.
* functools::
* gzip: gzip<2>.
* hashlib: hashlib<2>.
* IDLE and idlelib::
* inspect: inspect<2>.
* locale: locale<3>.
* logging::
* math: math<3>.
* operator::
* os: os<3>.
* pathlib: pathlib<4>.
* re: re<3>.
* shutil: shutil<3>.
* socket::
* sqlite3: sqlite3<4>.
* string::
* sys: sys<4>.
* sysconfig::
* tempfile: tempfile<3>.
* threading: threading<2>.
* time: time<2>.
* tkinter: tkinter<3>.
* traceback: traceback<2>.
* typing: typing<4>.
* unicodedata: unicodedata<3>.
* unittest: unittest<4>.
* venv: venv<2>.
* warnings: warnings<2>.
* zipfile::
File: python.info, Node: asyncio<3>, Next: contextlib, Up: Improved Modules<3>
1.3.7.1 asyncio
...............
* Added the *note TaskGroup: 1b5. class, an *note asynchronous
context manager: 5f8. holding a group of tasks that will wait for
all of them upon exit. For new code this is recommended over using
*note create_task(): 1bf. and *note gather(): 5f9. directly.
(Contributed by Yury Selivanov and others in gh-90908(1).)
* Added *note timeout(): 5fa, an asynchronous context manager for
setting a timeout on asynchronous operations. For new code this is
recommended over using *note wait_for(): 5fb. directly.
(Contributed by Andrew Svetlov in gh-90927(2).)
* Added the *note Runner: 5fc. class, which exposes the machinery
used by *note run(): 478. (Contributed by Andrew Svetlov in
gh-91218(3).)
* Added the *note Barrier: 5fd. class to the synchronization
primitives in the asyncio library, and the related *note
BrokenBarrierError: 5fe. exception. (Contributed by Yves Duprat
and Andrew Svetlov in gh-87518(4).)
* Added keyword argument 'all_errors' to *note
asyncio.loop.create_connection(): 5ff. so that multiple connection
errors can be raised as an *note ExceptionGroup: 1b6.
* Added the *note asyncio.StreamWriter.start_tls(): 600. method for
upgrading existing stream-based connections to TLS. (Contributed by
Ian Good in bpo-34975(5).)
* Added raw datagram socket functions to the event loop: *note
sock_sendto(): 601, *note sock_recvfrom(): 602. and *note
sock_recvfrom_into(): 603. These have implementations in *note
SelectorEventLoop: 604. and *note ProactorEventLoop: 605.
(Contributed by Alex Grönholm in bpo-46805(6).)
* Added *note cancelling(): 1ba. and *note uncancel(): 1bb. methods
to *note Task: 1be. These are primarily intended for internal use,
notably by *note TaskGroup: 1b5.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/90908
(2) https://github.com/python/cpython/issues/90927
(3) https://github.com/python/cpython/issues/91218
(4) https://github.com/python/cpython/issues/87518
(5) https://bugs.python.org/issue?@action=redirect&bpo=34975
(6) https://bugs.python.org/issue?@action=redirect&bpo=46805
File: python.info, Node: contextlib, Next: dataclasses, Prev: asyncio<3>, Up: Improved Modules<3>
1.3.7.2 contextlib
..................
* Added non parallel-safe *note chdir(): 608. context manager to
change the current working directory and then restore it on exit.
Simple wrapper around *note chdir(): 609. (Contributed by Filipe
Laíns in bpo-25625(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25625
File: python.info, Node: dataclasses, Next: datetime, Prev: contextlib, Up: Improved Modules<3>
1.3.7.3 dataclasses
...................
* Change field default mutability check, allowing only defaults which
are *note hashable: 60c. instead of any object which is not an
instance of *note dict: 258, *note list: 60d. or *note set: 5d5.
(Contributed by Eric V. Smith in bpo-44674(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=44674
File: python.info, Node: datetime, Next: enum<3>, Prev: dataclasses, Up: Improved Modules<3>
1.3.7.4 datetime
................
* Add *note datetime.UTC: 4ea, a convenience alias for *note
datetime.timezone.utc: 610. (Contributed by Kabir Kwatra in
gh-91973(1).)
* *note datetime.date.fromisoformat(): 611, *note
datetime.time.fromisoformat(): 612. and *note
datetime.datetime.fromisoformat(): 613. can now be used to parse
most ISO 8601 formats (barring only those that support fractional
hours and minutes). (Contributed by Paul Ganssle in gh-80010(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/91973
(2) https://github.com/python/cpython/issues/80010
File: python.info, Node: enum<3>, Next: fcntl, Prev: datetime, Up: Improved Modules<3>
1.3.7.5 enum
............
* Renamed ‘EnumMeta’ to *note EnumType: 1e7. (‘EnumMeta’ kept as an
alias).
* Added *note StrEnum: 616, with members that can be used as (and
must be) strings.
* Added *note ReprEnum: 617, which only modifies the *note
__repr__(): 618. of members while returning their literal values
(rather than names) for *note __str__(): 619. and *note
__format__(): 61a. (used by *note str(): 447, *note format(): 61b.
and *note f-string: 431.s).
* Changed *note Enum.__format__(): 61c. (the default for *note
format(): 61b, *note str.format(): 61d. and *note f-string: 431.s)
to always produce the same result as *note Enum.__str__(): 61e.:
for enums inheriting from *note ReprEnum: 617. it will be the
member’s value; for all other enums it will be the enum and member
name (e.g. ‘Color.RED’).
* Added a new 'boundary' class parameter to *note Flag: 61f. enums
and the *note FlagBoundary: 620. enum with its options, to control
how to handle out-of-range flag values.
* Added the *note verify(): 621. enum decorator and the *note
EnumCheck: 622. enum with its options, to check enum classes
against several specific constraints.
* Added the *note member(): 623. and *note nonmember(): 624.
decorators, to ensure the decorated object is/is not converted to
an enum member.
* Added the *note property(): 625. decorator, which works like *note
property(): 194. except for enums. Use this instead of *note
types.DynamicClassAttribute(): 626.
* Added the *note global_enum(): 627. enum decorator, which adjusts
*note __repr__(): 618. and *note __str__(): 619. to show values as
members of their module rather than the enum class. For example,
‘'re.ASCII'’ for the *note ASCII: 628. member of *note
re.RegexFlag: 629. rather than ‘'RegexFlag.ASCII'’.
* Enhanced *note Flag: 61f. to support *note len(): 62a, iteration
and *note in: 2ee./*note not in: 62b. on its members. For example,
the following now works: ‘len(AFlag(3)) == 2 and list(AFlag(3)) ==
(AFlag.ONE, AFlag.TWO)’
* Changed *note Enum: 62c. and *note Flag: 61f. so that members are
now defined before *note __init_subclass__(): 62d. is called; *note
dir(): 62e. now includes methods, etc., from mixed-in data types.
* Changed *note Flag: 61f. to only consider primary values (power of
two) canonical while composite values (‘3’, ‘6’, ‘10’, etc.) are
considered aliases; inverted flags are coerced to their positive
equivalent.
File: python.info, Node: fcntl, Next: fractions<3>, Prev: enum<3>, Up: Improved Modules<3>
1.3.7.6 fcntl
.............
* On FreeBSD, the ‘F_DUP2FD’ and ‘F_DUP2FD_CLOEXEC’ flags
respectively are supported, the former equals to ‘dup2’ usage while
the latter set the ‘FD_CLOEXEC’ flag in addition.
File: python.info, Node: fractions<3>, Next: functools, Prev: fcntl, Up: Improved Modules<3>
1.3.7.7 fractions
.................
* Support PEP 515(1)-style initialization of *note Fraction: 1e9.
from string. (Contributed by Sergey B Kirpichev in bpo-44258(2).)
* *note Fraction: 1e9. now implements an ‘__int__’ method, so that an
‘isinstance(some_fraction, typing.SupportsInt)’ check passes.
(Contributed by Mark Dickinson in bpo-44547(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0515/
(2) https://bugs.python.org/issue?@action=redirect&bpo=44258
(3) https://bugs.python.org/issue?@action=redirect&bpo=44547
File: python.info, Node: functools, Next: gzip<2>, Prev: fractions<3>, Up: Improved Modules<3>
1.3.7.8 functools
.................
* *note functools.singledispatch(): 635. now supports *note
types.UnionType: 636. and *note typing.Union: 637. as annotations
to the dispatch argument.:
>>> from functools import singledispatch
>>> @singledispatch
... def fun(arg, verbose=False):
... if verbose:
... print("Let me just say,", end=" ")
... print(arg)
...
>>> @fun.register
... def _(arg: int | float, verbose=False):
... if verbose:
... print("Strength in numbers, eh?", end=" ")
... print(arg)
...
>>> from typing import Union
>>> @fun.register
... def _(arg: Union[list, set], verbose=False):
... if verbose:
... print("Enumerate this:")
... for i, elem in enumerate(arg):
... print(i, elem)
...
(Contributed by Yurii Karabas in bpo-46014(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=46014
File: python.info, Node: gzip<2>, Next: hashlib<2>, Prev: functools, Up: Improved Modules<3>
1.3.7.9 gzip
............
* The *note gzip.compress(): 63a. function is now faster when used
with the 'mtime=0' argument as it delegates the compression
entirely to a single *note zlib.compress(): 63b. operation. There
is one side effect of this change: The gzip file header contains an
“OS” byte in its header. That was traditionally always set to a
value of 255 representing “unknown” by the *note gzip: 67. module.
Now, when using *note compress(): 63a. with 'mtime=0', it may be
set to a different value by the underlying zlib C library Python
was linked against. (See gh-112346(1) for details on the side
effect.)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/112346
File: python.info, Node: hashlib<2>, Next: IDLE and idlelib, Prev: gzip<2>, Up: Improved Modules<3>
1.3.7.10 hashlib
................
* *note hashlib.blake2b(): 63e. and *note hashlib.blake2s(): 63f. now
prefer libb2(1) over Python’s vendored copy. (Contributed by
Christian Heimes in bpo-47095(2).)
* The internal ‘_sha3’ module with SHA3 and SHAKE algorithms now uses
'tiny_sha3' instead of the 'Keccak Code Package' to reduce code and
binary size. The *note hashlib: 68. module prefers optimized SHA3
and SHAKE implementations from OpenSSL. The change affects only
installations without OpenSSL support. (Contributed by Christian
Heimes in bpo-47098(3).)
* Add *note hashlib.file_digest(): 640, a helper function for
efficient hashing of files or file-like objects. (Contributed by
Christian Heimes in gh-89313(4).)
---------- Footnotes ----------
(1) https://www.blake2.net/
(2) https://bugs.python.org/issue?@action=redirect&bpo=47095
(3) https://bugs.python.org/issue?@action=redirect&bpo=47098
(4) https://github.com/python/cpython/issues/89313
File: python.info, Node: IDLE and idlelib, Next: inspect<2>, Prev: hashlib<2>, Up: Improved Modules<3>
1.3.7.11 IDLE and idlelib
.........................
* Apply syntax highlighting to ‘.pyi’ files. (Contributed by Alex
Waygood and Terry Jan Reedy in bpo-45447(1).)
* Include prompts when saving Shell with inputs and outputs.
(Contributed by Terry Jan Reedy in gh-95191(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45447
(2) https://github.com/python/cpython/issues/95191
File: python.info, Node: inspect<2>, Next: locale<3>, Prev: IDLE and idlelib, Up: Improved Modules<3>
1.3.7.12 inspect
................
* Add *note getmembers_static(): 645. to return all members without
triggering dynamic lookup via the descriptor protocol.
(Contributed by Weipeng Hong in bpo-30533(1).)
* Add *note ismethodwrapper(): 646. for checking if the type of an
object is a *note MethodWrapperType: 647. (Contributed by Hakan
Çelik in bpo-29418(2).)
* Change the frame-related functions in the *note inspect: 7e. module
to return new *note FrameInfo: 648. and *note Traceback: 649. class
instances (backwards compatible with the previous *note named
tuple: 64a.-like interfaces) that includes the extended PEP 657(3)
position information (end line number, column and end column). The
affected functions are:
* *note inspect.getframeinfo(): 64b.
* *note inspect.getouterframes(): 64c.
* *note inspect.getinnerframes(): 64d,
* *note inspect.stack(): 64e.
* *note inspect.trace(): 64f.
(Contributed by Pablo Galindo in gh-88116(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30533
(2) https://bugs.python.org/issue?@action=redirect&bpo=29418
(3) https://peps.python.org/pep-0657/
(4) https://github.com/python/cpython/issues/88116
File: python.info, Node: locale<3>, Next: logging, Prev: inspect<2>, Up: Improved Modules<3>
1.3.7.13 locale
...............
* Add *note locale.getencoding(): 2e0. to get the current locale
encoding. It is similar to ‘locale.getpreferredencoding(False)’
but ignores the *note Python UTF-8 Mode: 652.
File: python.info, Node: logging, Next: math<3>, Prev: locale<3>, Up: Improved Modules<3>
1.3.7.14 logging
................
* Added *note getLevelNamesMapping(): 655. to return a mapping from
logging level names (e.g. ‘'CRITICAL'’) to the values of their
corresponding *note Logging Levels: 656. (e.g. ‘50’, by default).
(Contributed by Andrei Kulakovin in gh-88024(1).)
* Added a *note createSocket(): 657. method to *note SysLogHandler:
658, to match *note SocketHandler.createSocket(): 659. It is
called automatically during handler initialization and when
emitting an event, if there is no active socket. (Contributed by
Kirill Pinchuk in gh-88457(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/88024
(2) https://github.com/python/cpython/issues/88457
File: python.info, Node: math<3>, Next: operator, Prev: logging, Up: Improved Modules<3>
1.3.7.15 math
.............
* Add *note math.exp2(): 65c.: return 2 raised to the power of x.
(Contributed by Gideon Mitchell in bpo-45917(1).)
* Add *note math.cbrt(): 65d.: return the cube root of x.
(Contributed by Ajith Ramachandran in bpo-44357(2).)
* The behaviour of two *note math.pow(): 65e. corner cases was
changed, for consistency with the IEEE 754 specification. The
operations ‘math.pow(0.0, -math.inf)’ and ‘math.pow(-0.0,
-math.inf)’ now return ‘inf’. Previously they raised *note
ValueError: 204. (Contributed by Mark Dickinson in bpo-44339(3).)
* The *note math.nan: 65f. value is now always available.
(Contributed by Victor Stinner in bpo-46917(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45917
(2) https://bugs.python.org/issue?@action=redirect&bpo=44357
(3) https://bugs.python.org/issue?@action=redirect&bpo=44339
(4) https://bugs.python.org/issue?@action=redirect&bpo=46917
File: python.info, Node: operator, Next: os<3>, Prev: math<3>, Up: Improved Modules<3>
1.3.7.16 operator
.................
* A new function ‘operator.call’ has been added, such that
‘operator.call(obj, *args, **kwargs) == obj(*args, **kwargs)’.
(Contributed by Antony Lee in bpo-44019(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=44019
File: python.info, Node: os<3>, Next: pathlib<4>, Prev: operator, Up: Improved Modules<3>
1.3.7.17 os
...........
* On Windows, *note os.urandom(): 51e. now uses ‘BCryptGenRandom()’,
instead of ‘CryptGenRandom()’ which is deprecated. (Contributed by
Donghee Na in bpo-44611(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=44611
File: python.info, Node: pathlib<4>, Next: re<3>, Prev: os<3>, Up: Improved Modules<3>
1.3.7.18 pathlib
................
* *note glob(): 22f. and *note rglob(): 230. return only directories
if 'pattern' ends with a pathname components separator: *note sep:
666. or *note altsep: 667. (Contributed by Eisuke Kawasima in
bpo-22276(1) and bpo-33392(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22276
(2) https://bugs.python.org/issue?@action=redirect&bpo=33392
File: python.info, Node: re<3>, Next: shutil<3>, Prev: pathlib<4>, Up: Improved Modules<3>
1.3.7.19 re
...........
* Atomic grouping (‘(?>...)’) and possessive quantifiers (‘*+’, ‘++’,
‘?+’, ‘{m,n}+’) are now supported in regular expressions.
(Contributed by Jeffrey C. Jacobs and Serhiy Storchaka in
bpo-433030(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=433030
File: python.info, Node: shutil<3>, Next: socket, Prev: re<3>, Up: Improved Modules<3>
1.3.7.20 shutil
...............
* Add optional parameter 'dir_fd' in *note shutil.rmtree(): 2fb.
(Contributed by Serhiy Storchaka in bpo-46245(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=46245
File: python.info, Node: socket, Next: sqlite3<4>, Prev: shutil<3>, Up: Improved Modules<3>
1.3.7.21 socket
...............
* Add CAN Socket support for NetBSD. (Contributed by Thomas Klausner
in bpo-30512(1).)
* *note create_connection(): 66e. has an option to raise, in case of
failure to connect, an *note ExceptionGroup: 1b6. containing all
errors instead of only raising the last error. (Contributed by
Irit Katriel in bpo-29980(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30512
(2) https://bugs.python.org/issue?@action=redirect&bpo=29980
File: python.info, Node: sqlite3<4>, Next: string, Prev: socket, Up: Improved Modules<3>
1.3.7.22 sqlite3
................
* You can now disable the authorizer by passing *note None: 671. to
*note set_authorizer(): 2ad. (Contributed by Erlend E. Aasland in
bpo-44491(1).)
* Collation name *note create_collation(): 672. can now contain any
Unicode character. Collation names with invalid characters now
raise *note UnicodeEncodeError: 673. instead of *note
sqlite3.ProgrammingError: 4ee. (Contributed by Erlend E. Aasland
in bpo-44688(2).)
* *note sqlite3: cf. exceptions now include the SQLite extended error
code as *note sqlite_errorcode: 674. and the SQLite error name as
*note sqlite_errorname: 675. (Contributed by Aviv Palivoda, Daniel
Shahaf, and Erlend E. Aasland in bpo-16379(3) and bpo-24139(4).)
* Add *note setlimit(): 676. and *note getlimit(): 677. to *note
sqlite3.Connection: 247. for setting and getting SQLite limits by
connection basis. (Contributed by Erlend E. Aasland in
bpo-45243(5).)
* *note sqlite3: cf. now sets *note sqlite3.threadsafety: 678. based
on the default threading mode the underlying SQLite library has
been compiled with. (Contributed by Erlend E. Aasland in
bpo-45613(6).)
* *note sqlite3: cf. C callbacks now use unraisable exceptions if
callback tracebacks are enabled. Users can now register an *note
unraisable hook handler: 1f9. to improve their debug experience.
(Contributed by Erlend E. Aasland in bpo-45828(7).)
* Fetch across rollback no longer raises *note InterfaceError: 679.
Instead we leave it to the SQLite library to handle these cases.
(Contributed by Erlend E. Aasland in bpo-44092(8).)
* Add *note serialize(): 67a. and *note deserialize(): 67b. to *note
sqlite3.Connection: 247. for serializing and deserializing
databases. (Contributed by Erlend E. Aasland in bpo-41930(9).)
* Add *note create_window_function(): 67c. to *note
sqlite3.Connection: 247. for creating aggregate window functions.
(Contributed by Erlend E. Aasland in bpo-34916(10).)
* Add *note blobopen(): 67d. to *note sqlite3.Connection: 247. *note
sqlite3.Blob: 67e. allows incremental I/O operations on blobs.
(Contributed by Aviv Palivoda and Erlend E. Aasland in
bpo-24905(11).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=44491
(2) https://bugs.python.org/issue?@action=redirect&bpo=44688
(3) https://bugs.python.org/issue?@action=redirect&bpo=16379
(4) https://bugs.python.org/issue?@action=redirect&bpo=24139
(5) https://bugs.python.org/issue?@action=redirect&bpo=45243
(6) https://bugs.python.org/issue?@action=redirect&bpo=45613
(7) https://bugs.python.org/issue?@action=redirect&bpo=45828
(8) https://bugs.python.org/issue?@action=redirect&bpo=44092
(9) https://bugs.python.org/issue?@action=redirect&bpo=41930
(10) https://bugs.python.org/issue?@action=redirect&bpo=34916
(11) https://bugs.python.org/issue?@action=redirect&bpo=24905
File: python.info, Node: string, Next: sys<4>, Prev: sqlite3<4>, Up: Improved Modules<3>
1.3.7.23 string
...............
* Add *note get_identifiers(): 681. and *note is_valid(): 682. to
*note string.Template: 683, which respectively return all valid
placeholders, and whether any invalid placeholders are present.
(Contributed by Ben Kehoe in gh-90465(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/90465
File: python.info, Node: sys<4>, Next: sysconfig, Prev: string, Up: Improved Modules<3>
1.3.7.24 sys
............
* *note sys.exc_info(): 686. now derives the ‘type’ and ‘traceback’
fields from the ‘value’ (the exception instance), so when an
exception is modified while it is being handled, the changes are
reflected in the results of subsequent calls to ‘exc_info()’.
(Contributed by Irit Katriel in bpo-45711(1).)
* Add *note sys.exception(): 687. which returns the active exception
instance (equivalent to ‘sys.exc_info()[1]’). (Contributed by Irit
Katriel in bpo-46328(2).)
* Add the *note sys.flags.safe_path: 688. flag. (Contributed by
Victor Stinner in gh-57684(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45711
(2) https://bugs.python.org/issue?@action=redirect&bpo=46328
(3) https://github.com/python/cpython/issues/57684
File: python.info, Node: sysconfig, Next: tempfile<3>, Prev: sys<4>, Up: Improved Modules<3>
1.3.7.25 sysconfig
..................
* Three new *note installation schemes: 68b. ('posix_venv', 'nt_venv'
and 'venv') were added and are used when Python creates new virtual
environments or when it is running from a virtual environment. The
first two schemes ('posix_venv' and 'nt_venv') are OS-specific for
non-Windows and Windows, the 'venv' is essentially an alias to one
of them according to the OS Python runs on. This is useful for
downstream distributors who modify *note
sysconfig.get_preferred_scheme(): 68c. Third party code that
creates new virtual environments should use the new 'venv'
installation scheme to determine the paths, as does *note venv:
111. (Contributed by Miro Hrončok in bpo-45413(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45413
File: python.info, Node: tempfile<3>, Next: threading<2>, Prev: sysconfig, Up: Improved Modules<3>
1.3.7.26 tempfile
.................
* *note SpooledTemporaryFile: 68f. objects now fully implement the
methods of *note io.BufferedIOBase: 690. or *note io.TextIOBase:
691. (depending on file mode). This lets them work correctly with
APIs that expect file-like objects, such as compression modules.
(Contributed by Carey Metcalfe in gh-70363(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/70363
File: python.info, Node: threading<2>, Next: time<2>, Prev: tempfile<3>, Up: Improved Modules<3>
1.3.7.27 threading
..................
* On Unix, if the ‘sem_clockwait()’ function is available in the C
library (glibc 2.30 and newer), the *note threading.Lock.acquire():
694. method now uses the monotonic clock (*note
time.CLOCK_MONOTONIC: 695.) for the timeout, rather than using the
system clock (*note time.CLOCK_REALTIME: 696.), to not be affected
by system clock changes. (Contributed by Victor Stinner in
bpo-41710(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41710
File: python.info, Node: time<2>, Next: tkinter<3>, Prev: threading<2>, Up: Improved Modules<3>
1.3.7.28 time
.............
* On Unix, *note time.sleep(): 699. now uses the ‘clock_nanosleep()’
or ‘nanosleep()’ function, if available, which has a resolution of
1 nanosecond (10^-9 seconds), rather than using ‘select()’ which
has a resolution of 1 microsecond (10^-6 seconds). (Contributed by
Benjamin Szőke and Victor Stinner in bpo-21302(1).)
* On Windows 8.1 and newer, *note time.sleep(): 699. now uses a
waitable timer based on high-resolution timers(2) which has a
resolution of 100 nanoseconds (10^-7 seconds). Previously, it had
a resolution of 1 millisecond (10^-3 seconds). (Contributed by
Benjamin Szőke, Donghee Na, Eryk Sun and Victor Stinner in
bpo-21302(3) and bpo-45429(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21302
(2)
https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/high-resolution-timers
(3) https://bugs.python.org/issue?@action=redirect&bpo=21302
(4) https://bugs.python.org/issue?@action=redirect&bpo=45429
File: python.info, Node: tkinter<3>, Next: traceback<2>, Prev: time<2>, Up: Improved Modules<3>
1.3.7.29 tkinter
................
* Added method ‘info_patchlevel()’ which returns the exact version of
the Tcl library as a named tuple similar to *note sys.version_info:
69c. (Contributed by Serhiy Storchaka in gh-91827(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/91827
File: python.info, Node: traceback<2>, Next: typing<4>, Prev: tkinter<3>, Up: Improved Modules<3>
1.3.7.30 traceback
..................
* Add *note traceback.StackSummary.format_frame_summary(): 69f. to
allow users to override which frames appear in the traceback, and
how they are formatted. (Contributed by Ammar Askar in
bpo-44569(1).)
* Add *note traceback.TracebackException.print(): 6a0, which prints
the formatted *note TracebackException: 25e. instance to a file.
(Contributed by Irit Katriel in bpo-33809(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=44569
(2) https://bugs.python.org/issue?@action=redirect&bpo=33809
File: python.info, Node: typing<4>, Next: unicodedata<3>, Prev: traceback<2>, Up: Improved Modules<3>
1.3.7.31 typing
...............
For major changes, see *note New Features Related to Type Hints: 5bb.
* Add *note typing.assert_never(): 6a3. and *note typing.Never: 6a4.
*note typing.assert_never(): 6a3. is useful for asking a type
checker to confirm that a line of code is not reachable. At
runtime, it raises an *note AssertionError: 6a5. (Contributed by
Jelle Zijlstra in gh-90633(1).)
* Add *note typing.reveal_type(): 6a6. This is useful for asking a
type checker what type it has inferred for a given expression. At
runtime it prints the type of the received value. (Contributed by
Jelle Zijlstra in gh-90572(2).)
* Add *note typing.assert_type(): 6a7. This is useful for asking a
type checker to confirm that the type it has inferred for a given
expression matches the given type. At runtime it simply returns
the received value. (Contributed by Jelle Zijlstra in
gh-90638(3).)
* *note typing.TypedDict: 162. types can now be generic.
(Contributed by Samodya Abeysiriwardane in gh-89026(4).)
* *note NamedTuple: 2b2. types can now be generic. (Contributed by
Serhiy Storchaka in bpo-43923(5).)
* Allow subclassing of *note typing.Any: 6a8. This is useful for
avoiding type checker errors related to highly dynamic class, such
as mocks. (Contributed by Shantanu Jain in gh-91154(6).)
* The *note typing.final(): 6a9. decorator now sets the ‘__final__’
attributed on the decorated object. (Contributed by Jelle Zijlstra
in gh-90500(7).)
* The *note typing.get_overloads(): 6aa. function can be used for
introspecting the overloads of a function. *note
typing.clear_overloads(): 6ab. can be used to clear all registered
overloads of a function. (Contributed by Jelle Zijlstra in
gh-89263(8).)
* The *note __init__(): 6ac. method of *note Protocol: 266.
subclasses is now preserved. (Contributed by Adrian Garcia
Badarasco in gh-88970(9).)
* The representation of empty tuple types (‘Tuple[()]’) is
simplified. This affects introspection, e.g.
‘get_args(Tuple[()])’ now evaluates to ‘()’ instead of ‘((),)’.
(Contributed by Serhiy Storchaka in gh-91137(10).)
* Loosen runtime requirements for type annotations by removing the
callable check in the private ‘typing._type_check’ function.
(Contributed by Gregory Beauregard in gh-90802(11).)
* *note typing.get_type_hints(): 6ad. now supports evaluating strings
as forward references in *note PEP 585 generic aliases: 6ae.
(Contributed by Niklas Rosenstein in gh-85542(12).)
* *note typing.get_type_hints(): 6ad. no longer adds *note Optional:
6af. to parameters with ‘None’ as a default. (Contributed by
Nikita Sobolev in gh-90353(13).)
* *note typing.get_type_hints(): 6ad. now supports evaluating bare
stringified *note ClassVar: 268. annotations. (Contributed by
Gregory Beauregard in gh-90711(14).)
* *note typing.no_type_check(): 6b0. no longer modifies external
classes and functions. It also now correctly marks classmethods as
not to be type checked. (Contributed by Nikita Sobolev in
gh-90729(15).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/90633
(2) https://github.com/python/cpython/issues/90572
(3) https://github.com/python/cpython/issues/90638
(4) https://github.com/python/cpython/issues/89026
(5) https://bugs.python.org/issue?@action=redirect&bpo=43923
(6) https://github.com/python/cpython/issues/91154
(7) https://github.com/python/cpython/issues/90500
(8) https://github.com/python/cpython/issues/89263
(9) https://github.com/python/cpython/issues/88970
(10) https://github.com/python/cpython/issues/91137
(11) https://github.com/python/cpython/issues/90802
(12) https://github.com/python/cpython/issues/85542
(13) https://github.com/python/cpython/issues/90353
(14) https://github.com/python/cpython/issues/90711
(15) https://github.com/python/cpython/issues/90729
File: python.info, Node: unicodedata<3>, Next: unittest<4>, Prev: typing<4>, Up: Improved Modules<3>
1.3.7.32 unicodedata
....................
* The Unicode database has been updated to version 14.0.0.
(Contributed by Benjamin Peterson in bpo-45190(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45190
File: python.info, Node: unittest<4>, Next: venv<2>, Prev: unicodedata<3>, Up: Improved Modules<3>
1.3.7.33 unittest
.................
* Added methods *note enterContext(): 6b5. and *note
enterClassContext(): 6b6. of class *note TestCase: 449, method
*note enterAsyncContext(): 6b7. of class *note
IsolatedAsyncioTestCase: 306. and function *note
unittest.enterModuleContext(): 6b8. (Contributed by Serhiy
Storchaka in bpo-45046(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45046
File: python.info, Node: venv<2>, Next: warnings<2>, Prev: unittest<4>, Up: Improved Modules<3>
1.3.7.34 venv
.............
* When new Python virtual environments are created, the 'venv' *note
sysconfig installation scheme: 68b. is used to determine the paths
inside the environment. When Python runs in a virtual environment,
the same installation scheme is the default. That means that
downstream distributors can change the default sysconfig install
scheme without changing behavior of virtual environments. Third
party code that also creates new virtual environments should do the
same. (Contributed by Miro Hrončok in bpo-45413(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45413
File: python.info, Node: warnings<2>, Next: zipfile, Prev: venv<2>, Up: Improved Modules<3>
1.3.7.35 warnings
.................
* *note warnings.catch_warnings(): 581. now accepts arguments for
*note warnings.simplefilter(): 6bd, providing a more concise way to
locally ignore warnings or convert them to errors. (Contributed by
Zac Hatfield-Dodds in bpo-47074(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=47074
File: python.info, Node: zipfile, Prev: warnings<2>, Up: Improved Modules<3>
1.3.7.36 zipfile
................
* Added support for specifying member name encoding for reading
metadata in a *note ZipFile: 6c0.’s directory and file headers.
(Contributed by Stephen J. Turnbull and Serhiy Storchaka in
bpo-28080(1).)
* Added *note ZipFile.mkdir(): 6c1. for creating new directories
inside ZIP archives. (Contributed by Sam Ezeh in gh-49083(2).)
* Added *note stem: 6c2, *note suffix: 6c3. and *note suffixes: 6c4.
to *note zipfile.Path: 6c5. (Contributed by Miguel Brito in
gh-88261(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28080
(2) https://github.com/python/cpython/issues/49083
(3) https://github.com/python/cpython/issues/88261
File: python.info, Node: Optimizations<3>, Next: Faster CPython, Prev: Improved Modules<3>, Up: What’s New In Python 3 11
1.3.8 Optimizations
-------------------
This section covers specific optimizations independent of the *note
Faster CPython: 59c. project, which is covered in its own section.
* The compiler now optimizes simple *note printf-style % formatting:
6c8. on string literals containing only the format codes ‘%s’, ‘%r’
and ‘%a’ and makes it as fast as a corresponding *note f-string:
431. expression. (Contributed by Serhiy Storchaka in
bpo-28307(1).)
* Integer division (‘//’) is better tuned for optimization by
compilers. It is now around 20% faster on x86-64 when dividing an
*note int: 259. by a value smaller than ‘2**30’. (Contributed by
Gregory P. Smith and Tim Peters in gh-90564(2).)
* *note sum(): 466. is now nearly 30% faster for integers smaller
than ‘2**30’. (Contributed by Stefan Behnel in gh-68264(3).)
* Resizing lists is streamlined for the common case, speeding up
‘list.append()’ by ≈15% and simple *note list comprehension: 6c9.s
by up to 20-30% (Contributed by Dennis Sweeney in gh-91165(4).)
* Dictionaries don’t store hash values when all keys are Unicode
objects, decreasing *note dict: 258. size. For example,
‘sys.getsizeof(dict.fromkeys("abcdefg"))’ is reduced from 352 bytes
to 272 bytes (23% smaller) on 64-bit platforms. (Contributed by
Inada Naoki in bpo-46845(5).)
* Using *note asyncio.DatagramProtocol: 6ca. is now orders of
magnitude faster when transferring large files over UDP, with
speeds over 100 times higher for a ≈60 MiB file. (Contributed by
msoxzw in gh-91487(6).)
* *note math: 8e. functions *note comb(): 6cb. and *note perm(): 6cc.
are now ≈10 times faster for large arguments (with a larger speedup
for larger 'k'). (Contributed by Serhiy Storchaka in
bpo-37295(7).)
* The *note statistics: d2. functions *note mean(): 6cd, *note
variance(): 6ce. and *note stdev(): 6cf. now consume iterators in
one pass rather than converting them to a *note list: 60d. first.
This is twice as fast and can save substantial memory.
(Contributed by Raymond Hettinger in gh-90415(8).)
* *note unicodedata.normalize(): 6d0. now normalizes pure-ASCII
strings in constant time. (Contributed by Donghee Na in
bpo-44987(9).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28307
(2) https://github.com/python/cpython/issues/90564
(3) https://github.com/python/cpython/issues/68264
(4) https://github.com/python/cpython/issues/91165
(5) https://bugs.python.org/issue?@action=redirect&bpo=46845
(6) https://github.com/python/cpython/issues/91487
(7) https://bugs.python.org/issue?@action=redirect&bpo=37295
(8) https://github.com/python/cpython/issues/90415
(9) https://bugs.python.org/issue?@action=redirect&bpo=44987
File: python.info, Node: Faster CPython, Next: CPython bytecode changes<2>, Prev: Optimizations<3>, Up: What’s New In Python 3 11
1.3.9 Faster CPython
--------------------
CPython 3.11 is an average of 25% faster(1) than CPython 3.10 as
measured with the pyperformance(2) benchmark suite, when compiled with
GCC on Ubuntu Linux. Depending on your workload, the overall speedup
could be 10-60%.
This project focuses on two major areas in Python: *note Faster Startup:
6d2. and *note Faster Runtime: 6d3. Optimizations not covered by this
project are listed separately under *note Optimizations: 6c7.
* Menu:
* Faster Startup::
* Faster Runtime::
* Misc::
* FAQ::
* About::
---------- Footnotes ----------
(1) https://github.com/faster-cpython/ideas#published-results
(2) https://github.com/python/pyperformance
File: python.info, Node: Faster Startup, Next: Faster Runtime, Up: Faster CPython
1.3.9.1 Faster Startup
......................
* Menu:
* Frozen imports / Static code objects::
File: python.info, Node: Frozen imports / Static code objects, Up: Faster Startup
1.3.9.2 Frozen imports / Static code objects
............................................
Python caches *note bytecode: 187. in the *note __pycache__: 6d7.
directory to speed up module loading.
Previously in 3.10, Python module execution looked like this:
Read __pycache__ -> Unmarshal -> Heap allocated code object -> Evaluate
In Python 3.11, the core modules essential for Python startup are
“frozen”. This means that their *note Code Objects: 5b1. (and bytecode)
are statically allocated by the interpreter. This reduces the steps in
module execution process to:
Statically allocated code object -> Evaluate
Interpreter startup is now 10-15% faster in Python 3.11. This has a big
impact for short-running programs using Python.
(Contributed by Eric Snow, Guido van Rossum and Kumar Aditya in many
issues.)
File: python.info, Node: Faster Runtime, Next: Misc, Prev: Faster Startup, Up: Faster CPython
1.3.9.3 Faster Runtime
......................
* Menu:
* Cheaper, lazy Python frames: Cheaper lazy Python frames.
* Inlined Python function calls::
* PEP 659; Specializing Adaptive Interpreter: PEP 659 Specializing Adaptive Interpreter.
File: python.info, Node: Cheaper lazy Python frames, Next: Inlined Python function calls, Up: Faster Runtime
1.3.9.4 Cheaper, lazy Python frames
...................................
Python frames, holding execution information, are created whenever
Python calls a Python function. The following are new frame
optimizations:
- Streamlined the frame creation process.
- Avoided memory allocation by generously re-using frame space on the
C stack.
- Streamlined the internal frame struct to contain only essential
information. Frames previously held extra debugging and memory
management information.
Old-style *note frame objects: 6db. are now created only when requested
by debuggers or by Python introspection functions such as *note
sys._getframe(): 6dc. and *note inspect.currentframe(): 6dd. For most
user code, no frame objects are created at all. As a result, nearly all
Python functions calls have sped up significantly. We measured a 3-7%
speedup in pyperformance.
(Contributed by Mark Shannon in bpo-44590(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=44590
File: python.info, Node: Inlined Python function calls, Next: PEP 659 Specializing Adaptive Interpreter, Prev: Cheaper lazy Python frames, Up: Faster Runtime
1.3.9.5 Inlined Python function calls
.....................................
During a Python function call, Python will call an evaluating C function
to interpret that function’s code. This effectively limits pure Python
recursion to what’s safe for the C stack.
In 3.11, when CPython detects Python code calling another Python
function, it sets up a new frame, and “jumps” to the new code inside the
new frame. This avoids calling the C interpreting function altogether.
Most Python function calls now consume no C stack space, speeding them
up. In simple recursive functions like fibonacci or factorial, we
observed a 1.7x speedup. This also means recursive functions can
recurse significantly deeper (if the user increases the recursion limit
with *note sys.setrecursionlimit(): 4bf.). We measured a 1-3%
improvement in pyperformance.
(Contributed by Pablo Galindo and Mark Shannon in bpo-45256(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45256
File: python.info, Node: PEP 659 Specializing Adaptive Interpreter, Prev: Inlined Python function calls, Up: Faster Runtime
1.3.9.6 PEP 659: Specializing Adaptive Interpreter
..................................................
PEP 659(1) is one of the key parts of the Faster CPython project. The
general idea is that while Python is a dynamic language, most code has
regions where objects and types rarely change. This concept is known as
'type stability'.
At runtime, Python will try to look for common patterns and type
stability in the executing code. Python will then replace the current
operation with a more specialized one. This specialized operation uses
fast paths available only to those use cases/types, which generally
outperform their generic counterparts. This also brings in another
concept called 'inline caching', where Python caches the results of
expensive operations directly in the *note bytecode: 187.
The specializer will also combine certain common instruction pairs into
one superinstruction, reducing the overhead during execution.
Python will only specialize when it sees code that is “hot” (executed
multiple times). This prevents Python from wasting time on run-once
code. Python can also de-specialize when code is too dynamic or when
the use changes. Specialization is attempted periodically, and
specialization attempts are not too expensive, allowing specialization
to adapt to new circumstances.
(PEP written by Mark Shannon, with ideas inspired by Stefan Brunthaler.
See PEP 659(2) for more information. Implementation by Mark Shannon and
Brandt Bucher, with additional help from Irit Katriel and Dennis
Sweeney.)
Operation Form Specialization Operation speedup (up Contributor(s)
to)
---------------------------------------------------------------------------------------------------------------------------------------------------------
Binary operations ‘x + x’ Binary add, multiply and subtract for common types such 10% Mark Shannon, Donghee
as *note int: 259, *note float: 2f1. and *note str: 447. Na, Brandt Bucher,
‘x - x’ take custom fast paths for their underlying types. Dennis Sweeney
‘x * x’
Subscript ‘a[i]’ Subscripting container types such as *note list: 60d, 10-25% Irit Katriel, Mark
*note tuple: 36b. and *note dict: 258. directly index the Shannon
underlying data structures.
Subscripting custom *note __getitem__(): 285. is also
inlined similar to
*note Inlined Python function calls: 6de.
Store subscript ‘a[i] = z’ Similar to subscripting specialization above. 10-25% Dennis Sweeney
Calls ‘f(arg)’ Calls to common builtin (C) functions and types such as 20% Mark Shannon, Ken Jin
*note len(): 62a. and *note str: 447. directly call their
‘C(arg)’ underlying C version. This avoids going through the
internal calling convention.
Load global ‘print’ The object’s index in the globals/builtins namespace is (3) Mark Shannon
variable cached. Loading globals and builtins require zero
‘len’ namespace lookups.
Load attribute ‘o.attr’ Similar to loading global variables. The attribute’s (4) Mark Shannon
index inside the class/object’s namespace is cached. In
most cases, attribute loading will require zero namespace
lookups.
Load methods for ‘o.meth()’ The actual address of the method is cached. Method 10-20% Ken Jin, Mark Shannon
call loading now has no namespace lookups – even for classes
with long inheritance chains.
Store attribute ‘o.attr = z’ Similar to load attribute optimization. 2% in pyperformance Mark Shannon
Unpack Sequence ‘*seq’ Specialized for common containers such as 8% Brandt Bucher
*note list: 60d. and *note tuple: 36b. Avoids internal
calling convention.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0659/
(2) https://peps.python.org/pep-0659/
(3) A similar optimization already existed since Python 3.8. 3.11
specializes for more forms and reduces some overhead.
(4) A similar optimization already existed since Python 3.10. 3.11
specializes for more forms. Furthermore, all attribute loads should be
sped up by bpo-45947
(https://bugs.python.org/issue?@action=redirect&bpo=45947).
File: python.info, Node: Misc, Next: FAQ, Prev: Faster Runtime, Up: Faster CPython
1.3.9.7 Misc
............
* Objects now require less memory due to lazily created object
namespaces. Their namespace dictionaries now also share keys more
freely. (Contributed Mark Shannon in bpo-45340(1) and
bpo-40116(2).)
* “Zero-cost” exceptions are implemented, eliminating the cost of
*note try: 6e4. statements when no exception is raised.
(Contributed by Mark Shannon in bpo-40222(3).)
* A more concise representation of exceptions in the interpreter
reduced the time required for catching an exception by about 10%.
(Contributed by Irit Katriel in bpo-45711(4).)
* *note re: b9.’s regular expression matching engine has been
partially refactored, and now uses computed gotos (or “threaded
code”) on supported platforms. As a result, Python 3.11 executes
the pyperformance regular expression benchmarks(5) up to 10% faster
than Python 3.10. (Contributed by Brandt Bucher in gh-91404(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45340
(2) https://bugs.python.org/issue?@action=redirect&bpo=40116
(3) https://bugs.python.org/issue?@action=redirect&bpo=40222
(4) https://bugs.python.org/issue?@action=redirect&bpo=45711
(5) https://pyperformance.readthedocs.io/benchmarks.html#regex-dna
(6) https://github.com/python/cpython/issues/91404
File: python.info, Node: FAQ, Next: About, Prev: Misc, Up: Faster CPython
1.3.9.8 FAQ
...........
* Menu:
* How should I write my code to utilize these speedups?::
* Will CPython 3.11 use more memory?: Will CPython 3 11 use more memory?.
* I don’t see any speedups in my workload. Why?: I don’t see any speedups in my workload Why?.
* Is there a JIT compiler?::
File: python.info, Node: How should I write my code to utilize these speedups?, Next: Will CPython 3 11 use more memory?, Up: FAQ
1.3.9.9 How should I write my code to utilize these speedups?
.............................................................
Write Pythonic code that follows common best practices; you don’t have
to change your code. The Faster CPython project optimizes for common
code patterns we observe.
File: python.info, Node: Will CPython 3 11 use more memory?, Next: I don’t see any speedups in my workload Why?, Prev: How should I write my code to utilize these speedups?, Up: FAQ
1.3.9.10 Will CPython 3.11 use more memory?
...........................................
Maybe not; we don’t expect memory use to exceed 20% higher than 3.10.
This is offset by memory optimizations for frame objects and object
dictionaries as mentioned above.
File: python.info, Node: I don’t see any speedups in my workload Why?, Next: Is there a JIT compiler?, Prev: Will CPython 3 11 use more memory?, Up: FAQ
1.3.9.11 I don’t see any speedups in my workload. Why?
......................................................
Certain code won’t have noticeable benefits. If your code spends most
of its time on I/O operations, or already does most of its computation
in a C extension library like NumPy, there won’t be significant
speedups. This project currently benefits pure-Python workloads the
most.
Furthermore, the pyperformance figures are a geometric mean. Even
within the pyperformance benchmarks, certain benchmarks have slowed down
slightly, while others have sped up by nearly 2x!
File: python.info, Node: Is there a JIT compiler?, Prev: I don’t see any speedups in my workload Why?, Up: FAQ
1.3.9.12 Is there a JIT compiler?
.................................
No. We’re still exploring other optimizations.
File: python.info, Node: About, Prev: FAQ, Up: Faster CPython
1.3.9.13 About
..............
Faster CPython explores optimizations for *note CPython: 6f1. The main
team is funded by Microsoft to work on this full-time. Pablo Galindo
Salgado is also funded by Bloomberg LP to work on the project part-time.
Finally, many contributors are volunteers from the community.
File: python.info, Node: CPython bytecode changes<2>, Next: Deprecated<3>, Prev: Faster CPython, Up: What’s New In Python 3 11
1.3.10 CPython bytecode changes
-------------------------------
The bytecode now contains inline cache entries, which take the form of
the newly-added *note CACHE: 6f4. instructions. Many opcodes expect to
be followed by an exact number of caches, and instruct the interpreter
to skip over them at runtime. Populated caches can look like arbitrary
instructions, so great care should be taken when reading or modifying
raw, adaptive bytecode containing quickened data.
* Menu:
* New opcodes::
* Replaced opcodes::
* Changed/removed opcodes::
File: python.info, Node: New opcodes, Next: Replaced opcodes, Up: CPython bytecode changes<2>
1.3.10.1 New opcodes
....................
* ‘ASYNC_GEN_WRAP’, *note RETURN_GENERATOR: 6f7. and *note SEND: 6f8,
used in generators and co-routines.
* *note COPY_FREE_VARS: 6f9, which avoids needing special caller-side
code for closures.
* *note JUMP_BACKWARD_NO_INTERRUPT: 6fa, for use in certain loops
where handling interrupts is undesirable.
* *note MAKE_CELL: 6fb, to create *note Cell Objects: 6fc.
* *note CHECK_EG_MATCH: 6fd. and ‘PREP_RERAISE_STAR’, to handle the
*note new exception groups and except*: 59d. added in PEP 654(1).
* *note PUSH_EXC_INFO: 6fe, for use in exception handlers.
* *note RESUME: 30f, a no-op, for internal tracing, debugging and
optimization checks.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0654/
File: python.info, Node: Replaced opcodes, Next: Changed/removed opcodes, Prev: New opcodes, Up: CPython bytecode changes<2>
1.3.10.2 Replaced opcodes
.........................
Replaced Opcode(s) New Opcode(s) Notes
--------------------------------------------------------------------------------------------------------------------------------
‘BINARY_*’ *note BINARY_OP: 701. Replaced all numeric binary/in-place
‘INPLACE_*’ opcodes with a single opcode
‘CALL_FUNCTION’ *note CALL: 702. Decouples argument shifting for methods
‘CALL_FUNCTION_KW’ ‘KW_NAMES’ from handling of keyword arguments; allows
‘CALL_METHOD’ ‘PRECALL’ better specialization of calls
*note PUSH_NULL: 703.
‘DUP_TOP’ *note COPY: 704. Stack manipulation instructions
‘DUP_TOP_TWO’ *note SWAP: 705.
‘ROT_TWO’
‘ROT_THREE’
‘ROT_FOUR’
‘ROT_N’
‘JUMP_IF_NOT_EXC_MATCH’ *note CHECK_EXC_MATCH: 706. Now performs check but doesn’t jump
‘JUMP_ABSOLUTE’ *note JUMP_BACKWARD: 707. See (1); ‘TRUE’, ‘FALSE’, ‘NONE’ and
‘POP_JUMP_IF_FALSE’ ‘POP_JUMP_BACKWARD_IF_*’ ‘NOT_NONE’ variants for each direction
‘POP_JUMP_IF_TRUE’ ‘POP_JUMP_FORWARD_IF_*’
‘SETUP_WITH’ *note BEFORE_WITH: 708. *note with: 5ce. block setup
‘SETUP_ASYNC_WITH’
---------- Footnotes ----------
(1) All jump opcodes are now relative, including the existing
‘JUMP_IF_TRUE_OR_POP’ and ‘JUMP_IF_FALSE_OR_POP’. The argument is now
an offset from the current instruction rather than an absolute location.
File: python.info, Node: Changed/removed opcodes, Prev: Replaced opcodes, Up: CPython bytecode changes<2>
1.3.10.3 Changed/removed opcodes
................................
* Changed *note MATCH_CLASS: 70d. and *note MATCH_KEYS: 70e. to no
longer push an additional boolean value to indicate
success/failure. Instead, ‘None’ is pushed on failure in place of
the tuple of extracted values.
* Changed opcodes that work with exceptions to reflect them now being
represented as one item on the stack instead of three (see
gh-89874(1)).
* Removed ‘COPY_DICT_WITHOUT_KEYS’, ‘GEN_START’, ‘POP_BLOCK’,
‘SETUP_FINALLY’ and ‘YIELD_FROM’.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/89874
File: python.info, Node: Deprecated<3>, Next: Pending Removal in Python 3 12, Prev: CPython bytecode changes<2>, Up: What’s New In Python 3 11
1.3.11 Deprecated
-----------------
This section lists Python APIs that have been deprecated in Python 3.11.
Deprecated C APIs are *note listed separately: 712.
* Menu:
* Language/Builtins::
* Modules::
* Standard Library::
File: python.info, Node: Language/Builtins, Next: Modules, Up: Deprecated<3>
1.3.11.1 Language/Builtins
..........................
* Chaining *note classmethod: 166. descriptors (introduced in
bpo-19072(1)) is now deprecated. It can no longer be used to wrap
other descriptors such as *note property: 194. The core design of
this feature was flawed and caused a number of downstream problems.
To “pass-through” a *note classmethod: 166, consider using the
‘__wrapped__’ attribute that was added in Python 3.10.
(Contributed by Raymond Hettinger in gh-89519(2).)
* Octal escapes in string and bytes literals with values larger than
‘0o377’ (255 in decimal) now produce a *note DeprecationWarning:
1a5. In a future Python version, they will raise a *note
SyntaxWarning: 461. and eventually a *note SyntaxError: 18d.
(Contributed by Serhiy Storchaka in gh-81548(3).)
* The delegation of *note int(): 259. to *note __trunc__(): 716. is
now deprecated. Calling ‘int(a)’ when ‘type(a)’ implements
‘__trunc__()’ but not *note __int__(): 717. or *note __index__():
718. now raises a *note DeprecationWarning: 1a5. (Contributed by
Zackery Spytz in bpo-44977(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19072
(2) https://github.com/python/cpython/issues/89519
(3) https://github.com/python/cpython/issues/81548
(4) https://bugs.python.org/issue?@action=redirect&bpo=44977
File: python.info, Node: Modules, Next: Standard Library, Prev: Language/Builtins, Up: Deprecated<3>
1.3.11.2 Modules
................
* PEP 594(1) led to the deprecations of the following modules slated
for removal in Python 3.13:
‘aifc’ ‘chunk’ ‘msilib’ ‘pipes’ ‘telnetlib’
‘audioop’ ‘crypt’ ‘nis’ ‘sndhdr’ ‘uu’
‘cgi’ ‘imghdr’ ‘nntplib’ ‘spwd’ ‘xdrlib’
‘cgitb’ ‘mailcap’ ‘ossaudiodev’ ‘sunau’
(Contributed by Brett Cannon in bpo-47061(2) and Victor Stinner in
gh-68966(3).)
* The ‘asynchat’, ‘asyncore’ and ‘smtpd’ modules have been deprecated
since at least Python 3.6. Their documentation and deprecation
warnings have now been updated to note they will be removed in
Python 3.12. (Contributed by Hugo van Kemenade in bpo-47022(4).)
* The ‘lib2to3’ package and ‘2to3’ tool are now deprecated and may
not be able to parse Python 3.10 or newer. See PEP 617(5),
introducing the new PEG parser, for details. (Contributed by
Victor Stinner in bpo-40360(6).)
* Undocumented modules ‘sre_compile’, ‘sre_constants’ and ‘sre_parse’
are now deprecated. (Contributed by Serhiy Storchaka in
bpo-47152(7).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0594/
(2) https://bugs.python.org/issue?@action=redirect&bpo=47061
(3) https://github.com/python/cpython/issues/68966
(4) https://bugs.python.org/issue?@action=redirect&bpo=47022
(5) https://peps.python.org/pep-0617/
(6) https://bugs.python.org/issue?@action=redirect&bpo=40360
(7) https://bugs.python.org/issue?@action=redirect&bpo=47152
File: python.info, Node: Standard Library, Prev: Modules, Up: Deprecated<3>
1.3.11.3 Standard Library
.........................
* The following have been deprecated in *note configparser: 22. since
Python 3.2. Their deprecation warnings have now been updated to
note they will be removed in Python 3.12:
* the ‘configparser.SafeConfigParser’ class
* the ‘configparser.ParsingError.filename’ property
* the ‘configparser.RawConfigParser.readfp()’ method
(Contributed by Hugo van Kemenade in bpo-45173(1).)
* ‘configparser.LegacyInterpolation’ has been deprecated in the
docstring since Python 3.2, and is not listed in the *note
configparser: 22. documentation. It now emits a *note
DeprecationWarning: 1a5. and will be removed in Python 3.13. Use
*note configparser.BasicInterpolation: 71d. or *note
configparser.ExtendedInterpolation: 71e. instead. (Contributed by
Hugo van Kemenade in bpo-46607(2).)
* The older set of *note importlib.resources: 7b. functions were
deprecated in favor of the replacements added in Python 3.9 and
will be removed in a future Python version, due to not supporting
resources located within package subdirectories:
* ‘importlib.resources.contents()’
* ‘importlib.resources.is_resource()’
* ‘importlib.resources.open_binary()’
* ‘importlib.resources.open_text()’
* ‘importlib.resources.read_binary()’
* ‘importlib.resources.read_text()’
* ‘importlib.resources.path()’
* The *note locale.getdefaultlocale(): 2dd. function is deprecated
and will be removed in Python 3.15. Use *note locale.setlocale():
2df, *note locale.getpreferredencoding(False): 536. and *note
locale.getlocale(): 2de. functions instead. (Contributed by Victor
Stinner in gh-90817(3).)
* The ‘locale.resetlocale()’ function is deprecated and will be
removed in Python 3.13. Use ‘locale.setlocale(locale.LC_ALL, "")’
instead. (Contributed by Victor Stinner in gh-90817(4).)
* Stricter rules will now be applied for numerical group references
and group names in *note regular expressions: 71f. Only sequences
of ASCII digits will now be accepted as a numerical reference, and
the group name in *note bytes: 1c2. patterns and replacement
strings can only contain ASCII letters, digits and underscores.
For now, a deprecation warning is raised for syntax violating these
rules. (Contributed by Serhiy Storchaka in gh-91760(5).)
* In the *note re: b9. module, the ‘re.template()’ function and the
corresponding ‘re.TEMPLATE’ and ‘re.T’ flags are deprecated, as
they were undocumented and lacked an obvious purpose. They will be
removed in Python 3.13. (Contributed by Serhiy Storchaka and Miro
Hrončok in gh-92728(6).)
* ‘turtle.settiltangle()’ has been deprecated since Python 3.1; it
now emits a deprecation warning and will be removed in Python 3.13.
Use *note turtle.tiltangle(): 720. instead (it was earlier
incorrectly marked as deprecated, and its docstring is now
corrected). (Contributed by Hugo van Kemenade in bpo-45837(7).)
* *note typing.Text: 305, which exists solely to provide
compatibility support between Python 2 and Python 3 code, is now
deprecated. Its removal is currently unplanned, but users are
encouraged to use *note str: 447. instead wherever possible.
(Contributed by Alex Waygood in gh-92332(8).)
* The keyword argument syntax for constructing *note
typing.TypedDict: 162. types is now deprecated. Support will be
removed in Python 3.13. (Contributed by Jingchen Ye in
gh-90224(9).)
* ‘webbrowser.MacOSX’ is deprecated and will be removed in Python
3.13. It is untested, undocumented, and not used by *note
webbrowser: 115. itself. (Contributed by Donghee Na in
bpo-42255(10).)
* The behavior of returning a value from a *note TestCase: 449. and
*note IsolatedAsyncioTestCase: 306. test methods (other than the
default ‘None’ value) is now deprecated.
* Deprecated the following not-formally-documented *note unittest:
106. functions, scheduled for removal in Python 3.13:
* ‘unittest.findTestCases()’
* ‘unittest.makeSuite()’
* ‘unittest.getTestCaseNames()’
Use *note TestLoader: 290. methods instead:
* *note unittest.TestLoader.loadTestsFromModule(): 291.
* *note unittest.TestLoader.loadTestsFromTestCase(): 292.
* *note unittest.TestLoader.getTestCaseNames(): 293.
(Contributed by Erlend E. Aasland in bpo-5846(11).)
* ‘unittest.TestProgram.usageExit()’ is marked deprecated, to be
removed in 3.13. (Contributed by Carlos Damázio in gh-67048(12).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=45173
(2) https://bugs.python.org/issue?@action=redirect&bpo=46607
(3) https://github.com/python/cpython/issues/90817
(4) https://github.com/python/cpython/issues/90817
(5) https://github.com/python/cpython/issues/91760
(6) https://github.com/python/cpython/issues/92728
(7) https://bugs.python.org/issue?@action=redirect&bpo=45837
(8) https://github.com/python/cpython/issues/92332
(9) https://github.com/python/cpython/issues/90224
(10) https://bugs.python.org/issue?@action=redirect&bpo=42255
(11) https://bugs.python.org/issue?@action=redirect&bpo=5846
(12) https://github.com/python/cpython/issues/67048
File: python.info, Node: Pending Removal in Python 3 12, Next: Removed<3>, Prev: Deprecated<3>, Up: What’s New In Python 3 11
1.3.12 Pending Removal in Python 3.12
-------------------------------------
The following Python APIs have been deprecated in earlier Python
releases, and will be removed in Python 3.12.
C APIs pending removal are *note listed separately: 724.
* The ‘asynchat’ module
* The ‘asyncore’ module
* The *note entire distutils package: 725.
* The ‘imp’ module
* The *note typing.io: 726. namespace
* The *note typing.re: 727. namespace
* ‘cgi.log()’
* ‘importlib.find_loader()’
* ‘importlib.abc.Loader.module_repr()’
* ‘importlib.abc.MetaPathFinder.find_module()’
* ‘importlib.abc.PathEntryFinder.find_loader()’
* ‘importlib.abc.PathEntryFinder.find_module()’
* ‘importlib.machinery.BuiltinImporter.find_module()’
* ‘importlib.machinery.BuiltinLoader.module_repr()’
* ‘importlib.machinery.FileFinder.find_loader()’
* ‘importlib.machinery.FileFinder.find_module()’
* ‘importlib.machinery.FrozenImporter.find_module()’
* ‘importlib.machinery.FrozenLoader.module_repr()’
* ‘importlib.machinery.PathFinder.find_module()’
* ‘importlib.machinery.WindowsRegistryFinder.find_module()’
* ‘importlib.util.module_for_loader()’
* ‘importlib.util.set_loader_wrapper()’
* ‘importlib.util.set_package_wrapper()’
* ‘pkgutil.ImpImporter’
* ‘pkgutil.ImpLoader’
* ‘pathlib.Path.link_to()’
* ‘sqlite3.enable_shared_cache()’
* ‘sqlite3.OptimizedUnicode()’
* ‘PYTHONTHREADDEBUG’ environment variable
* The following deprecated aliases in *note unittest: 106.:
Deprecated alias Method Name Deprecated in
-----------------------------------------------------------------------------------------
‘failUnless’ *note assertTrue(): 522. 3.1
‘failIf’ *note assertFalse(): 523. 3.1
‘failUnlessEqual’ *note assertEqual(): 524. 3.1
‘failIfEqual’ *note assertNotEqual(): 525. 3.1
‘failUnlessAlmostEqual’ *note assertAlmostEqual(): 526. 3.1
‘failIfAlmostEqual’ *note assertNotAlmostEqual(): 527. 3.1
‘failUnlessRaises’ *note assertRaises(): 528. 3.1
‘assert_’ *note assertTrue(): 522. 3.2
‘assertEquals’ *note assertEqual(): 524. 3.2
‘assertNotEquals’ *note assertNotEqual(): 525. 3.2
‘assertAlmostEquals’ *note assertAlmostEqual(): 526. 3.2
‘assertNotAlmostEquals’ *note assertNotAlmostEqual(): 527. 3.2
‘assertRegexpMatches’ *note assertRegex(): 529. 3.2
‘assertRaisesRegexp’ *note assertRaisesRegex(): 52a. 3.2
‘assertNotRegexpMatches’ *note assertNotRegex(): 52b. 3.5
File: python.info, Node: Removed<3>, Next: Porting to Python 3 11, Prev: Pending Removal in Python 3 12, Up: What’s New In Python 3 11
1.3.13 Removed
--------------
This section lists Python APIs that have been removed in Python 3.11.
Removed C APIs are *note listed separately: 72b.
* Removed the ‘@asyncio.coroutine()’ *note decorator: 72c. enabling
legacy generator-based coroutines to be compatible with *note
async: 72d. / *note await: 1b9. code. The function has been
deprecated since Python 3.8 and the removal was initially scheduled
for Python 3.10. Use *note async def: 5cd. instead. (Contributed
by Illia Volochii in bpo-43216(1).)
* Removed ‘asyncio.coroutines.CoroWrapper’ used for wrapping legacy
generator-based coroutine objects in the debug mode. (Contributed
by Illia Volochii in bpo-43216(2).)
* Due to significant security concerns, the 'reuse_address' parameter
of *note asyncio.loop.create_datagram_endpoint(): 72e, disabled in
Python 3.9, is now entirely removed. This is because of the
behavior of the socket option ‘SO_REUSEADDR’ in UDP. (Contributed
by Hugo van Kemenade in bpo-45129(3).)
* Removed the ‘binhex’ module, deprecated in Python 3.9. Also
removed the related, similarly-deprecated *note binascii: 10.
functions:
* ‘binascii.a2b_hqx()’
* ‘binascii.b2a_hqx()’
* ‘binascii.rlecode_hqx()’
* ‘binascii.rldecode_hqx()’
The *note binascii.crc_hqx(): 72f. function remains available.
(Contributed by Victor Stinner in bpo-45085(4).)
* Removed the ‘distutils’ ‘bdist_msi’ command deprecated in Python
3.9. Use ‘bdist_wheel’ (wheel packages) instead. (Contributed by
Hugo van Kemenade in bpo-45124(5).)
* Removed the *note __getitem__(): 285. methods of *note
xml.dom.pulldom.DOMEventStream: 730, *note
wsgiref.util.FileWrapper: 731. and *note fileinput.FileInput: 732,
deprecated since Python 3.9. (Contributed by Hugo van Kemenade in
bpo-45132(6).)
* Removed the deprecated *note gettext: 63. functions ‘lgettext()’,
‘ldgettext()’, ‘lngettext()’ and ‘ldngettext()’. Also removed the
‘bind_textdomain_codeset()’ function, the
‘NullTranslations.output_charset()’ and
‘NullTranslations.set_output_charset()’ methods, and the 'codeset'
parameter of ‘translation()’ and ‘install()’, since they are only
used for the ‘l*gettext()’ functions. (Contributed by Donghee Na
and Serhiy Storchaka in bpo-44235(7).)
* Removed from the *note inspect: 7e. module:
* The ‘getargspec()’ function, deprecated since Python 3.0; use
*note inspect.signature(): 733. or *note
inspect.getfullargspec(): 734. instead.
* The ‘formatargspec()’ function, deprecated since Python 3.5;
use the *note inspect.signature(): 733. function or the *note
inspect.Signature: 1cf. object directly.
* The undocumented ‘Signature.from_builtin()’ and
‘Signature.from_function()’ methods, deprecated since Python
3.5; use the *note Signature.from_callable(): 735. method
instead.
(Contributed by Hugo van Kemenade in bpo-45320(8).)
* Removed the *note __class_getitem__(): 736. method from *note
pathlib.PurePath: 4a2, because it was not used and added by mistake
in previous versions. (Contributed by Nikita Sobolev in
bpo-46483(9).)
* Removed the ‘MailmanProxy’ class in the ‘smtpd’ module, as it is
unusable without the external ‘mailman’ package. (Contributed by
Donghee Na in bpo-35800(10).)
* Removed the deprecated ‘split()’ method of ‘_tkinter.TkappType’.
(Contributed by Erlend E. Aasland in bpo-38371(11).)
* Removed namespace package support from *note unittest: 106.
discovery. It was introduced in Python 3.4 but has been broken
since Python 3.7. (Contributed by Inada Naoki in bpo-23882(12).)
* Removed the undocumented private ‘float.__set_format__()’ method,
previously known as ‘float.__setformat__()’ in Python 3.7. Its
docstring said: “You probably don’t want to use this function. It
exists mainly to be used in Python’s test suite.” (Contributed by
Victor Stinner in bpo-46852(13).)
* The ‘--experimental-isolated-subinterpreters’ configure flag (and
corresponding ‘EXPERIMENTAL_ISOLATED_SUBINTERPRETERS’ macro) have
been removed.
* Pynche(14) — The Pythonically Natural Color and Hue Editor — has
been moved out of ‘Tools/scripts’ and is being developed
independently(15) from the Python source tree.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43216
(2) https://bugs.python.org/issue?@action=redirect&bpo=43216
(3) https://bugs.python.org/issue?@action=redirect&bpo=45129
(4) https://bugs.python.org/issue?@action=redirect&bpo=45085
(5) https://bugs.python.org/issue?@action=redirect&bpo=45124
(6) https://bugs.python.org/issue?@action=redirect&bpo=45132
(7) https://bugs.python.org/issue?@action=redirect&bpo=44235
(8) https://bugs.python.org/issue?@action=redirect&bpo=45320
(9) https://bugs.python.org/issue?@action=redirect&bpo=46483
(10) https://bugs.python.org/issue?@action=redirect&bpo=35800
(11) https://bugs.python.org/issue?@action=redirect&bpo=38371
(12) https://bugs.python.org/issue?@action=redirect&bpo=23882
(13) https://bugs.python.org/issue?@action=redirect&bpo=46852
(14) https://pypi.org/project/Pynche/
(15) https://gitlab.com/warsaw/pynche/-/tree/main
File: python.info, Node: Porting to Python 3 11, Next: Build Changes<3>, Prev: Removed<3>, Up: What’s New In Python 3 11
1.3.14 Porting to Python 3.11
-----------------------------
This section lists previously described changes and other bugfixes in
the Python API that may require changes to your Python code.
Porting notes for the C API are *note listed separately: 73a.
* *note open(): 517, *note io.open(): 518, *note codecs.open(): 73b.
and *note fileinput.FileInput: 732. no longer accept ‘'U'’
(“universal newline”) in the file mode. In Python 3, “universal
newline” mode is used by default whenever a file is opened in text
mode, and the ‘'U'’ flag has been deprecated since Python 3.3. The
*note newline parameter: 73c. to these functions controls how
universal newlines work. (Contributed by Victor Stinner in
bpo-37330(1).)
* *note ast.AST: 1a6. node positions are now validated when provided
to *note compile(): 192. and other related functions. If invalid
positions are detected, a *note ValueError: 204. will be raised.
(Contributed by Pablo Galindo in gh-93351(2))
* Prohibited passing non-*note concurrent.futures.ThreadPoolExecutor:
73d. executors to *note asyncio.loop.set_default_executor(): 73e.
following a deprecation in Python 3.8. (Contributed by Illia
Volochii in bpo-43234(3).)
* *note calendar: 14.: The *note calendar.LocaleTextCalendar: 73f.
and *note calendar.LocaleHTMLCalendar: 740. classes now use *note
locale.getlocale(): 2de, instead of using *note
locale.getdefaultlocale(): 2dd, if no locale is specified.
(Contributed by Victor Stinner in bpo-46659(4).)
* The *note pdb: a5. module now reads the ‘.pdbrc’ configuration file
with the ‘'UTF-8'’ encoding. (Contributed by Srinivas Reddy
Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి) in bpo-41137(5).)
* The 'population' parameter of *note random.sample(): 741. must be a
sequence, and automatic conversion of *note set: 5d5.s to *note
list: 60d.s is no longer supported. Also, if the sample size is
larger than the population size, a *note ValueError: 204. is
raised. (Contributed by Raymond Hettinger in bpo-40465(6).)
* The 'random' optional parameter of *note random.shuffle(): 742. was
removed. It was previously an arbitrary random function to use for
the shuffle; now, *note random.random(): 743. (its previous
default) will always be used.
* In *note re: b9. *note Regular Expression Syntax: 71f, global
inline flags (e.g. ‘(?i)’) can now only be used at the start of
regular expressions. Using them elsewhere has been deprecated
since Python 3.6. (Contributed by Serhiy Storchaka in
bpo-47066(7).)
* In the *note re: b9. module, several long-standing bugs where fixed
that, in rare cases, could cause capture groups to get the wrong
result. Therefore, this could change the captured output in these
cases. (Contributed by Ma Lin in bpo-35859(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37330
(2) https://github.com/python/cpython/issues/93351
(3) https://bugs.python.org/issue?@action=redirect&bpo=43234
(4) https://bugs.python.org/issue?@action=redirect&bpo=46659
(5) https://bugs.python.org/issue?@action=redirect&bpo=41137
(6) https://bugs.python.org/issue?@action=redirect&bpo=40465
(7) https://bugs.python.org/issue?@action=redirect&bpo=47066
(8) https://bugs.python.org/issue?@action=redirect&bpo=35859
File: python.info, Node: Build Changes<3>, Next: C API Changes<3>, Prev: Porting to Python 3 11, Up: What’s New In Python 3 11
1.3.15 Build Changes
--------------------
* CPython now has PEP 11(1) Tier 3 support(2) for cross compiling to
the WebAssembly(3) platforms Emscripten(4)
(‘wasm32-unknown-emscripten’, i.e. Python in the browser) and
WebAssembly System Interface (WASI)(5) (‘wasm32-unknown-wasi’).
The effort is inspired by previous work like Pyodide(6). These
platforms provide a limited subset of POSIX APIs; Python standard
libraries features and modules related to networking, processes,
threading, signals, mmap, and users/groups are not available or
don’t work. (Emscripten contributed by Christian Heimes and Ethan
Smith in gh-84461(7) and WASI contributed by Christian Heimes in
gh-90473(8); platforms promoted in gh-95085(9))
* Building CPython now requires:
* A C11(10) compiler and standard library. Optional C11
features(11) are not required. (Contributed by Victor Stinner
in bpo-46656(12), bpo-45440(13) and bpo-46640(14).)
* Support for IEEE 754(15) floating-point numbers. (Contributed
by Victor Stinner in bpo-46917(16).)
* The ‘Py_NO_NAN’ macro has been removed. Since CPython now requires
IEEE 754 floats, NaN values are always available. (Contributed by
Victor Stinner in bpo-46656(17).)
* The *note tkinter: f0. package now requires Tcl/Tk(18) version
8.5.12 or newer. (Contributed by Serhiy Storchaka in
bpo-46996(19).)
* Build dependencies, compiler flags, and linker flags for most
stdlib extension modules are now detected by ‘configure’. libffi,
libnsl, libsqlite3, zlib, bzip2, liblzma, libcrypt, Tcl/Tk, and
uuid flags are detected by pkg-config(20) (when available). *note
tkinter: f0. now requires a pkg-config command to detect
development settings for Tcl/Tk(21) headers and libraries.
(Contributed by Christian Heimes and Erlend Egeberg Aasland in
bpo-45847(22), bpo-45747(23), and bpo-45763(24).)
* libpython is no longer linked against libcrypt. (Contributed by
Mike Gilbert in bpo-45433(25).)
* CPython can now be built with the ThinLTO(26) option via passing
‘thin’ to *note -with-lto: 746, i.e. ‘--with-lto=thin’.
(Contributed by Donghee Na and Brett Holman in bpo-44340(27).)
* Freelists for object structs can now be disabled. A new
‘configure’ option *note -without-freelists: 747. can be used to
disable all freelists except empty tuple singleton. (Contributed
by Christian Heimes in bpo-45522(28).)
* ‘Modules/Setup’ and ‘Modules/makesetup’ have been improved and tied
up. Extension modules can now be built through ‘makesetup’. All
except some test modules can be linked statically into a main
binary or library. (Contributed by Brett Cannon and Christian
Heimes in bpo-45548(29), bpo-45570(30), bpo-45571(31), and
bpo-43974(32).)
Note: Use the environment variables ‘TCLTK_CFLAGS’ and
‘TCLTK_LIBS’ to manually specify the location of Tcl/Tk
headers and libraries. The ‘configure’ options
‘--with-tcltk-includes’ and ‘--with-tcltk-libs’ have been
removed.
On RHEL 7 and CentOS 7 the development packages do not provide
‘tcl.pc’ and ‘tk.pc’; use ‘TCLTK_LIBS="-ltk8.5 -ltkstub8.5
-ltcl8.5"’. The directory ‘Misc/rhel7’ contains ‘.pc’ files
and instructions on how to build Python with RHEL 7’s and
CentOS 7’s Tcl/Tk and OpenSSL.
* CPython will now use 30-bit digits by default for the Python *note
int: 259. implementation. Previously, the default was to use
30-bit digits on platforms with ‘SIZEOF_VOID_P >= 8’, and 15-bit
digits otherwise. It’s still possible to explicitly request use of
15-bit digits via either the *note -enable-big-digits: 748. option
to the configure script or (for Windows) the ‘PYLONG_BITS_IN_DIGIT’
variable in ‘PC/pyconfig.h’, but this option may be removed at some
point in the future. (Contributed by Mark Dickinson in
bpo-45569(33).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0011/
(2) https://peps.python.org/pep-0011/#tier-3
(3) https://webassembly.org/
(4) https://emscripten.org/
(5) https://wasi.dev/
(6) https://pyodide.org/
(7) https://github.com/python/cpython/issues/84461
(8) https://github.com/python/cpython/issues/90473
(9) https://github.com/python/cpython/issues/95085
(10) https://en.cppreference.com/w/c/11
(11)
https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Optional_features
(12) https://bugs.python.org/issue?@action=redirect&bpo=46656
(13) https://bugs.python.org/issue?@action=redirect&bpo=45440
(14) https://bugs.python.org/issue?@action=redirect&bpo=46640
(15) https://en.wikipedia.org/wiki/IEEE_754
(16) https://bugs.python.org/issue?@action=redirect&bpo=46917
(17) https://bugs.python.org/issue?@action=redirect&bpo=46656
(18) https://www.tcl.tk
(19) https://bugs.python.org/issue?@action=redirect&bpo=46996
(20) https://www.freedesktop.org/wiki/Software/pkg-config/
(21) https://www.tcl.tk
(22) https://bugs.python.org/issue?@action=redirect&bpo=45847
(23) https://bugs.python.org/issue?@action=redirect&bpo=45747
(24) https://bugs.python.org/issue?@action=redirect&bpo=45763
(25) https://bugs.python.org/issue?@action=redirect&bpo=45433
(26) https://clang.llvm.org/docs/ThinLTO.html
(27) https://bugs.python.org/issue?@action=redirect&bpo=44340
(28) https://bugs.python.org/issue?@action=redirect&bpo=45522
(29) https://bugs.python.org/issue?@action=redirect&bpo=45548
(30) https://bugs.python.org/issue?@action=redirect&bpo=45570
(31) https://bugs.python.org/issue?@action=redirect&bpo=45571
(32) https://bugs.python.org/issue?@action=redirect&bpo=43974
(33) https://bugs.python.org/issue?@action=redirect&bpo=45569
File: python.info, Node: C API Changes<3>, Next: Notable changes in 3 11 4, Prev: Build Changes<3>, Up: What’s New In Python 3 11
1.3.16 C API Changes
--------------------
* Menu:
* New Features: New Features<6>.
* Porting to Python 3.11: Porting to Python 3 11<2>.
* Deprecated: Deprecated<4>.
* Pending Removal in Python 3.12: Pending Removal in Python 3 12<2>.
* Removed: Removed<4>.
File: python.info, Node: New Features<6>, Next: Porting to Python 3 11<2>, Up: C API Changes<3>
1.3.16.1 New Features
.....................
* Add a new *note PyType_GetName(): 74d. function to get type’s short
name. (Contributed by Hai Shi in bpo-42035(1).)
* Add a new *note PyType_GetQualName(): 74e. function to get type’s
qualified name. (Contributed by Hai Shi in bpo-42035(2).)
* Add new *note PyThreadState_EnterTracing(): 74f. and *note
PyThreadState_LeaveTracing(): 750. functions to the limited C API
to suspend and resume tracing and profiling. (Contributed by
Victor Stinner in bpo-43760(3).)
* Added the *note Py_Version: 751. constant which bears the same
value as *note PY_VERSION_HEX: 752. (Contributed by Gabriele N.
Tornetta in bpo-43931(4).)
* *note Py_buffer: 753. and APIs are now part of the limited API and
the stable ABI:
* *note PyObject_CheckBuffer(): 394.
* *note PyObject_GetBuffer(): 395.
* *note PyBuffer_GetPointer(): 754.
* *note PyBuffer_SizeFromFormat(): 755.
* *note PyBuffer_ToContiguous(): 756.
* *note PyBuffer_FromContiguous(): 757.
* *note PyObject_CopyData(): 758.
* *note PyBuffer_IsContiguous(): 759.
* *note PyBuffer_FillContiguousStrides(): 75a.
* *note PyBuffer_FillInfo(): 75b.
* *note PyBuffer_Release(): 396.
* *note PyMemoryView_FromBuffer(): 75c.
* *note bf_getbuffer: 75d. and *note bf_releasebuffer: 75e. type
slots
(Contributed by Christian Heimes in bpo-45459(5).)
* Added the *note PyType_GetModuleByDef(): 38f. function, used to get
the module in which a method was defined, in cases where this
information is not available directly (via *note PyCMethod: 75f.).
(Contributed by Petr Viktorin in bpo-46613(6).)
* Add new functions to pack and unpack C double (serialize and
deserialize): *note PyFloat_Pack2(): 760, *note PyFloat_Pack4():
761, *note PyFloat_Pack8(): 762, *note PyFloat_Unpack2(): 763,
*note PyFloat_Unpack4(): 764. and *note PyFloat_Unpack8(): 765.
(Contributed by Victor Stinner in bpo-46906(7).)
* Add new functions to get frame object attributes: *note
PyFrame_GetBuiltins(): 766, *note PyFrame_GetGenerator(): 767,
*note PyFrame_GetGlobals(): 768, *note PyFrame_GetLasti(): 769.
* Added two new functions to get and set the active exception
instance: *note PyErr_GetHandledException(): 76a. and *note
PyErr_SetHandledException(): 76b. These are alternatives to *note
PyErr_SetExcInfo(): 76c. and *note PyErr_GetExcInfo(): 76d. which
work with the legacy 3-tuple representation of exceptions.
(Contributed by Irit Katriel in bpo-46343(8).)
* Added the *note PyConfig.safe_path: 76e. member. (Contributed by
Victor Stinner in gh-57684(9).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42035
(2) https://bugs.python.org/issue?@action=redirect&bpo=42035
(3) https://bugs.python.org/issue?@action=redirect&bpo=43760
(4) https://bugs.python.org/issue?@action=redirect&bpo=43931
(5) https://bugs.python.org/issue?@action=redirect&bpo=45459
(6) https://bugs.python.org/issue?@action=redirect&bpo=46613
(7) https://bugs.python.org/issue?@action=redirect&bpo=46906
(8) https://bugs.python.org/issue?@action=redirect&bpo=46343
(9) https://github.com/python/cpython/issues/57684
File: python.info, Node: Porting to Python 3 11<2>, Next: Deprecated<4>, Prev: New Features<6>, Up: C API Changes<3>
1.3.16.2 Porting to Python 3.11
...............................
* Some macros have been converted to static inline functions to avoid
macro pitfalls(1). The change should be mostly transparent to
users, as the replacement functions will cast their arguments to
the expected types to avoid compiler warnings due to static type
checks. However, when the limited C API is set to >=3.11, these
casts are not done, and callers will need to cast arguments to
their expected types. See PEP 670(2) for more details.
(Contributed by Victor Stinner and Erlend E. Aasland in
gh-89653(3).)
* *note PyErr_SetExcInfo(): 76c. no longer uses the ‘type’ and
‘traceback’ arguments, the interpreter now derives those values
from the exception instance (the ‘value’ argument). The function
still steals references of all three arguments. (Contributed by
Irit Katriel in bpo-45711(4).)
* *note PyErr_GetExcInfo(): 76d. now derives the ‘type’ and
‘traceback’ fields of the result from the exception instance (the
‘value’ field). (Contributed by Irit Katriel in bpo-45711(5).)
* *note _frozen: 770. has a new ‘is_package’ field to indicate
whether or not the frozen module is a package. Previously, a
negative value in the ‘size’ field was the indicator. Now only
non-negative values be used for ‘size’. (Contributed by Kumar
Aditya in bpo-46608(6).)
* *note _PyFrameEvalFunction(): 771. now takes ‘_PyInterpreterFrame*’
as its second parameter, instead of ‘PyFrameObject*’. See PEP
523(7) for more details of how to use this function pointer type.
* ‘PyCode_New()’ and ‘PyCode_NewWithPosOnlyArgs()’ now take an
additional ‘exception_table’ argument. Using these functions
should be avoided, if at all possible. To get a custom code
object: create a code object using the compiler, then get a
modified version with the ‘replace’ method.
* *note PyCodeObject: 772. no longer has the ‘co_code’,
‘co_varnames’, ‘co_cellvars’ and ‘co_freevars’ fields. Instead,
use *note PyCode_GetCode(): 773, *note PyCode_GetVarnames(): 774,
*note PyCode_GetCellvars(): 775. and *note PyCode_GetFreevars():
776. respectively to access them via the C API. (Contributed by
Brandt Bucher in bpo-46841(8) and Ken Jin in gh-92154(9) and
gh-94936(10).)
* The old trashcan macros
(‘Py_TRASHCAN_SAFE_BEGIN’/‘Py_TRASHCAN_SAFE_END’) are now
deprecated. They should be replaced by the new macros
‘Py_TRASHCAN_BEGIN’ and ‘Py_TRASHCAN_END’.
A tp_dealloc function that has the old macros, such as:
static void
mytype_dealloc(mytype *p)
{
PyObject_GC_UnTrack(p);
Py_TRASHCAN_SAFE_BEGIN(p);
...
Py_TRASHCAN_SAFE_END
}
should migrate to the new macros as follows:
static void
mytype_dealloc(mytype *p)
{
PyObject_GC_UnTrack(p);
Py_TRASHCAN_BEGIN(p, mytype_dealloc)
...
Py_TRASHCAN_END
}
Note that ‘Py_TRASHCAN_BEGIN’ has a second argument which should be
the deallocation function it is in.
To support older Python versions in the same codebase, you can
define the following macros and use them throughout the code
(credit: these were copied from the ‘mypy’ codebase):
#if PY_VERSION_HEX >= 0x03080000
# define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_BEGIN(op, dealloc)
# define CPy_TRASHCAN_END(op) Py_TRASHCAN_END
#else
# define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_SAFE_BEGIN(op)
# define CPy_TRASHCAN_END(op) Py_TRASHCAN_SAFE_END(op)
#endif
* The *note PyType_Ready(): 777. function now raises an error if a
type is defined with the *note Py_TPFLAGS_HAVE_GC: 778. flag set
but has no traverse function (*note PyTypeObject.tp_traverse:
779.). (Contributed by Victor Stinner in bpo-44263(11).)
* Heap types with the *note Py_TPFLAGS_IMMUTABLETYPE: 3be. flag can
now inherit the PEP 590(12) vectorcall protocol. Previously, this
was only possible for *note static types: 77a. (Contributed by
Erlend E. Aasland in bpo-43908(13))
* Since *note Py_TYPE(): 77b. is changed to a inline static function,
‘Py_TYPE(obj) = new_type’ must be replaced with ‘Py_SET_TYPE(obj,
new_type)’: see the *note Py_SET_TYPE(): 77c. function (available
since Python 3.9). For backward compatibility, this macro can be
used:
#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_TYPE)
static inline void _Py_SET_TYPE(PyObject *ob, PyTypeObject *type)
{ ob->ob_type = type; }
#define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type)
#endif
(Contributed by Victor Stinner in bpo-39573(14).)
* Since *note Py_SIZE(): 77d. is changed to a inline static function,
‘Py_SIZE(obj) = new_size’ must be replaced with ‘Py_SET_SIZE(obj,
new_size)’: see the *note Py_SET_SIZE(): 77e. function (available
since Python 3.9). For backward compatibility, this macro can be
used:
#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_SIZE)
static inline void _Py_SET_SIZE(PyVarObject *ob, Py_ssize_t size)
{ ob->ob_size = size; }
#define Py_SET_SIZE(ob, size) _Py_SET_SIZE((PyVarObject*)(ob), size)
#endif
(Contributed by Victor Stinner in bpo-39573(15).)
* ‘’ no longer includes the header files ‘’,
‘’, ‘’ and ‘’ when the ‘Py_LIMITED_API’
macro is set to ‘0x030b0000’ (Python 3.11) or higher. C extensions
should explicitly include the header files after ‘#include
’. (Contributed by Victor Stinner in bpo-45434(16).)
* The non-limited API files ‘cellobject.h’, ‘classobject.h’,
‘code.h’, ‘context.h’, ‘funcobject.h’, ‘genobject.h’ and
‘longintrepr.h’ have been moved to the ‘Include/cpython’ directory.
Moreover, the ‘eval.h’ header file was removed. These files must
not be included directly, as they are already included in
‘Python.h’: *note Include Files: 77f. If they have been included
directly, consider including ‘Python.h’ instead. (Contributed by
Victor Stinner in bpo-35134(17).)
* The ‘PyUnicode_CHECK_INTERNED()’ macro has been excluded from the
limited C API. It was never usable there, because it used internal
structures which are not available in the limited C API.
(Contributed by Victor Stinner in bpo-46007(18).)
* The following frame functions and type are now directly available
with ‘#include ’, it’s no longer needed to add ‘#include
’:
* *note PyFrame_Check(): 780.
* *note PyFrame_GetBack(): 781.
* *note PyFrame_GetBuiltins(): 766.
* *note PyFrame_GetGenerator(): 767.
* *note PyFrame_GetGlobals(): 768.
* *note PyFrame_GetLasti(): 769.
* *note PyFrame_GetLocals(): 41b.
* *note PyFrame_Type: 782.
(Contributed by Victor Stinner in gh-93937(19).)
* The *note PyFrameObject: 784. structure members have been removed
from the public C API.
While the documentation notes that the *note PyFrameObject: 784.
fields are subject to change at any time, they have been stable for
a long time and were used in several popular extensions.
In Python 3.11, the frame struct was reorganized to allow
performance optimizations. Some fields were removed entirely, as
they were details of the old implementation.
*note PyFrameObject: 784. fields:
* ‘f_back’: use *note PyFrame_GetBack(): 781.
* ‘f_blockstack’: removed.
* ‘f_builtins’: use *note PyFrame_GetBuiltins(): 766.
* ‘f_code’: use *note PyFrame_GetCode(): 785.
* ‘f_gen’: use *note PyFrame_GetGenerator(): 767.
* ‘f_globals’: use *note PyFrame_GetGlobals(): 768.
* ‘f_iblock’: removed.
* ‘f_lasti’: use *note PyFrame_GetLasti(): 769. Code using
‘f_lasti’ with ‘PyCode_Addr2Line()’ should use *note
PyFrame_GetLineNumber(): 786. instead; it may be faster.
* ‘f_lineno’: use *note PyFrame_GetLineNumber(): 786.
* ‘f_locals’: use *note PyFrame_GetLocals(): 41b.
* ‘f_stackdepth’: removed.
* ‘f_state’: no public API (renamed to ‘f_frame.f_state’).
* ‘f_trace’: no public API.
* ‘f_trace_lines’: use ‘PyObject_GetAttrString((PyObject*)frame,
"f_trace_lines")’.
* ‘f_trace_opcodes’: use
‘PyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes")’.
* ‘f_localsplus’: no public API (renamed to
‘f_frame.localsplus’).
* ‘f_valuestack’: removed.
The Python frame object is now created lazily. A side effect is
that the *note f_back: 787. member must not be accessed directly,
since its value is now also computed lazily. The *note
PyFrame_GetBack(): 781. function must be called instead.
Debuggers that accessed the *note f_locals: 182. directly 'must'
call *note PyFrame_GetLocals(): 41b. instead. They no longer need
to call ‘PyFrame_FastToLocalsWithError()’ or
‘PyFrame_LocalsToFast()’, in fact they should not call those
functions. The necessary updating of the frame is now managed by
the virtual machine.
Code defining ‘PyFrame_GetCode()’ on Python 3.8 and older:
#if PY_VERSION_HEX < 0x030900B1
static inline PyCodeObject* PyFrame_GetCode(PyFrameObject *frame)
{
Py_INCREF(frame->f_code);
return frame->f_code;
}
#endif
Code defining ‘PyFrame_GetBack()’ on Python 3.8 and older:
#if PY_VERSION_HEX < 0x030900B1
static inline PyFrameObject* PyFrame_GetBack(PyFrameObject *frame)
{
Py_XINCREF(frame->f_back);
return frame->f_back;
}
#endif
Or use the pythoncapi_compat project(20) to get these two functions
on older Python versions.
* Changes of the *note PyThreadState: 788. structure members:
* ‘frame’: removed, use *note PyThreadState_GetFrame(): 789.
(function added to Python 3.9 by bpo-40429(21)). Warning: the
function returns a *note strong reference: 338, need to call
*note Py_XDECREF(): 78a.
* ‘tracing’: changed, use *note PyThreadState_EnterTracing():
74f. and *note PyThreadState_LeaveTracing(): 750. (functions
added to Python 3.11 by bpo-43760(22)).
* ‘recursion_depth’: removed, use ‘(tstate->recursion_limit -
tstate->recursion_remaining)’ instead.
* ‘stackcheck_counter’: removed.
Code defining ‘PyThreadState_GetFrame()’ on Python 3.8 and older:
#if PY_VERSION_HEX < 0x030900B1
static inline PyFrameObject* PyThreadState_GetFrame(PyThreadState *tstate)
{
Py_XINCREF(tstate->frame);
return tstate->frame;
}
#endif
Code defining ‘PyThreadState_EnterTracing()’ and
‘PyThreadState_LeaveTracing()’ on Python 3.10 and older:
#if PY_VERSION_HEX < 0x030B00A2
static inline void PyThreadState_EnterTracing(PyThreadState *tstate)
{
tstate->tracing++;
#if PY_VERSION_HEX >= 0x030A00A1
tstate->cframe->use_tracing = 0;
#else
tstate->use_tracing = 0;
#endif
}
static inline void PyThreadState_LeaveTracing(PyThreadState *tstate)
{
int use_tracing = (tstate->c_tracefunc != NULL || tstate->c_profilefunc != NULL);
tstate->tracing--;
#if PY_VERSION_HEX >= 0x030A00A1
tstate->cframe->use_tracing = use_tracing;
#else
tstate->use_tracing = use_tracing;
#endif
}
#endif
Or use the pythoncapi-compat project(23) to get these functions on
old Python functions.
* Distributors are encouraged to build Python with the optimized
Blake2 library libb2(24).
* The *note PyConfig.module_search_paths_set: 5eb. field must now be
set to 1 for initialization to use *note
PyConfig.module_search_paths: 39e. to initialize *note sys.path:
3b0. Otherwise, initialization will recalculate the path and
replace any values added to ‘module_search_paths’.
* *note PyConfig_Read(): 78b. no longer calculates the initial search
path, and will not fill any values into *note
PyConfig.module_search_paths: 39e. To calculate default paths and
then modify them, finish initialization and use *note
PySys_GetObject(): 384. to retrieve *note sys.path: 3b0. as a
Python list object and modify it directly.
---------- Footnotes ----------
(1) https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html
(2) https://peps.python.org/pep-0670/
(3) https://github.com/python/cpython/issues/89653
(4) https://bugs.python.org/issue?@action=redirect&bpo=45711
(5) https://bugs.python.org/issue?@action=redirect&bpo=45711
(6) https://bugs.python.org/issue?@action=redirect&bpo=46608
(7) https://peps.python.org/pep-0523/
(8) https://bugs.python.org/issue?@action=redirect&bpo=46841
(9) https://github.com/python/cpython/issues/92154
(10) https://github.com/python/cpython/issues/94936
(11) https://bugs.python.org/issue?@action=redirect&bpo=44263
(12) https://peps.python.org/pep-0590/
(13) https://bugs.python.org/issue?@action=redirect&bpo=43908
(14) https://bugs.python.org/issue?@action=redirect&bpo=39573
(15) https://bugs.python.org/issue?@action=redirect&bpo=39573
(16) https://bugs.python.org/issue?@action=redirect&bpo=45434
(17) https://bugs.python.org/issue?@action=redirect&bpo=35134
(18) https://bugs.python.org/issue?@action=redirect&bpo=46007
(19) https://github.com/python/cpython/issues/93937
(20) https://github.com/python/pythoncapi-compat
(21) https://bugs.python.org/issue?@action=redirect&bpo=40429
(22) https://bugs.python.org/issue?@action=redirect&bpo=43760
(23) https://github.com/python/pythoncapi-compat
(24) https://www.blake2.net/
File: python.info, Node: Deprecated<4>, Next: Pending Removal in Python 3 12<2>, Prev: Porting to Python 3 11<2>, Up: C API Changes<3>
1.3.16.3 Deprecated
...................
* Deprecate the following functions to configure the Python
initialization:
* ‘PySys_AddWarnOptionUnicode()’
* ‘PySys_AddWarnOption()’
* ‘PySys_AddXOption()’
* ‘PySys_HasWarnOptions()’
* ‘PySys_SetArgvEx()’
* ‘PySys_SetArgv()’
* ‘PySys_SetPath()’
* ‘Py_SetPath()’
* ‘Py_SetProgramName()’
* ‘Py_SetPythonHome()’
* ‘Py_SetStandardStreamEncoding()’
* ‘_Py_SetProgramFullPath()’
Use the new *note PyConfig: 3a2. API of the *note Python
Initialization Configuration: 3a3. instead ( PEP 587(1)).
(Contributed by Victor Stinner in gh-88279(2).)
* Deprecate the ‘ob_shash’ member of the *note PyBytesObject: 78d.
Use *note PyObject_Hash(): 3ff. instead. (Contributed by Inada
Naoki in bpo-46864(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0587/
(2) https://github.com/python/cpython/issues/88279
(3) https://bugs.python.org/issue?@action=redirect&bpo=46864
File: python.info, Node: Pending Removal in Python 3 12<2>, Next: Removed<4>, Prev: Deprecated<4>, Up: C API Changes<3>
1.3.16.4 Pending Removal in Python 3.12
.......................................
The following C APIs have been deprecated in earlier Python releases,
and will be removed in Python 3.12.
* ‘PyUnicode_AS_DATA()’
* ‘PyUnicode_AS_UNICODE()’
* ‘PyUnicode_AsUnicodeAndSize()’
* ‘PyUnicode_AsUnicode()’
* ‘PyUnicode_FromUnicode()’
* ‘PyUnicode_GET_DATA_SIZE()’
* ‘PyUnicode_GET_SIZE()’
* ‘PyUnicode_GetSize()’
* ‘PyUnicode_IS_COMPACT()’
* ‘PyUnicode_IS_READY()’
* *note PyUnicode_READY(): 3fd.
* ‘PyUnicode_WSTR_LENGTH()’
* ‘_PyUnicode_AsUnicode()’
* ‘PyUnicode_WCHAR_KIND’
* *note PyUnicodeObject: 78f.
* ‘PyUnicode_InternImmortal()’
File: python.info, Node: Removed<4>, Prev: Pending Removal in Python 3 12<2>, Up: C API Changes<3>
1.3.16.5 Removed
................
* ‘PyFrame_BlockSetup()’ and ‘PyFrame_BlockPop()’ have been removed.
(Contributed by Mark Shannon in bpo-40222(1).)
* Remove the following math macros using the ‘errno’ variable:
* ‘Py_ADJUST_ERANGE1()’
* ‘Py_ADJUST_ERANGE2()’
* ‘Py_OVERFLOWED()’
* ‘Py_SET_ERANGE_IF_OVERFLOW()’
* ‘Py_SET_ERRNO_ON_MATH_ERROR()’
(Contributed by Victor Stinner in bpo-45412(2).)
* Remove ‘Py_UNICODE_COPY()’ and ‘Py_UNICODE_FILL()’ macros,
deprecated since Python 3.3. Use ‘PyUnicode_CopyCharacters()’ or
‘memcpy()’ (‘wchar_t*’ string), and ‘PyUnicode_Fill()’ functions
instead. (Contributed by Victor Stinner in bpo-41123(3).)
* Remove the ‘pystrhex.h’ header file. It only contains private
functions. C extensions should only include the main ‘’
header file. (Contributed by Victor Stinner in bpo-45434(4).)
* Remove the ‘Py_FORCE_DOUBLE()’ macro. It was used by the
‘Py_IS_INFINITY()’ macro. (Contributed by Victor Stinner in
bpo-45440(5).)
* The following items are no longer available when *note
Py_LIMITED_API: 41a. is defined:
* *note PyMarshal_WriteLongToFile(): 791.
* *note PyMarshal_WriteObjectToFile(): 792.
* *note PyMarshal_ReadObjectFromString(): 793.
* *note PyMarshal_WriteObjectToString(): 794.
* the ‘Py_MARSHAL_VERSION’ macro
These are not part of the *note limited API: 795.
(Contributed by Victor Stinner in bpo-45474(6).)
* Exclude *note PyWeakref_GET_OBJECT(): 3bb. from the limited C API.
It never worked since the ‘PyWeakReference’ structure is opaque in
the limited C API. (Contributed by Victor Stinner in bpo-35134(7).)
* Remove the ‘PyHeapType_GET_MEMBERS()’ macro. It was exposed in the
public C API by mistake, it must only be used by Python internally.
Use the ‘PyTypeObject.tp_members’ member instead. (Contributed by
Victor Stinner in bpo-40170(8).)
* Remove the ‘HAVE_PY_SET_53BIT_PRECISION’ macro (moved to the
internal C API). (Contributed by Victor Stinner in bpo-45412(9).)
* Remove the *note Py_UNICODE: 3e9. encoder APIs, as they have been
deprecated since Python 3.3, are little used and are inefficient
relative to the recommended alternatives.
The removed functions are:
* ‘PyUnicode_Encode()’
* ‘PyUnicode_EncodeASCII()’
* ‘PyUnicode_EncodeLatin1()’
* ‘PyUnicode_EncodeUTF7()’
* ‘PyUnicode_EncodeUTF8()’
* ‘PyUnicode_EncodeUTF16()’
* ‘PyUnicode_EncodeUTF32()’
* ‘PyUnicode_EncodeUnicodeEscape()’
* ‘PyUnicode_EncodeRawUnicodeEscape()’
* ‘PyUnicode_EncodeCharmap()’
* ‘PyUnicode_TranslateCharmap()’
* ‘PyUnicode_EncodeDecimal()’
* ‘PyUnicode_TransformDecimalToASCII()’
See PEP 624(10) for details and migration guidance(11).
(Contributed by Inada Naoki in bpo-44029(12).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40222
(2) https://bugs.python.org/issue?@action=redirect&bpo=45412
(3) https://bugs.python.org/issue?@action=redirect&bpo=41123
(4) https://bugs.python.org/issue?@action=redirect&bpo=45434
(5) https://bugs.python.org/issue?@action=redirect&bpo=45440
(6) https://bugs.python.org/issue?@action=redirect&bpo=45474
(7) https://bugs.python.org/issue?@action=redirect&bpo=35134
(8) https://bugs.python.org/issue?@action=redirect&bpo=40170
(9) https://bugs.python.org/issue?@action=redirect&bpo=45412
(10) https://peps.python.org/pep-0624/
(11) https://peps.python.org/pep-0624/#alternative-apis
(12) https://bugs.python.org/issue?@action=redirect&bpo=44029
File: python.info, Node: Notable changes in 3 11 4, Next: Notable changes in 3 11 5, Prev: C API Changes<3>, Up: What’s New In Python 3 11
1.3.17 Notable changes in 3.11.4
--------------------------------
* Menu:
* tarfile: tarfile<2>.
File: python.info, Node: tarfile<2>, Up: Notable changes in 3 11 4
1.3.17.1 tarfile
................
* The extraction methods in *note tarfile: de, and *note
shutil.unpack_archive(): 467, have a new a 'filter' argument that
allows limiting tar features than may be surprising or dangerous,
such as creating files outside the destination directory. See
*note Extraction filters: 468. for details. In Python 3.12, use
without the 'filter' argument will show a *note DeprecationWarning:
1a5. In Python 3.14, the default will switch to ‘'data'’.
(Contributed by Petr Viktorin in PEP 706(1).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0706/
File: python.info, Node: Notable changes in 3 11 5, Prev: Notable changes in 3 11 4, Up: What’s New In Python 3 11
1.3.18 Notable changes in 3.11.5
--------------------------------
* Menu:
* OpenSSL::
File: python.info, Node: OpenSSL, Up: Notable changes in 3 11 5
1.3.18.1 OpenSSL
................
* Windows builds and macOS installers from python.org now use OpenSSL
3.0.
File: python.info, Node: What’s New In Python 3 10, Next: What’s New In Python 3 9, Prev: What’s New In Python 3 11, Up: What’s New in Python
1.4 What’s New In Python 3.10
=============================
Editor: Pablo Galindo Salgado
This article explains the new features in Python 3.10, compared to 3.9.
Python 3.10 was released on October 4, 2021. For full details, see the
*note changelog: 13c.
* Menu:
* Summary – Release highlights: Summary – Release highlights<3>.
* New Features: New Features<7>.
* New Features Related to Type Hints: New Features Related to Type Hints<3>.
* Other Language Changes: Other Language Changes<4>.
* New Modules: New Modules<4>.
* Improved Modules: Improved Modules<4>.
* Optimizations: Optimizations<4>.
* Deprecated: Deprecated<5>.
* Removed: Removed<5>.
* Porting to Python 3.10: Porting to Python 3 10.
* CPython bytecode changes: CPython bytecode changes<3>.
* Build Changes: Build Changes<4>.
* C API Changes: C API Changes<4>.
* Notable security feature in 3.10.7: Notable security feature in 3 10 7.
* Notable security feature in 3.10.8: Notable security feature in 3 10 8.
* Notable changes in 3.10.12: Notable changes in 3 10 12.
File: python.info, Node: Summary – Release highlights<3>, Next: New Features<7>, Up: What’s New In Python 3 10
1.4.1 Summary – Release highlights
----------------------------------
New syntax features:
* PEP 634(1), Structural Pattern Matching: Specification
* PEP 635(2), Structural Pattern Matching: Motivation and Rationale
* PEP 636(3), Structural Pattern Matching: Tutorial
* bpo-12782(4), Parenthesized context managers are now officially
allowed.
New features in the standard library:
* PEP 618(5), Add Optional Length-Checking To zip.
Interpreter improvements:
* PEP 626(6), Precise line numbers for debugging and other tools.
New typing features:
* PEP 604(7), Allow writing union types as X | Y
* PEP 612(8), Parameter Specification Variables
* PEP 613(9), Explicit Type Aliases
* PEP 647(10), User-Defined Type Guards
Important deprecations, removals or restrictions:
* PEP 644(11), Require OpenSSL 1.1.1 or newer
* PEP 632(12), Deprecate distutils module.
* PEP 623(13), Deprecate and prepare for the removal of the wstr
member in PyUnicodeObject.
* PEP 624(14), Remove Py_UNICODE encoder APIs
* PEP 597(15), Add optional EncodingWarning
---------- Footnotes ----------
(1) https://peps.python.org/pep-0634/
(2) https://peps.python.org/pep-0635/
(3) https://peps.python.org/pep-0636/
(4) https://bugs.python.org/issue?@action=redirect&bpo=12782
(5) https://peps.python.org/pep-0618/
(6) https://peps.python.org/pep-0626/
(7) https://peps.python.org/pep-0604/
(8) https://peps.python.org/pep-0612/
(9) https://peps.python.org/pep-0613/
(10) https://peps.python.org/pep-0647/
(11) https://peps.python.org/pep-0644/
(12) https://peps.python.org/pep-0632/
(13) https://peps.python.org/pep-0623/
(14) https://peps.python.org/pep-0624/
(15) https://peps.python.org/pep-0597/
File: python.info, Node: New Features<7>, Next: New Features Related to Type Hints<3>, Prev: Summary – Release highlights<3>, Up: What’s New In Python 3 10
1.4.2 New Features
------------------
* Menu:
* Parenthesized context managers::
* Better error messages::
* PEP 626; Precise line numbers for debugging and other tools: PEP 626 Precise line numbers for debugging and other tools.
* PEP 634; Structural Pattern Matching: PEP 634 Structural Pattern Matching.
* Optional EncodingWarning and encoding="locale" option::
File: python.info, Node: Parenthesized context managers, Next: Better error messages, Up: New Features<7>
1.4.2.1 Parenthesized context managers
......................................
Using enclosing parentheses for continuation across multiple lines in
context managers is now supported. This allows formatting a long
collection of context managers in multiple lines in a similar way as it
was previously possible with import statements. For instance, all these
examples are now valid:
with (CtxManager() as example):
...
with (
CtxManager1(),
CtxManager2()
):
...
with (CtxManager1() as example,
CtxManager2()):
...
with (CtxManager1(),
CtxManager2() as example):
...
with (
CtxManager1() as example1,
CtxManager2() as example2
):
...
it is also possible to use a trailing comma at the end of the enclosed
group:
with (
CtxManager1() as example1,
CtxManager2() as example2,
CtxManager3() as example3,
):
...
This new syntax uses the non LL(1) capacities of the new parser. Check
PEP 617(1) for more details.
(Contributed by Guido van Rossum, Pablo Galindo and Lysandros Nikolaou
in bpo-12782(2) and bpo-40334(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0617/
(2) https://bugs.python.org/issue?@action=redirect&bpo=12782
(3) https://bugs.python.org/issue?@action=redirect&bpo=40334
File: python.info, Node: Better error messages, Next: PEP 626 Precise line numbers for debugging and other tools, Prev: Parenthesized context managers, Up: New Features<7>
1.4.2.2 Better error messages
.............................
* Menu:
* SyntaxErrors::
* IndentationErrors::
* AttributeErrors::
* NameErrors::
File: python.info, Node: SyntaxErrors, Next: IndentationErrors, Up: Better error messages
1.4.2.3 SyntaxErrors
....................
When parsing code that contains unclosed parentheses or brackets the
interpreter now includes the location of the unclosed bracket of
parentheses instead of displaying 'SyntaxError: unexpected EOF while
parsing' or pointing to some incorrect location. For instance, consider
the following code (notice the unclosed ‘{‘):
expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4,
38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6,
some_other_code = foo()
Previous versions of the interpreter reported confusing places as the
location of the syntax error:
File "example.py", line 3
some_other_code = foo()
^
SyntaxError: invalid syntax
but in Python 3.10 a more informative error is emitted:
File "example.py", line 1
expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4,
^
SyntaxError: '{' was never closed
In a similar way, errors involving unclosed string literals (single and
triple quoted) now point to the start of the string instead of reporting
EOF/EOL.
These improvements are inspired by previous work in the PyPy
interpreter.
(Contributed by Pablo Galindo in bpo-42864(1) and Batuhan Taskaya in
bpo-40176(2).)
*note SyntaxError: 18d. exceptions raised by the interpreter will now
highlight the full error range of the expression that constitutes the
syntax error itself, instead of just where the problem is detected. In
this way, instead of displaying (before Python 3.10):
>>> foo(x, z for z in range(10), t, w)
File "", line 1
foo(x, z for z in range(10), t, w)
^
SyntaxError: Generator expression must be parenthesized
now Python 3.10 will display the exception as:
>>> foo(x, z for z in range(10), t, w)
File "", line 1
foo(x, z for z in range(10), t, w)
^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized
This improvement was contributed by Pablo Galindo in bpo-43914(3).
A considerable amount of new specialized messages for *note SyntaxError:
18d. exceptions have been incorporated. Some of the most notable ones
are as follows:
* Missing ‘:’ before blocks:
>>> if rocket.position > event_horizon
File "", line 1
if rocket.position > event_horizon
^
SyntaxError: expected ':'
(Contributed by Pablo Galindo in bpo-42997(4).)
* Unparenthesised tuples in comprehensions targets:
>>> {x,y for x,y in zip('abcd', '1234')}
File "", line 1
{x,y for x,y in zip('abcd', '1234')}
^
SyntaxError: did you forget parentheses around the comprehension target?
(Contributed by Pablo Galindo in bpo-43017(5).)
* Missing commas in collection literals and between expressions:
>>> items = {
... x: 1,
... y: 2
... z: 3,
File "", line 3
y: 2
^
SyntaxError: invalid syntax. Perhaps you forgot a comma?
(Contributed by Pablo Galindo in bpo-43822(6).)
* Multiple Exception types without parentheses:
>>> try:
... build_dyson_sphere()
... except NotEnoughScienceError, NotEnoughResourcesError:
File "", line 3
except NotEnoughScienceError, NotEnoughResourcesError:
^
SyntaxError: multiple exception types must be parenthesized
(Contributed by Pablo Galindo in bpo-43149(7).)
* Missing ‘:’ and values in dictionary literals:
>>> values = {
... x: 1,
... y: 2,
... z:
... }
File "", line 4
z:
^
SyntaxError: expression expected after dictionary key and ':'
>>> values = {x:1, y:2, z w:3}
File "", line 1
values = {x:1, y:2, z w:3}
^
SyntaxError: ':' expected after dictionary key
(Contributed by Pablo Galindo in bpo-43823(8).)
* ‘try’ blocks without ‘except’ or ‘finally’ blocks:
>>> try:
... x = 2
... something = 3
File "", line 3
something = 3
^^^^^^^^^
SyntaxError: expected 'except' or 'finally' block
(Contributed by Pablo Galindo in bpo-44305(9).)
* Usage of ‘=’ instead of ‘==’ in comparisons:
>>> if rocket.position = event_horizon:
File "", line 1
if rocket.position = event_horizon:
^
SyntaxError: cannot assign to attribute here. Maybe you meant '==' instead of '='?
(Contributed by Pablo Galindo in bpo-43797(10).)
* Usage of ‘*’ in f-strings:
>>> f"Black holes {*all_black_holes} and revelations"
File "", line 1
(*all_black_holes)
^
SyntaxError: f-string: cannot use starred expression here
(Contributed by Pablo Galindo in bpo-41064(11).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42864
(2) https://bugs.python.org/issue?@action=redirect&bpo=40176
(3) https://bugs.python.org/issue?@action=redirect&bpo=43914
(4) https://bugs.python.org/issue?@action=redirect&bpo=42997
(5) https://bugs.python.org/issue?@action=redirect&bpo=43017
(6) https://bugs.python.org/issue?@action=redirect&bpo=43822
(7) https://bugs.python.org/issue?@action=redirect&bpo=43149
(8) https://bugs.python.org/issue?@action=redirect&bpo=43823
(9) https://bugs.python.org/issue?@action=redirect&bpo=44305
(10) https://bugs.python.org/issue?@action=redirect&bpo=43797
(11) https://bugs.python.org/issue?@action=redirect&bpo=41064
File: python.info, Node: IndentationErrors, Next: AttributeErrors, Prev: SyntaxErrors, Up: Better error messages
1.4.2.4 IndentationErrors
.........................
Many *note IndentationError: 7a3. exceptions now have more context
regarding what kind of block was expecting an indentation, including the
location of the statement:
>>> def foo():
... if lel:
... x = 2
File "", line 3
x = 2
^
IndentationError: expected an indented block after 'if' statement in line 2
File: python.info, Node: AttributeErrors, Next: NameErrors, Prev: IndentationErrors, Up: Better error messages
1.4.2.5 AttributeErrors
.......................
When printing *note AttributeError: 348, ‘PyErr_Display()’ will offer
suggestions of similar attribute names in the object that the exception
was raised from:
>>> collections.namedtoplo
Traceback (most recent call last):
File "", line 1, in
AttributeError: module 'collections' has no attribute 'namedtoplo'. Did you mean: namedtuple?
(Contributed by Pablo Galindo in bpo-38530(1).)
Warning: Notice this won’t work if ‘PyErr_Display()’ is not called
to display the error which can happen if some other custom error
display function is used. This is a common scenario in some REPLs
like IPython.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38530
File: python.info, Node: NameErrors, Prev: AttributeErrors, Up: Better error messages
1.4.2.6 NameErrors
..................
When printing *note NameError: 43a. raised by the interpreter,
‘PyErr_Display()’ will offer suggestions of similar variable names in
the function that the exception was raised from:
>>> schwarzschild_black_hole = None
>>> schwarschild_black_hole
Traceback (most recent call last):
File "", line 1, in
NameError: name 'schwarschild_black_hole' is not defined. Did you mean: schwarzschild_black_hole?
(Contributed by Pablo Galindo in bpo-38530(1).)
Warning: Notice this won’t work if ‘PyErr_Display()’ is not called
to display the error, which can happen if some other custom error
display function is used. This is a common scenario in some REPLs
like IPython.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38530
File: python.info, Node: PEP 626 Precise line numbers for debugging and other tools, Next: PEP 634 Structural Pattern Matching, Prev: Better error messages, Up: New Features<7>
1.4.2.7 PEP 626: Precise line numbers for debugging and other tools
...................................................................
PEP 626 brings more precise and reliable line numbers for debugging,
profiling and coverage tools. Tracing events, with the correct line
number, are generated for all lines of code executed and only for lines
of code that are executed.
The *note f_lineno: 7a7. attribute of frame objects will always contain
the expected line number.
The *note co_lnotab: 2e4. attribute of *note code objects: 1d2. is
deprecated and will be removed in 3.12. Code that needs to convert from
offset to line number should use the new *note co_lines(): 2f5. method
instead.
File: python.info, Node: PEP 634 Structural Pattern Matching, Next: Optional EncodingWarning and encoding="locale" option, Prev: PEP 626 Precise line numbers for debugging and other tools, Up: New Features<7>
1.4.2.8 PEP 634: Structural Pattern Matching
............................................
Structural pattern matching has been added in the form of a 'match
statement' and 'case statements' of patterns with associated actions.
Patterns consist of sequences, mappings, primitive data types as well as
class instances. Pattern matching enables programs to extract
information from complex data types, branch on the structure of data,
and apply specific actions based on different forms of data.
* Menu:
* Syntax and operations::
* Declarative approach::
* Simple pattern; match to a literal: Simple pattern match to a literal.
* Patterns with a literal and variable::
* Patterns and classes::
* Nested patterns::
* Complex patterns and the wildcard::
* Guard::
* Other Key Features::
File: python.info, Node: Syntax and operations, Next: Declarative approach, Up: PEP 634 Structural Pattern Matching
1.4.2.9 Syntax and operations
.............................
The generic syntax of pattern matching is:
match subject:
case :
case :
case :
case _:
A match statement takes an expression and compares its value to
successive patterns given as one or more case blocks. Specifically,
pattern matching operates by:
1. using data with type and shape (the ‘subject’)
2. evaluating the ‘subject’ in the ‘match’ statement
3. comparing the subject with each pattern in a ‘case’ statement from
top to bottom until a match is confirmed.
4. executing the action associated with the pattern of the confirmed
match
5. If an exact match is not confirmed, the last case, a wildcard ‘_’,
if provided, will be used as the matching case. If an exact match
is not confirmed and a wildcard case does not exist, the entire
match block is a no-op.
File: python.info, Node: Declarative approach, Next: Simple pattern match to a literal, Prev: Syntax and operations, Up: PEP 634 Structural Pattern Matching
1.4.2.10 Declarative approach
.............................
Readers may be aware of pattern matching through the simple example of
matching a subject (data object) to a literal (pattern) with the switch
statement found in C, Java or JavaScript (and many other languages).
Often the switch statement is used for comparison of an
object/expression with case statements containing literals.
More powerful examples of pattern matching can be found in languages
such as Scala and Elixir. With structural pattern matching, the
approach is “declarative” and explicitly states the conditions (the
patterns) for data to match.
While an “imperative” series of instructions using nested “if”
statements could be used to accomplish something similar to structural
pattern matching, it is less clear than the “declarative” approach.
Instead the “declarative” approach states the conditions to meet for a
match and is more readable through its explicit patterns. While
structural pattern matching can be used in its simplest form comparing a
variable to a literal in a case statement, its true value for Python
lies in its handling of the subject’s type and shape.
File: python.info, Node: Simple pattern match to a literal, Next: Patterns with a literal and variable, Prev: Declarative approach, Up: PEP 634 Structural Pattern Matching
1.4.2.11 Simple pattern: match to a literal
...........................................
Let’s look at this example as pattern matching in its simplest form: a
value, the subject, being matched to several literals, the patterns. In
the example below, ‘status’ is the subject of the match statement. The
patterns are each of the case statements, where literals represent
request status codes. The associated action to the case is executed
after a match:
def http_error(status):
match status:
case 400:
return "Bad request"
case 404:
return "Not found"
case 418:
return "I'm a teapot"
case _:
return "Something's wrong with the internet"
If the above function is passed a ‘status’ of 418, “I’m a teapot” is
returned. If the above function is passed a ‘status’ of 500, the case
statement with ‘_’ will match as a wildcard, and “Something’s wrong with
the internet” is returned. Note the last block: the variable name, ‘_’,
acts as a 'wildcard' and insures the subject will always match. The use
of ‘_’ is optional.
You can combine several literals in a single pattern using ‘|’ (“or”):
case 401 | 403 | 404:
return "Not allowed"
* Menu:
* Behavior without the wildcard::
File: python.info, Node: Behavior without the wildcard, Up: Simple pattern match to a literal
1.4.2.12 Behavior without the wildcard
......................................
If we modify the above example by removing the last case block, the
example becomes:
def http_error(status):
match status:
case 400:
return "Bad request"
case 404:
return "Not found"
case 418:
return "I'm a teapot"
Without the use of ‘_’ in a case statement, a match may not exist. If
no match exists, the behavior is a no-op. For example, if ‘status’ of
500 is passed, a no-op occurs.
File: python.info, Node: Patterns with a literal and variable, Next: Patterns and classes, Prev: Simple pattern match to a literal, Up: PEP 634 Structural Pattern Matching
1.4.2.13 Patterns with a literal and variable
.............................................
Patterns can look like unpacking assignments, and a pattern may be used
to bind variables. In this example, a data point can be unpacked to its
x-coordinate and y-coordinate:
# point is an (x, y) tuple
match point:
case (0, 0):
print("Origin")
case (0, y):
print(f"Y={y}")
case (x, 0):
print(f"X={x}")
case (x, y):
print(f"X={x}, Y={y}")
case _:
raise ValueError("Not a point")
The first pattern has two literals, ‘(0, 0)’, and may be thought of as
an extension of the literal pattern shown above. The next two patterns
combine a literal and a variable, and the variable 'binds' a value from
the subject (‘point’). The fourth pattern captures two values, which
makes it conceptually similar to the unpacking assignment ‘(x, y) =
point’.
File: python.info, Node: Patterns and classes, Next: Nested patterns, Prev: Patterns with a literal and variable, Up: PEP 634 Structural Pattern Matching
1.4.2.14 Patterns and classes
.............................
If you are using classes to structure your data, you can use as a
pattern the class name followed by an argument list resembling a
constructor. This pattern has the ability to capture instance
attributes into variables:
class Point:
def __init__(self, x, y):
self.x = x
self.y = y
def location(point):
match point:
case Point(x=0, y=0):
print("Origin is the point's location.")
case Point(x=0, y=y):
print(f"Y={y} and the point is on the y-axis.")
case Point(x=x, y=0):
print(f"X={x} and the point is on the x-axis.")
case Point():
print("The point is located somewhere else on the plane.")
case _:
print("Not a point")
* Menu:
* Patterns with positional parameters::
File: python.info, Node: Patterns with positional parameters, Up: Patterns and classes
1.4.2.15 Patterns with positional parameters
............................................
You can use positional parameters with some builtin classes that provide
an ordering for their attributes (e.g. dataclasses). You can also
define a specific position for attributes in patterns by setting the
‘__match_args__’ special attribute in your classes. If it’s set to
(“x”, “y”), the following patterns are all equivalent (and all bind the
‘y’ attribute to the ‘var’ variable):
Point(1, var)
Point(1, y=var)
Point(x=1, y=var)
Point(y=var, x=1)
File: python.info, Node: Nested patterns, Next: Complex patterns and the wildcard, Prev: Patterns and classes, Up: PEP 634 Structural Pattern Matching
1.4.2.16 Nested patterns
........................
Patterns can be arbitrarily nested. For example, if our data is a short
list of points, it could be matched like this:
match points:
case []:
print("No points in the list.")
case [Point(0, 0)]:
print("The origin is the only point in the list.")
case [Point(x, y)]:
print(f"A single point {x}, {y} is in the list.")
case [Point(0, y1), Point(0, y2)]:
print(f"Two points on the Y axis at {y1}, {y2} are in the list.")
case _:
print("Something else is found in the list.")
File: python.info, Node: Complex patterns and the wildcard, Next: Guard, Prev: Nested patterns, Up: PEP 634 Structural Pattern Matching
1.4.2.17 Complex patterns and the wildcard
..........................................
To this point, the examples have used ‘_’ alone in the last case
statement. A wildcard can be used in more complex patterns, such as
‘('error', code, _)’. For example:
match test_variable:
case ('warning', code, 40):
print("A warning has been received.")
case ('error', code, _):
print(f"An error {code} occurred.")
In the above case, ‘test_variable’ will match for (‘error’, code, 100)
and (‘error’, code, 800).
File: python.info, Node: Guard, Next: Other Key Features, Prev: Complex patterns and the wildcard, Up: PEP 634 Structural Pattern Matching
1.4.2.18 Guard
..............
We can add an ‘if’ clause to a pattern, known as a “guard”. If the
guard is false, ‘match’ goes on to try the next case block. Note that
value capture happens before the guard is evaluated:
match point:
case Point(x, y) if x == y:
print(f"The point is located on the diagonal Y=X at {x}.")
case Point(x, y):
print(f"Point is not on the diagonal.")
File: python.info, Node: Other Key Features, Prev: Guard, Up: PEP 634 Structural Pattern Matching
1.4.2.19 Other Key Features
...........................
Several other key features:
- Like unpacking assignments, tuple and list patterns have exactly
the same meaning and actually match arbitrary sequences.
Technically, the subject must be a sequence. Therefore, an
important exception is that patterns don’t match iterators. Also,
to prevent a common mistake, sequence patterns don’t match strings.
- Sequence patterns support wildcards: ‘[x, y, *rest]’ and ‘(x, y,
*rest)’ work similar to wildcards in unpacking assignments. The
name after ‘*’ may also be ‘_’, so ‘(x, y, *_)’ matches a sequence
of at least two items without binding the remaining items.
- Mapping patterns: ‘{"bandwidth": b, "latency": l}’ captures the
‘"bandwidth"’ and ‘"latency"’ values from a dict. Unlike sequence
patterns, extra keys are ignored. A wildcard ‘**rest’ is also
supported. (But ‘**_’ would be redundant, so is not allowed.)
- Subpatterns may be captured using the ‘as’ keyword:
case (Point(x1, y1), Point(x2, y2) as p2): ...
This binds x1, y1, x2, y2 like you would expect without the ‘as’
clause, and p2 to the entire second item of the subject.
- Most literals are compared by equality. However, the singletons
‘True’, ‘False’ and ‘None’ are compared by identity.
- Named constants may be used in patterns. These named constants
must be dotted names to prevent the constant from being interpreted
as a capture variable:
from enum import Enum
class Color(Enum):
RED = 0
GREEN = 1
BLUE = 2
color = Color.GREEN
match color:
case Color.RED:
print("I see red!")
case Color.GREEN:
print("Grass is green")
case Color.BLUE:
print("I'm feeling the blues :(")
For the full specification see PEP 634(1). Motivation and rationale are
in PEP 635(2), and a longer tutorial is in PEP 636(3).
---------- Footnotes ----------
(1) https://peps.python.org/pep-0634/
(2) https://peps.python.org/pep-0635/
(3) https://peps.python.org/pep-0636/
File: python.info, Node: Optional EncodingWarning and encoding="locale" option, Prev: PEP 634 Structural Pattern Matching, Up: New Features<7>
1.4.2.20 Optional ‘EncodingWarning’ and ‘encoding="locale"’ option
..................................................................
The default encoding of *note TextIOWrapper: 7b6. and *note open(): 517.
is platform and locale dependent. Since UTF-8 is used on most Unix
platforms, omitting ‘encoding’ option when opening UTF-8 files (e.g.
JSON, YAML, TOML, Markdown) is a very common bug. For example:
# BUG: "rb" mode or encoding="utf-8" should be used.
with open("data.json") as f:
data = json.load(f)
To find this type of bug, an optional ‘EncodingWarning’ is added. It is
emitted when *note sys.flags.warn_default_encoding: 688. is true and
locale-specific default encoding is used.
‘-X warn_default_encoding’ option and *note PYTHONWARNDEFAULTENCODING:
7b7. are added to enable the warning.
See *note Text Encoding: 7b8. for more information.
File: python.info, Node: New Features Related to Type Hints<3>, Next: Other Language Changes<4>, Prev: New Features<7>, Up: What’s New In Python 3 10
1.4.3 New Features Related to Type Hints
----------------------------------------
This section covers major changes affecting PEP 484(1) type hints and
the *note typing: 104. module.
* Menu:
* PEP 604; New Type Union Operator: PEP 604 New Type Union Operator.
* PEP 612; Parameter Specification Variables: PEP 612 Parameter Specification Variables.
* PEP 613; TypeAlias: PEP 613 TypeAlias.
* PEP 647; User-Defined Type Guards: PEP 647 User-Defined Type Guards.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
File: python.info, Node: PEP 604 New Type Union Operator, Next: PEP 612 Parameter Specification Variables, Up: New Features Related to Type Hints<3>
1.4.3.1 PEP 604: New Type Union Operator
........................................
A new type union operator was introduced which enables the syntax ‘X |
Y’. This provides a cleaner way of expressing ‘either type X or type Y’
instead of using *note typing.Union: 637, especially in type hints.
In previous versions of Python, to apply a type hint for functions
accepting arguments of multiple types, *note typing.Union: 637. was
used:
def square(number: Union[int, float]) -> Union[int, float]:
return number ** 2
Type hints can now be written in a more succinct manner:
def square(number: int | float) -> int | float:
return number ** 2
This new syntax is also accepted as the second argument to *note
isinstance(): 43d. and *note issubclass(): 7bc.:
>>> isinstance(1, int | str)
True
See *note Union Type: 7bd. and PEP 604(1) for more details.
(Contributed by Maggie Moss and Philippe Prados in bpo-41428(2), with
additions by Yurii Karabas and Serhiy Storchaka in bpo-44490(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0604/
(2) https://bugs.python.org/issue?@action=redirect&bpo=41428
(3) https://bugs.python.org/issue?@action=redirect&bpo=44490
File: python.info, Node: PEP 612 Parameter Specification Variables, Next: PEP 613 TypeAlias, Prev: PEP 604 New Type Union Operator, Up: New Features Related to Type Hints<3>
1.4.3.2 PEP 612: Parameter Specification Variables
..................................................
Two new options to improve the information provided to static type
checkers for PEP 484(1)‘s ‘Callable’ have been added to the *note
typing: 104. module.
The first is the parameter specification variable. They are used to
forward the parameter types of one callable to another callable – a
pattern commonly found in higher order functions and decorators.
Examples of usage can be found in *note typing.ParamSpec: 15e.
Previously, there was no easy way to type annotate dependency of
parameter types in such a precise manner.
The second option is the new ‘Concatenate’ operator. It’s used in
conjunction with parameter specification variables to type annotate a
higher order callable which adds or removes parameters of another
callable. Examples of usage can be found in *note typing.Concatenate:
7bf.
See *note typing.Callable: 7c0, *note typing.ParamSpec: 15e, *note
typing.Concatenate: 7bf, *note typing.ParamSpecArgs: 7c1, *note
typing.ParamSpecKwargs: 7c2, and PEP 612(2) for more details.
(Contributed by Ken Jin in bpo-41559(3), with minor enhancements by
Jelle Zijlstra in bpo-43783(4). PEP written by Mark Mendoza.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0612/
(3) https://bugs.python.org/issue?@action=redirect&bpo=41559
(4) https://bugs.python.org/issue?@action=redirect&bpo=43783
File: python.info, Node: PEP 613 TypeAlias, Next: PEP 647 User-Defined Type Guards, Prev: PEP 612 Parameter Specification Variables, Up: New Features Related to Type Hints<3>
1.4.3.3 PEP 613: TypeAlias
..........................
PEP 484(1) introduced the concept of type aliases, only requiring them
to be top-level unannotated assignments. This simplicity sometimes made
it difficult for type checkers to distinguish between type aliases and
ordinary assignments, especially when forward references or invalid
types were involved. Compare:
StrCache = 'Cache[str]' # a type alias
LOG_PREFIX = 'LOG[DEBUG]' # a module constant
Now the *note typing: 104. module has a special value *note TypeAlias:
7c4. which lets you declare type aliases more explicitly:
StrCache: TypeAlias = 'Cache[str]' # a type alias
LOG_PREFIX = 'LOG[DEBUG]' # a module constant
See PEP 613(2) for more details.
(Contributed by Mikhail Golubev in bpo-41923(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0613/
(3) https://bugs.python.org/issue?@action=redirect&bpo=41923
File: python.info, Node: PEP 647 User-Defined Type Guards, Prev: PEP 613 TypeAlias, Up: New Features Related to Type Hints<3>
1.4.3.4 PEP 647: User-Defined Type Guards
.........................................
*note TypeGuard: 164. has been added to the *note typing: 104. module to
annotate type guard functions and improve information provided to static
type checkers during type narrowing. For more information, please see
*note TypeGuard: 164.‘s documentation, and PEP 647(1).
(Contributed by Ken Jin and Guido van Rossum in bpo-43766(2). PEP
written by Eric Traut.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0647/
(2) https://bugs.python.org/issue?@action=redirect&bpo=43766
File: python.info, Node: Other Language Changes<4>, Next: New Modules<4>, Prev: New Features Related to Type Hints<3>, Up: What’s New In Python 3 10
1.4.4 Other Language Changes
----------------------------
* The *note int: 259. type has a new method *note int.bit_count():
7c7, returning the number of ones in the binary expansion of a
given integer, also known as the population count. (Contributed by
Niklas Fiekas in bpo-29882(1).)
* The views returned by *note dict.keys(): 7c8, *note dict.values():
7c9. and *note dict.items(): 7ca. now all have a ‘mapping’
attribute that gives a *note types.MappingProxyType: 469. object
wrapping the original dictionary. (Contributed by Dennis Sweeney
in bpo-40890(2).)
* PEP 618(3): The *note zip(): 7cb. function now has an optional
‘strict’ flag, used to require that all the iterables have an equal
length.
* Builtin and extension functions that take integer arguments no
longer accept *note Decimal: 29f.s, *note Fraction: 1e9.s and other
objects that can be converted to integers only with a loss (e.g.
that have the *note __int__(): 717. method but do not have the
*note __index__(): 718. method). (Contributed by Serhiy Storchaka
in bpo-37999(4).)
* If *note object.__ipow__(): 7cc. returns *note NotImplemented: 7cd,
the operator will correctly fall back to *note object.__pow__():
7ce. and *note object.__rpow__(): 7cf. as expected. (Contributed
by Alex Shkop in bpo-38302(5).)
* Assignment expressions can now be used unparenthesized within set
literals and set comprehensions, as well as in sequence indexes
(but not slices).
* Functions have a new ‘__builtins__’ attribute which is used to look
for builtin symbols when a function is executed, instead of looking
into ‘__globals__['__builtins__']’. The attribute is initialized
from ‘__globals__["__builtins__"]’ if it exists, else from the
current builtins. (Contributed by Mark Shannon in bpo-42990(6).)
* Two new builtin functions – *note aiter(): 7d0. and *note anext():
7d1. have been added to provide asynchronous counterparts to *note
iter(): 7d2. and *note next(): 7d3, respectively. (Contributed by
Joshua Bronson, Daniel Pope, and Justin Wang in bpo-31861(7).)
* Static methods (*note @staticmethod: 412.) and class methods (*note
@classmethod: 166.) now inherit the method attributes
(‘__module__’, ‘__name__’, ‘__qualname__’, ‘__doc__’,
‘__annotations__’) and have a new ‘__wrapped__’ attribute.
Moreover, static methods are now callable as regular functions.
(Contributed by Victor Stinner in bpo-43682(8).)
* Annotations for complex targets (everything beside ‘simple name’
targets defined by PEP 526(9)) no longer cause any runtime effects
with ‘from __future__ import annotations’. (Contributed by Batuhan
Taskaya in bpo-42737(10).)
* Class and module objects now lazy-create empty annotations dicts on
demand. The annotations dicts are stored in the object’s
‘__dict__’ for backwards compatibility. This improves the best
practices for working with ‘__annotations__’; for more information,
please see *note Annotations Best Practices: 7d4. (Contributed by
Larry Hastings in bpo-43901(11).)
* Annotations consist of ‘yield’, ‘yield from’, ‘await’ or named
expressions are now forbidden under ‘from __future__ import
annotations’ due to their side effects. (Contributed by Batuhan
Taskaya in bpo-42725(12).)
* Usage of unbound variables, ‘super()’ and other expressions that
might alter the processing of symbol table as annotations are now
rendered effectless under ‘from __future__ import annotations’.
(Contributed by Batuhan Taskaya in bpo-42725(13).)
* Hashes of NaN values of both *note float: 2f1. type and *note
decimal.Decimal: 29f. type now depend on object identity.
Formerly, they always hashed to ‘0’ even though NaN values are not
equal to one another. This caused potentially quadratic runtime
behavior due to excessive hash collisions when creating
dictionaries and sets containing multiple NaNs. (Contributed by
Raymond Hettinger in bpo-43475(14).)
* A *note SyntaxError: 18d. (instead of a *note NameError: 43a.) will
be raised when deleting the *note __debug__: 7d5. constant.
(Contributed by Donghee Na in bpo-45000(15).)
* *note SyntaxError: 18d. exceptions now have ‘end_lineno’ and
‘end_offset’ attributes. They will be ‘None’ if not determined.
(Contributed by Pablo Galindo in bpo-43914(16).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=29882
(2) https://bugs.python.org/issue?@action=redirect&bpo=40890
(3) https://peps.python.org/pep-0618/
(4) https://bugs.python.org/issue?@action=redirect&bpo=37999
(5) https://bugs.python.org/issue?@action=redirect&bpo=38302
(6) https://bugs.python.org/issue?@action=redirect&bpo=42990
(7) https://bugs.python.org/issue?@action=redirect&bpo=31861
(8) https://bugs.python.org/issue?@action=redirect&bpo=43682
(9) https://peps.python.org/pep-0526/
(10) https://bugs.python.org/issue?@action=redirect&bpo=42737
(11) https://bugs.python.org/issue?@action=redirect&bpo=43901
(12) https://bugs.python.org/issue?@action=redirect&bpo=42725
(13) https://bugs.python.org/issue?@action=redirect&bpo=42725
(14) https://bugs.python.org/issue?@action=redirect&bpo=43475
(15) https://bugs.python.org/issue?@action=redirect&bpo=45000
(16) https://bugs.python.org/issue?@action=redirect&bpo=43914
File: python.info, Node: New Modules<4>, Next: Improved Modules<4>, Prev: Other Language Changes<4>, Up: What’s New In Python 3 10
1.4.5 New Modules
-----------------
* None.
File: python.info, Node: Improved Modules<4>, Next: Optimizations<4>, Prev: New Modules<4>, Up: What’s New In Python 3 10
1.4.6 Improved Modules
----------------------
* Menu:
* asyncio: asyncio<4>.
* argparse: argparse<2>.
* array: array<3>.
* asynchat, asyncore, smtpd: asynchat asyncore smtpd.
* base64: base64<2>.
* bdb::
* bisect::
* codecs::
* collections.abc: collections abc.
* contextlib: contextlib<2>.
* curses::
* dataclasses: dataclasses<2>.
* distutils: distutils<2>.
* doctest: doctest<2>.
* encodings::
* enum: enum<4>.
* fileinput::
* faulthandler::
* gc::
* glob: glob<2>.
* hashlib: hashlib<3>.
* hmac::
* IDLE and idlelib: IDLE and idlelib<2>.
* importlib.metadata: importlib metadata<2>.
* inspect: inspect<3>.
* itertools: itertools<3>.
* linecache::
* os: os<4>.
* os.path: os path<4>.
* pathlib: pathlib<5>.
* platform: platform<2>.
* pprint::
* py_compile::
* pyclbr::
* shelve::
* statistics: statistics<3>.
* site: site<2>.
* socket: socket<2>.
* ssl: ssl<3>.
* sqlite3: sqlite3<5>.
* sys: sys<5>.
* _thread::
* threading: threading<3>.
* traceback: traceback<3>.
* types: types<3>.
* typing: typing<5>.
* unittest: unittest<5>.
* urllib.parse: urllib parse.
* xml: xml<2>.
* zipimport: zipimport<3>.
File: python.info, Node: asyncio<4>, Next: argparse<2>, Up: Improved Modules<4>
1.4.6.1 asyncio
...............
Add missing ‘connect_accepted_socket()’ method. (Contributed by Alex
Grönholm in bpo-41332(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41332
File: python.info, Node: argparse<2>, Next: array<3>, Prev: asyncio<4>, Up: Improved Modules<4>
1.4.6.2 argparse
................
Misleading phrase “optional arguments” was replaced with “options” in
argparse help. Some tests might require adaptation if they rely on
exact output match. (Contributed by Raymond Hettinger in bpo-9694(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9694
File: python.info, Node: array<3>, Next: asynchat asyncore smtpd, Prev: argparse<2>, Up: Improved Modules<4>
1.4.6.3 array
.............
The *note index(): 7db. method of *note array.array: 1a0. now has
optional 'start' and 'stop' parameters. (Contributed by Anders
Lorentsen and Zackery Spytz in bpo-31956(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31956
File: python.info, Node: asynchat asyncore smtpd, Next: base64<2>, Prev: array<3>, Up: Improved Modules<4>
1.4.6.4 asynchat, asyncore, smtpd
.................................
These modules have been marked as deprecated in their module
documentation since Python 3.6. An import-time *note
DeprecationWarning: 1a5. has now been added to all three of these
modules.
File: python.info, Node: base64<2>, Next: bdb, Prev: asynchat asyncore smtpd, Up: Improved Modules<4>
1.4.6.5 base64
..............
Add *note base64.b32hexencode(): 7de. and *note base64.b32hexdecode():
7df. to support the Base32 Encoding with Extended Hex Alphabet.
File: python.info, Node: bdb, Next: bisect, Prev: base64<2>, Up: Improved Modules<4>
1.4.6.6 bdb
...........
Add ‘clearBreakpoints()’ to reset all set breakpoints. (Contributed by
Irit Katriel in bpo-24160(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24160
File: python.info, Node: bisect, Next: codecs, Prev: bdb, Up: Improved Modules<4>
1.4.6.7 bisect
..............
Added the possibility of providing a 'key' function to the APIs in the
*note bisect: 11. module. (Contributed by Raymond Hettinger in
bpo-4356(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4356
File: python.info, Node: codecs, Next: collections abc, Prev: bisect, Up: Improved Modules<4>
1.4.6.8 codecs
..............
Add a *note codecs.unregister(): 7e3. function to unregister a codec
search function. (Contributed by Hai Shi in bpo-41842(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41842
File: python.info, Node: collections abc, Next: contextlib<2>, Prev: codecs, Up: Improved Modules<4>
1.4.6.9 collections.abc
.......................
The ‘__args__’ of the *note parameterized generic: 6ae. for *note
collections.abc.Callable: 7e5. are now consistent with *note
typing.Callable: 7c0. *note collections.abc.Callable: 7e5. generic now
flattens type parameters, similar to what *note typing.Callable: 7c0.
currently does. This means that ‘collections.abc.Callable[[int, str],
str]’ will have ‘__args__’ of ‘(int, str, str)’; previously this was
‘([int, str], str)’. To allow this change, *note types.GenericAlias:
7e6. can now be subclassed, and a subclass will be returned when
subscripting the *note collections.abc.Callable: 7e5. type. Note that a
*note TypeError: 534. may be raised for invalid forms of parameterizing
*note collections.abc.Callable: 7e5. which may have passed silently in
Python 3.9. (Contributed by Ken Jin in bpo-42195(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42195
File: python.info, Node: contextlib<2>, Next: curses, Prev: collections abc, Up: Improved Modules<4>
1.4.6.10 contextlib
...................
Add a *note contextlib.aclosing(): 7e8. context manager to safely close
async generators and objects representing asynchronously released
resources. (Contributed by Joongi Kim and John Belmonte in
bpo-41229(1).)
Add asynchronous context manager support to *note
contextlib.nullcontext(): 7e9. (Contributed by Tom Gringauz in
bpo-41543(2).)
Add *note AsyncContextDecorator: 7ea, for supporting usage of async
context managers as decorators.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41229
(2) https://bugs.python.org/issue?@action=redirect&bpo=41543
File: python.info, Node: curses, Next: dataclasses<2>, Prev: contextlib<2>, Up: Improved Modules<4>
1.4.6.11 curses
...............
The extended color functions added in ncurses 6.1 will be used
transparently by *note curses.color_content(): 7ec, *note
curses.init_color(): 7ed, *note curses.init_pair(): 7ee, and *note
curses.pair_content(): 7ef. A new function, *note
curses.has_extended_color_support(): 7f0, indicates whether extended
color support is provided by the underlying ncurses library.
(Contributed by Jeffrey Kintscher and Hans Petter Jansson in
bpo-36982(1).)
The ‘BUTTON5_*’ constants are now exposed in the *note curses: 2b.
module if they are provided by the underlying curses library.
(Contributed by Zackery Spytz in bpo-39273(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36982
(2) https://bugs.python.org/issue?@action=redirect&bpo=39273
File: python.info, Node: dataclasses<2>, Next: distutils<2>, Prev: curses, Up: Improved Modules<4>
1.4.6.12 dataclasses
....................
* Menu:
* __slots__::
* Keyword-only fields::
File: python.info, Node: __slots__, Next: Keyword-only fields, Up: dataclasses<2>
1.4.6.13 __slots__
..................
Added ‘slots’ parameter in *note dataclasses.dataclass(): 1cb.
decorator. (Contributed by Yurii Karabas in bpo-42269(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42269
File: python.info, Node: Keyword-only fields, Prev: __slots__, Up: dataclasses<2>
1.4.6.14 Keyword-only fields
............................
dataclasses now supports fields that are keyword-only in the generated
__init__ method. There are a number of ways of specifying keyword-only
fields.
You can say that every field is keyword-only:
from dataclasses import dataclass
@dataclass(kw_only=True)
class Birthday:
name: str
birthday: datetime.date
Both ‘name’ and ‘birthday’ are keyword-only parameters to the generated
__init__ method.
You can specify keyword-only on a per-field basis:
from dataclasses import dataclass, field
@dataclass
class Birthday:
name: str
birthday: datetime.date = field(kw_only=True)
Here only ‘birthday’ is keyword-only. If you set ‘kw_only’ on
individual fields, be aware that there are rules about re-ordering
fields due to keyword-only fields needing to follow non-keyword-only
fields. See the full dataclasses documentation for details.
You can also specify that all fields following a KW_ONLY marker are
keyword-only. This will probably be the most common usage:
from dataclasses import dataclass, KW_ONLY
@dataclass
class Point:
x: float
y: float
_: KW_ONLY
z: float = 0.0
t: float = 0.0
Here, ‘z’ and ‘t’ are keyword-only parameters, while ‘x’ and ‘y’ are
not. (Contributed by Eric V. Smith in bpo-43532(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43532
File: python.info, Node: distutils<2>, Next: doctest<2>, Prev: dataclasses<2>, Up: Improved Modules<4>
1.4.6.15 distutils
..................
The entire ‘distutils’ package is deprecated, to be removed in Python
3.12. Its functionality for specifying package builds has already been
completely replaced by third-party packages ‘setuptools’ and
‘packaging’, and most other commonly used APIs are available elsewhere
in the standard library (such as *note platform: aa, *note shutil: c5,
*note subprocess: d6. or *note sysconfig: db.). There are no plans to
migrate any other functionality from ‘distutils’, and applications that
are using other functions should plan to make private copies of the
code. Refer to PEP 632(1) for discussion.
The ‘bdist_wininst’ command deprecated in Python 3.8 has been removed.
The ‘bdist_wheel’ command is now recommended to distribute binary
packages on Windows. (Contributed by Victor Stinner in bpo-42802(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0632/
(2) https://bugs.python.org/issue?@action=redirect&bpo=42802
File: python.info, Node: doctest<2>, Next: encodings, Prev: distutils<2>, Up: Improved Modules<4>
1.4.6.16 doctest
................
When a module does not define ‘__loader__’, fall back to
‘__spec__.loader’. (Contributed by Brett Cannon in bpo-42133(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42133
File: python.info, Node: encodings, Next: enum<4>, Prev: doctest<2>, Up: Improved Modules<4>
1.4.6.17 encodings
..................
*note encodings.normalize_encoding(): 7f7. now ignores non-ASCII
characters. (Contributed by Hai Shi in bpo-39337(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39337
File: python.info, Node: enum<4>, Next: fileinput, Prev: encodings, Up: Improved Modules<4>
1.4.6.18 enum
.............
*note Enum: 62c. *note __repr__(): 618. now returns
‘enum_name.member_name’ and *note __str__(): 619. now returns
‘member_name’. Stdlib enums available as module constants have a *note
repr(): 7f9. of ‘module_name.member_name’. (Contributed by Ethan Furman
in bpo-40066(1).)
Add *note enum.StrEnum: 616. for enums where all members are strings.
(Contributed by Ethan Furman in bpo-41816(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40066
(2) https://bugs.python.org/issue?@action=redirect&bpo=41816
File: python.info, Node: fileinput, Next: faulthandler, Prev: enum<4>, Up: Improved Modules<4>
1.4.6.19 fileinput
..................
Add 'encoding' and 'errors' parameters in *note fileinput.input(): 7fb.
and *note fileinput.FileInput: 732. (Contributed by Inada Naoki in
bpo-43712(1).)
*note fileinput.hook_compressed(): 7fc. now returns *note TextIOWrapper:
7b6. object when 'mode' is “r” and file is compressed, like uncompressed
files. (Contributed by Inada Naoki in bpo-5758(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43712
(2) https://bugs.python.org/issue?@action=redirect&bpo=5758
File: python.info, Node: faulthandler, Next: gc, Prev: fileinput, Up: Improved Modules<4>
1.4.6.20 faulthandler
.....................
The *note faulthandler: 58. module now detects if a fatal error occurs
during a garbage collector collection. (Contributed by Victor Stinner
in bpo-44466(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=44466
File: python.info, Node: gc, Next: glob<2>, Prev: faulthandler, Up: Improved Modules<4>
1.4.6.21 gc
...........
Add audit hooks for *note gc.get_objects(): 7ff, *note
gc.get_referrers(): 800. and *note gc.get_referents(): 801.
(Contributed by Pablo Galindo in bpo-43439(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43439
File: python.info, Node: glob<2>, Next: hashlib<3>, Prev: gc, Up: Improved Modules<4>
1.4.6.22 glob
.............
Add the 'root_dir' and 'dir_fd' parameters in *note glob(): 2a1. and
*note iglob(): 803. which allow to specify the root directory for
searching. (Contributed by Serhiy Storchaka in bpo-38144(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38144
File: python.info, Node: hashlib<3>, Next: hmac, Prev: glob<2>, Up: Improved Modules<4>
1.4.6.23 hashlib
................
The hashlib module requires OpenSSL 1.1.1 or newer. (Contributed by
Christian Heimes in PEP 644(1) and bpo-43669(2).)
The hashlib module has preliminary support for OpenSSL 3.0.0.
(Contributed by Christian Heimes in bpo-38820(3) and other issues.)
The pure-Python fallback of *note pbkdf2_hmac(): 50a. is deprecated. In
the future PBKDF2-HMAC will only be available when Python has been built
with OpenSSL support. (Contributed by Christian Heimes in
bpo-43880(4).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0644/
(2) https://bugs.python.org/issue?@action=redirect&bpo=43669
(3) https://bugs.python.org/issue?@action=redirect&bpo=38820
(4) https://bugs.python.org/issue?@action=redirect&bpo=43880
File: python.info, Node: hmac, Next: IDLE and idlelib<2>, Prev: hashlib<3>, Up: Improved Modules<4>
1.4.6.24 hmac
.............
The hmac module now uses OpenSSL’s HMAC implementation internally.
(Contributed by Christian Heimes in bpo-40645(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40645
File: python.info, Node: IDLE and idlelib<2>, Next: importlib metadata<2>, Prev: hmac, Up: Improved Modules<4>
1.4.6.25 IDLE and idlelib
.........................
Make IDLE invoke *note sys.excepthook(): 807. (when started without
‘-n’). User hooks were previously ignored. (Contributed by Ken Hilton
in bpo-43008(1).)
Rearrange the settings dialog. Split the General tab into Windows and
Shell/Ed tabs. Move help sources, which extend the Help menu, to the
Extensions tab. Make space for new options and shorten the dialog. The
latter makes the dialog better fit small screens. (Contributed by Terry
Jan Reedy in bpo-40468(2).) Move the indent space setting from the Font
tab to the new Windows tab. (Contributed by Mark Roseman and Terry Jan
Reedy in bpo-33962(3).)
The changes above were backported to a 3.9 maintenance release.
Add a Shell sidebar. Move the primary prompt (‘>>>’) to the sidebar.
Add secondary prompts (’…’) to the sidebar. Left click and optional
drag selects one or more lines of text, as with the editor line number
sidebar. Right click after selecting text lines displays a context menu
with ‘copy with prompts’. This zips together prompts from the sidebar
with lines from the selected text. This option also appears on the
context menu for the text. (Contributed by Tal Einat in bpo-37903(4).)
Use spaces instead of tabs to indent interactive code. This makes
interactive code entries ‘look right’. Making this feasible was a major
motivation for adding the shell sidebar. (Contributed by Terry Jan
Reedy in bpo-37892(5).)
Highlight the new *note soft keywords: 808. *note match: 809, *note
case: 809, and *note _: 80a. in pattern-matching statements. However,
this highlighting is not perfect and will be incorrect in some rare
cases, including some ‘_’-s in ‘case’ patterns. (Contributed by Tal
Einat in bpo-44010(6).)
New in 3.10 maintenance releases.
Apply syntax highlighting to ‘.pyi’ files. (Contributed by Alex Waygood
and Terry Jan Reedy in bpo-45447(7).)
Include prompts when saving Shell with inputs and outputs. (Contributed
by Terry Jan Reedy in gh-95191(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43008
(2) https://bugs.python.org/issue?@action=redirect&bpo=40468
(3) https://bugs.python.org/issue?@action=redirect&bpo=33962
(4) https://bugs.python.org/issue?@action=redirect&bpo=37903
(5) https://bugs.python.org/issue?@action=redirect&bpo=37892
(6) https://bugs.python.org/issue?@action=redirect&bpo=44010
(7) https://bugs.python.org/issue?@action=redirect&bpo=45447
(8) https://github.com/python/cpython/issues/95191
File: python.info, Node: importlib metadata<2>, Next: inspect<3>, Prev: IDLE and idlelib<2>, Up: Improved Modules<4>
1.4.6.26 importlib.metadata
...........................
Feature parity with ‘importlib_metadata’ 4.6 (history(1)).
*note importlib.metadata entry points: 286. now provide a nicer
experience for selecting entry points by group and name through a new
*note importlib.metadata.EntryPoints: 286. class. See the Compatibility
Note in the docs for more info on the deprecation and usage.
Added *note importlib.metadata.packages_distributions(): 80c. for
resolving top-level Python modules and packages to their *note
importlib.metadata.Distribution: 80d.
---------- Footnotes ----------
(1) https://importlib-metadata.readthedocs.io/en/latest/history.html
File: python.info, Node: inspect<3>, Next: itertools<3>, Prev: importlib metadata<2>, Up: Improved Modules<4>
1.4.6.27 inspect
................
When a module does not define ‘__loader__’, fall back to
‘__spec__.loader’. (Contributed by Brett Cannon in bpo-42133(1).)
Add *note inspect.get_annotations(): 80f, which safely computes the
annotations defined on an object. It works around the quirks of
accessing the annotations on various types of objects, and makes very
few assumptions about the object it examines. *note
inspect.get_annotations(): 80f. can also correctly un-stringize
stringized annotations. *note inspect.get_annotations(): 80f. is now
considered best practice for accessing the annotations dict defined on
any Python object; for more information on best practices for working
with annotations, please see *note Annotations Best Practices: 7d4.
Relatedly, *note inspect.signature(): 733, *note
inspect.Signature.from_callable(): 735, and
‘inspect.Signature.from_function()’ now call *note
inspect.get_annotations(): 80f. to retrieve annotations. This means
*note inspect.signature(): 733. and *note
inspect.Signature.from_callable(): 735. can also now un-stringize
stringized annotations. (Contributed by Larry Hastings in
bpo-43817(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42133
(2) https://bugs.python.org/issue?@action=redirect&bpo=43817
File: python.info, Node: itertools<3>, Next: linecache, Prev: inspect<3>, Up: Improved Modules<4>
1.4.6.28 itertools
..................
Add *note itertools.pairwise(): 811. (Contributed by Raymond Hettinger
in bpo-38200(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38200
File: python.info, Node: linecache, Next: os<4>, Prev: itertools<3>, Up: Improved Modules<4>
1.4.6.29 linecache
..................
When a module does not define ‘__loader__’, fall back to
‘__spec__.loader’. (Contributed by Brett Cannon in bpo-42133(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42133
File: python.info, Node: os<4>, Next: os path<4>, Prev: linecache, Up: Improved Modules<4>
1.4.6.30 os
...........
Add *note os.cpu_count(): 1c5. support for VxWorks RTOS. (Contributed by
Peixing Xin in bpo-41440(1).)
Add a new function *note os.eventfd(): 814. and related helpers to wrap
the ‘eventfd2’ syscall on Linux. (Contributed by Christian Heimes in
bpo-41001(2).)
Add *note os.splice(): 815. that allows to move data between two file
descriptors without copying between kernel address space and user
address space, where one of the file descriptors must refer to a pipe.
(Contributed by Pablo Galindo in bpo-41625(3).)
Add *note O_EVTONLY: 816, *note O_FSYNC: 817, *note O_SYMLINK: 818. and
*note O_NOFOLLOW_ANY: 819. for macOS. (Contributed by Donghee Na in
bpo-43106(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41440
(2) https://bugs.python.org/issue?@action=redirect&bpo=41001
(3) https://bugs.python.org/issue?@action=redirect&bpo=41625
(4) https://bugs.python.org/issue?@action=redirect&bpo=43106
File: python.info, Node: os path<4>, Next: pathlib<5>, Prev: os<4>, Up: Improved Modules<4>
1.4.6.31 os.path
................
*note os.path.realpath(): 227. now accepts a 'strict' keyword-only
argument. When set to ‘True’, *note OSError: 413. is raised if a path
doesn’t exist or a symlink loop is encountered. (Contributed by Barney
Gale in bpo-43757(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43757
File: python.info, Node: pathlib<5>, Next: platform<2>, Prev: os path<4>, Up: Improved Modules<4>
1.4.6.32 pathlib
................
Add slice support to *note PurePath.parents: 81c. (Contributed by
Joshua Cannon in bpo-35498(1).)
Add negative indexing support to *note PurePath.parents: 81c.
(Contributed by Yaroslav Pankovych in bpo-21041(2).)
Add *note Path.hardlink_to: 81d. method that supersedes ‘link_to()’.
The new method has the same argument order as *note symlink_to(): 81e.
(Contributed by Barney Gale in bpo-39950(3).)
*note pathlib.Path.stat(): 81f. and *note chmod(): 820. now accept a
'follow_symlinks' keyword-only argument for consistency with
corresponding functions in the *note os: a1. module. (Contributed by
Barney Gale in bpo-39906(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35498
(2) https://bugs.python.org/issue?@action=redirect&bpo=21041
(3) https://bugs.python.org/issue?@action=redirect&bpo=39950
(4) https://bugs.python.org/issue?@action=redirect&bpo=39906
File: python.info, Node: platform<2>, Next: pprint, Prev: pathlib<5>, Up: Improved Modules<4>
1.4.6.33 platform
.................
Add *note platform.freedesktop_os_release(): 822. to retrieve operation
system identification from freedesktop.org os-release(1) standard file.
(Contributed by Christian Heimes in bpo-28468(2).)
---------- Footnotes ----------
(1) https://www.freedesktop.org/software/systemd/man/os-release.html
(2) https://bugs.python.org/issue?@action=redirect&bpo=28468
File: python.info, Node: pprint, Next: py_compile, Prev: platform<2>, Up: Improved Modules<4>
1.4.6.34 pprint
...............
*note pprint.pprint(): 824. now accepts a new ‘underscore_numbers’
keyword argument. (Contributed by sblondon in bpo-42914(1).)
*note pprint: ae. can now pretty-print *note dataclasses.dataclass: 1cb.
instances. (Contributed by Lewis Gaul in bpo-43080(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42914
(2) https://bugs.python.org/issue?@action=redirect&bpo=43080
File: python.info, Node: py_compile, Next: pyclbr, Prev: pprint, Up: Improved Modules<4>
1.4.6.35 py_compile
...................
Add ‘--quiet’ option to command-line interface of *note py_compile: b3.
(Contributed by Gregory Schevchenko in bpo-38731(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38731
File: python.info, Node: pyclbr, Next: shelve, Prev: py_compile, Up: Improved Modules<4>
1.4.6.36 pyclbr
...............
Add an ‘end_lineno’ attribute to the ‘Function’ and ‘Class’ objects in
the tree returned by *note pyclbr.readmodule(): 827. and *note
pyclbr.readmodule_ex(): 828. It matches the existing (start) ‘lineno’.
(Contributed by Aviral Srivastava in bpo-38307(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38307
File: python.info, Node: shelve, Next: statistics<3>, Prev: pyclbr, Up: Improved Modules<4>
1.4.6.37 shelve
...............
The *note shelve: c3. module now uses *note pickle.DEFAULT_PROTOCOL:
82a. by default instead of *note pickle: a6. protocol ‘3’ when creating
shelves. (Contributed by Zackery Spytz in bpo-34204(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34204
File: python.info, Node: statistics<3>, Next: site<2>, Prev: shelve, Up: Improved Modules<4>
1.4.6.38 statistics
...................
Add *note covariance(): 82c, Pearson’s *note correlation(): 4b8, and
simple *note linear_regression(): 82d. functions. (Contributed by
Tymoteusz Wołodźko in bpo-38490(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38490
File: python.info, Node: site<2>, Next: socket<2>, Prev: statistics<3>, Up: Improved Modules<4>
1.4.6.39 site
.............
When a module does not define ‘__loader__’, fall back to
‘__spec__.loader’. (Contributed by Brett Cannon in bpo-42133(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42133
File: python.info, Node: socket<2>, Next: ssl<3>, Prev: site<2>, Up: Improved Modules<4>
1.4.6.40 socket
...............
The exception *note socket.timeout: 830. is now an alias of *note
TimeoutError: 831. (Contributed by Christian Heimes in bpo-42413(1).)
Add option to create MPTCP sockets with ‘IPPROTO_MPTCP’ (Contributed by
Rui Cunha in bpo-43571(2).)
Add ‘IP_RECVTOS’ option to receive the type of service (ToS) or DSCP/ECN
fields (Contributed by Georg Sauthoff in bpo-44077(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42413
(2) https://bugs.python.org/issue?@action=redirect&bpo=43571
(3) https://bugs.python.org/issue?@action=redirect&bpo=44077
File: python.info, Node: ssl<3>, Next: sqlite3<5>, Prev: socket<2>, Up: Improved Modules<4>
1.4.6.41 ssl
............
The ssl module requires OpenSSL 1.1.1 or newer. (Contributed by
Christian Heimes in PEP 644(1) and bpo-43669(2).)
The ssl module has preliminary support for OpenSSL 3.0.0 and new option
*note OP_IGNORE_UNEXPECTED_EOF: 833. (Contributed by Christian Heimes
in bpo-38820(3), bpo-43794(4), bpo-43788(5), bpo-43791(6), bpo-43799(7),
bpo-43920(8), bpo-43789(9), and bpo-43811(10).)
Deprecated function and use of deprecated constants now result in a
*note DeprecationWarning: 1a5. *note ssl.SSLContext.options: 834. has
*note OP_NO_SSLv2: 835. and *note OP_NO_SSLv3: 836. set by default and
therefore cannot warn about setting the flag again. The *note
deprecation section: 837. has a list of deprecated features.
(Contributed by Christian Heimes in bpo-43880(11).)
The ssl module now has more secure default settings. Ciphers without
forward secrecy or SHA-1 MAC are disabled by default. Security level 2
prohibits weak RSA, DH, and ECC keys with less than 112 bits of
security. *note SSLContext: 296. defaults to minimum protocol version
TLS 1.2. Settings are based on Hynek Schlawack’s research.
(Contributed by Christian Heimes in bpo-43998(12).)
The deprecated protocols SSL 3.0, TLS 1.0, and TLS 1.1 are no longer
officially supported. Python does not block them actively. However
OpenSSL build options, distro configurations, vendor patches, and cipher
suites may prevent a successful handshake.
Add a 'timeout' parameter to the *note ssl.get_server_certificate():
838. function. (Contributed by Zackery Spytz in bpo-31870(13).)
The ssl module uses heap-types and multi-phase initialization.
(Contributed by Christian Heimes in bpo-42333(14).)
A new verify flag *note VERIFY_X509_PARTIAL_CHAIN: 156. has been added.
(Contributed by l0x in bpo-40849(15).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0644/
(2) https://bugs.python.org/issue?@action=redirect&bpo=43669
(3) https://bugs.python.org/issue?@action=redirect&bpo=38820
(4) https://bugs.python.org/issue?@action=redirect&bpo=43794
(5) https://bugs.python.org/issue?@action=redirect&bpo=43788
(6) https://bugs.python.org/issue?@action=redirect&bpo=43791
(7) https://bugs.python.org/issue?@action=redirect&bpo=43799
(8) https://bugs.python.org/issue?@action=redirect&bpo=43920
(9) https://bugs.python.org/issue?@action=redirect&bpo=43789
(10) https://bugs.python.org/issue?@action=redirect&bpo=43811
(11) https://bugs.python.org/issue?@action=redirect&bpo=43880
(12) https://bugs.python.org/issue?@action=redirect&bpo=43998
(13) https://bugs.python.org/issue?@action=redirect&bpo=31870
(14) https://bugs.python.org/issue?@action=redirect&bpo=42333
(15) https://bugs.python.org/issue?@action=redirect&bpo=40849
File: python.info, Node: sqlite3<5>, Next: sys<5>, Prev: ssl<3>, Up: Improved Modules<4>
1.4.6.42 sqlite3
................
Add audit events for ‘connect/handle()’, *note enable_load_extension():
83a, and *note load_extension(): 4b4. (Contributed by Erlend E. Aasland
in bpo-43762(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43762
File: python.info, Node: sys<5>, Next: _thread, Prev: sqlite3<5>, Up: Improved Modules<4>
1.4.6.43 sys
............
Add *note sys.orig_argv: 83c. attribute: the list of the original
command line arguments passed to the Python executable. (Contributed by
Victor Stinner in bpo-23427(1).)
Add *note sys.stdlib_module_names: 83d, containing the list of the
standard library module names. (Contributed by Victor Stinner in
bpo-42955(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23427
(2) https://bugs.python.org/issue?@action=redirect&bpo=42955
File: python.info, Node: _thread, Next: threading<3>, Prev: sys<5>, Up: Improved Modules<4>
1.4.6.44 _thread
................
*note _thread.interrupt_main(): 83f. now takes an optional signal number
to simulate (the default is still *note signal.SIGINT: 840.).
(Contributed by Antoine Pitrou in bpo-43356(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43356
File: python.info, Node: threading<3>, Next: traceback<3>, Prev: _thread, Up: Improved Modules<4>
1.4.6.45 threading
..................
Add *note threading.gettrace(): 842. and *note threading.getprofile():
843. to retrieve the functions set by *note threading.settrace(): 844.
and *note threading.setprofile(): 845. respectively. (Contributed by
Mario Corchero in bpo-42251(1).)
Add *note threading.__excepthook__: 846. to allow retrieving the
original value of *note threading.excepthook(): 847. in case it is set
to a broken or a different value. (Contributed by Mario Corchero in
bpo-42308(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42251
(2) https://bugs.python.org/issue?@action=redirect&bpo=42308
File: python.info, Node: traceback<3>, Next: types<3>, Prev: threading<3>, Up: Improved Modules<4>
1.4.6.46 traceback
..................
The *note format_exception(): 849, *note format_exception_only(): 84a,
and *note print_exception(): 84b. functions can now take an exception
object as a positional-only argument. (Contributed by Zackery Spytz and
Matthias Bussonnier in bpo-26389(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26389
File: python.info, Node: types<3>, Next: typing<5>, Prev: traceback<3>, Up: Improved Modules<4>
1.4.6.47 types
..............
Reintroduce the *note types.EllipsisType: 84d, *note types.NoneType:
84e. and *note types.NotImplementedType: 84f. classes, providing a new
set of types readily interpretable by type checkers. (Contributed by
Bas van Beek in bpo-41810(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41810
File: python.info, Node: typing<5>, Next: unittest<5>, Prev: types<3>, Up: Improved Modules<4>
1.4.6.48 typing
...............
For major changes, see *note New Features Related to Type Hints: 7b9.
The behavior of *note typing.Literal: 851. was changed to conform with
PEP 586(1) and to match the behavior of static type checkers specified
in the PEP.
1. ‘Literal’ now de-duplicates parameters.
2. Equality comparisons between ‘Literal’ objects are now order
independent.
3. ‘Literal’ comparisons now respect types. For example, ‘Literal[0]
== Literal[False]’ previously evaluated to ‘True’. It is now
‘False’. To support this change, the internally used type cache
now supports differentiating types.
4. ‘Literal’ objects will now raise a *note TypeError: 534. exception
during equality comparisons if any of their parameters are not
*note hashable: 60c. Note that declaring ‘Literal’ with unhashable
parameters will not throw an error:
>>> from typing import Literal
>>> Literal[{0}]
>>> Literal[{0}] == Literal[{False}]
Traceback (most recent call last):
File "", line 1, in
TypeError: unhashable type: 'set'
(Contributed by Yurii Karabas in bpo-42345(2).)
Add new function *note typing.is_typeddict(): 852. to introspect if an
annotation is a *note typing.TypedDict: 162. (Contributed by Patrick
Reader in bpo-41792(3).)
Subclasses of ‘typing.Protocol’ which only have data variables declared
will now raise a ‘TypeError’ when checked with ‘isinstance’ unless they
are decorated with *note runtime_checkable(): 43e. Previously, these
checks passed silently. Users should decorate their subclasses with the
‘runtime_checkable()’ decorator if they want runtime protocols.
(Contributed by Yurii Karabas in bpo-38908(4).)
Importing from the ‘typing.io’ and ‘typing.re’ submodules will now emit
*note DeprecationWarning: 1a5. These submodules have been deprecated
since Python 3.8 and will be removed in a future version of Python.
Anything belonging to those submodules should be imported directly from
*note typing: 104. instead. (Contributed by Sebastian Rittau in
bpo-38291(5).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0586/
(2) https://bugs.python.org/issue?@action=redirect&bpo=42345
(3) https://bugs.python.org/issue?@action=redirect&bpo=41792
(4) https://bugs.python.org/issue?@action=redirect&bpo=38908
(5) https://bugs.python.org/issue?@action=redirect&bpo=38291
File: python.info, Node: unittest<5>, Next: urllib parse, Prev: typing<5>, Up: Improved Modules<4>
1.4.6.49 unittest
.................
Add new method *note assertNoLogs(): 854. to complement the existing
*note assertLogs(): 855. (Contributed by Kit Yan Choi in bpo-39385(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39385
File: python.info, Node: urllib parse, Next: xml<2>, Prev: unittest<5>, Up: Improved Modules<4>
1.4.6.50 urllib.parse
.....................
Python versions earlier than Python 3.10 allowed using both ‘;’ and ‘&’
as query parameter separators in *note urllib.parse.parse_qs(): 27c. and
*note urllib.parse.parse_qsl(): 27b. Due to security concerns, and to
conform with newer W3C recommendations, this has been changed to allow
only a single separator key, with ‘&’ as the default. This change also
affects ‘cgi.parse()’ and ‘cgi.parse_multipart()’ as they use the
affected functions internally. For more details, please see their
respective documentation. (Contributed by Adam Goldschmidt, Senthil
Kumaran and Ken Jin in bpo-42967(1).)
The presence of newline or tab characters in parts of a URL allows for
some forms of attacks. Following the WHATWG specification that updates
RFC 3986(2), ASCII newline ‘\n’, ‘\r’ and tab ‘\t’ characters are
stripped from the URL by the parser in *note urllib.parse: 10a.
preventing such attacks. The removal characters are controlled by a new
module level variable ‘urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE’. (See
gh-88048(3))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42967
(2) https://datatracker.ietf.org/doc/html/rfc3986.html
(3) https://github.com/python/cpython/issues/88048
File: python.info, Node: xml<2>, Next: zipimport<3>, Prev: urllib parse, Up: Improved Modules<4>
1.4.6.51 xml
............
Add a *note LexicalHandler: 858. class to the *note xml.sax.handler:
12a. module. (Contributed by Jonathan Gossage and Zackery Spytz in
bpo-35018(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35018
File: python.info, Node: zipimport<3>, Prev: xml<2>, Up: Improved Modules<4>
1.4.6.52 zipimport
..................
Add methods related to PEP 451(1): *note find_spec(): 85a, *note
zipimport.zipimporter.create_module(): 85b, and *note
zipimport.zipimporter.exec_module(): 30c. (Contributed by Brett Cannon
in bpo-42131(2).)
Add *note invalidate_caches(): 85c. method. (Contributed by Desmond
Cheong in bpo-14678(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0451/
(2) https://bugs.python.org/issue?@action=redirect&bpo=42131
(3) https://bugs.python.org/issue?@action=redirect&bpo=14678
File: python.info, Node: Optimizations<4>, Next: Deprecated<5>, Prev: Improved Modules<4>, Up: What’s New In Python 3 10
1.4.7 Optimizations
-------------------
* Constructors *note str(): 447, *note bytes(): 1c2. and *note
bytearray(): 53a. are now faster (around 30–40% for small objects).
(Contributed by Serhiy Storchaka in bpo-41334(1).)
* The *note runpy: be. module now imports fewer modules. The
‘python3 -m module-name’ command startup time is 1.4x faster in
average. On Linux, ‘python3 -I -m module-name’ imports 69 modules
on Python 3.9, whereas it only imports 51 modules (-18) on Python
3.10. (Contributed by Victor Stinner in bpo-41006(2) and
bpo-41718(3).)
* The ‘LOAD_ATTR’ instruction now uses new “per opcode cache”
mechanism. It is about 36% faster now for regular attributes and
44% faster for slots. (Contributed by Pablo Galindo and Yury
Selivanov in bpo-42093(4) and Guido van Rossum in bpo-42927(5),
based on ideas implemented originally in PyPy and MicroPython.)
* When building Python with *note -enable-optimizations: 85e. now
‘-fno-semantic-interposition’ is added to both the compile and link
line. This speeds builds of the Python interpreter created with
*note -enable-shared: 85f. with ‘gcc’ by up to 30%. See this
article(6) for more details. (Contributed by Victor Stinner and
Pablo Galindo in bpo-38980(7).)
* Use a new output buffer management code for *note bz2: 13. / *note
lzma: 8a. / *note zlib: 133. modules, and add ‘.readall()’ function
to ‘_compression.DecompressReader’ class. bz2 decompression is now
1.09x ~ 1.17x faster, lzma decompression 1.20x ~ 1.32x faster,
‘GzipFile.read(-1)’ 1.11x ~ 1.18x faster. (Contributed by Ma Lin,
reviewed by Gregory P. Smith, in bpo-41486(8))
* When using stringized annotations, annotations dicts for functions
are no longer created when the function is created. Instead, they
are stored as a tuple of strings, and the function object lazily
converts this into the annotations dict on demand. This
optimization cuts the CPU time needed to define an annotated
function by half. (Contributed by Yurii Karabas and Inada Naoki in
bpo-42202(9).)
* Substring search functions such as ‘str1 in str2’ and
‘str2.find(str1)’ now sometimes use Crochemore & Perrin’s “Two-Way”
string searching algorithm to avoid quadratic behavior on long
strings. (Contributed by Dennis Sweeney in bpo-41972(10))
* Add micro-optimizations to ‘_PyType_Lookup()’ to improve type
attribute cache lookup performance in the common case of cache
hits. This makes the interpreter 1.04 times faster on average.
(Contributed by Dino Viehland in bpo-43452(11).)
* The following built-in functions now support the faster PEP 590(12)
vectorcall calling convention: *note map(): 860, *note filter():
861, *note reversed(): 862, *note bool(): 463. and *note float():
2f1. (Contributed by Donghee Na and Jeroen Demeyer in
bpo-43575(13), bpo-43287(14), bpo-41922(15), bpo-41873(16) and
bpo-41870(17).)
* *note BZ2File: 863. performance is improved by removing internal
‘RLock’. This makes ‘BZ2File’ thread unsafe in the face of
multiple simultaneous readers or writers, just like its equivalent
classes in *note gzip: 67. and *note lzma: 8a. have always been.
(Contributed by Inada Naoki in bpo-43785(18).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41334
(2) https://bugs.python.org/issue?@action=redirect&bpo=41006
(3) https://bugs.python.org/issue?@action=redirect&bpo=41718
(4) https://bugs.python.org/issue?@action=redirect&bpo=42093
(5) https://bugs.python.org/issue?@action=redirect&bpo=42927
(6)
https://developers.redhat.com/blog/2020/06/25/red-hat-enterprise-linux-8-2-brings-faster-python-3-8-run-speeds/
(7) https://bugs.python.org/issue?@action=redirect&bpo=38980
(8) https://bugs.python.org/issue?@action=redirect&bpo=41486
(9) https://bugs.python.org/issue?@action=redirect&bpo=42202
(10) https://bugs.python.org/issue?@action=redirect&bpo=41972
(11) https://bugs.python.org/issue?@action=redirect&bpo=43452
(12) https://peps.python.org/pep-0590/
(13) https://bugs.python.org/issue?@action=redirect&bpo=43575
(14) https://bugs.python.org/issue?@action=redirect&bpo=43287
(15) https://bugs.python.org/issue?@action=redirect&bpo=41922
(16) https://bugs.python.org/issue?@action=redirect&bpo=41873
(17) https://bugs.python.org/issue?@action=redirect&bpo=41870
(18) https://bugs.python.org/issue?@action=redirect&bpo=43785
File: python.info, Node: Deprecated<5>, Next: Removed<5>, Prev: Optimizations<4>, Up: What’s New In Python 3 10
1.4.8 Deprecated
----------------
* Currently Python accepts numeric literals immediately followed by
keywords, for example ‘0in x’, ‘1or x’, ‘0if 1else 2’. It allows
confusing and ambiguous expressions like ‘[0x1for x in y]’ (which
can be interpreted as ‘[0x1 for x in y]’ or ‘[0x1f or x in y]’).
Starting in this release, a deprecation warning is raised if the
numeric literal is immediately followed by one of keywords *note
and: 2eb, *note else: 18c, *note for: 2ec, *note if: 2ed, *note in:
2ee, *note is: 2ef. and *note or: 2f0. In future releases it will
be changed to syntax warning, and finally to syntax error.
(Contributed by Serhiy Storchaka in bpo-43833(1).)
* Starting in this release, there will be a concerted effort to begin
cleaning up old import semantics that were kept for Python 2.7
compatibility. Specifically, ‘find_loader()’/‘find_module()’
(superseded by *note find_spec(): 865.), *note load_module(): 866.
(superseded by *note exec_module(): 867.), ‘module_repr()’ (which
the import system takes care of for you), the ‘__package__’
attribute (superseded by ‘__spec__.parent’), the ‘__loader__’
attribute (superseded by ‘__spec__.loader’), and the ‘__cached__’
attribute (superseded by ‘__spec__.cached’) will slowly be removed
(as well as other classes and methods in *note importlib: 77.).
*note ImportWarning: 4f6. and/or *note DeprecationWarning: 1a5.
will be raised as appropriate to help identify code which needs
updating during this transition.
* The entire ‘distutils’ namespace is deprecated, to be removed in
Python 3.12. Refer to the *note module changes: 725. section for
more information.
* Non-integer arguments to *note random.randrange(): 868. are
deprecated. The *note ValueError: 204. is deprecated in favor of a
*note TypeError: 534. (Contributed by Serhiy Storchaka and Raymond
Hettinger in bpo-37319(2).)
* The various ‘load_module()’ methods of *note importlib: 77. have
been documented as deprecated since Python 3.6, but will now also
trigger a *note DeprecationWarning: 1a5. Use *note exec_module():
867. instead. (Contributed by Brett Cannon in bpo-26131(3).)
* ‘zimport.zipimporter.load_module()’ has been deprecated in
preference for *note exec_module(): 30c. (Contributed by Brett
Cannon in bpo-26131(4).)
* The use of *note load_module(): 866. by the import system now
triggers an *note ImportWarning: 4f6. as *note exec_module(): 867.
is preferred. (Contributed by Brett Cannon in bpo-26131(5).)
* The use of ‘importlib.abc.MetaPathFinder.find_module()’ and
‘importlib.abc.PathEntryFinder.find_module()’ by the import system
now trigger an *note ImportWarning: 4f6. as *note
importlib.abc.MetaPathFinder.find_spec(): 865. and *note
importlib.abc.PathEntryFinder.find_spec(): 869. are preferred,
respectively. You can use *note importlib.util.spec_from_loader():
86a. to help in porting. (Contributed by Brett Cannon in
bpo-42134(6).)
* The use of ‘importlib.abc.PathEntryFinder.find_loader()’ by the
import system now triggers an *note ImportWarning: 4f6. as *note
importlib.abc.PathEntryFinder.find_spec(): 869. is preferred. You
can use *note importlib.util.spec_from_loader(): 86a. to help in
porting. (Contributed by Brett Cannon in bpo-43672(7).)
* The various implementations of
‘importlib.abc.MetaPathFinder.find_module()’ (
‘importlib.machinery.BuiltinImporter.find_module()’,
‘importlib.machinery.FrozenImporter.find_module()’,
‘importlib.machinery.WindowsRegistryFinder.find_module()’,
‘importlib.machinery.PathFinder.find_module()’,
‘importlib.abc.MetaPathFinder.find_module()’ ),
‘importlib.abc.PathEntryFinder.find_module()’ (
‘importlib.machinery.FileFinder.find_module()’ ), and
‘importlib.abc.PathEntryFinder.find_loader()’ (
‘importlib.machinery.FileFinder.find_loader()’ ) now raise *note
DeprecationWarning: 1a5. and are slated for removal in Python 3.12
(previously they were documented as deprecated in Python 3.4).
(Contributed by Brett Cannon in bpo-42135(8).)
* ‘importlib.abc.Finder’ is deprecated (including its sole method,
‘find_module()’). Both *note importlib.abc.MetaPathFinder: 86b.
and *note importlib.abc.PathEntryFinder: 86c. no longer inherit
from the class. Users should inherit from one of these two classes
as appropriate instead. (Contributed by Brett Cannon in
bpo-42135(9).)
* The deprecations of ‘imp’, ‘importlib.find_loader()’,
‘importlib.util.set_package_wrapper()’,
‘importlib.util.set_loader_wrapper()’,
‘importlib.util.module_for_loader()’, ‘pkgutil.ImpImporter’, and
‘pkgutil.ImpLoader’ have all been updated to list Python 3.12 as
the slated version of removal (they began raising *note
DeprecationWarning: 1a5. in previous versions of Python).
(Contributed by Brett Cannon in bpo-43720(10).)
* The import system now uses the ‘__spec__’ attribute on modules
before falling back on ‘module_repr()’ for a module’s ‘__repr__()’
method. Removal of the use of ‘module_repr()’ is scheduled for
Python 3.12. (Contributed by Brett Cannon in bpo-42137(11).)
* ‘importlib.abc.Loader.module_repr()’,
‘importlib.machinery.FrozenLoader.module_repr()’, and
‘importlib.machinery.BuiltinLoader.module_repr()’ are deprecated
and slated for removal in Python 3.12. (Contributed by Brett
Cannon in bpo-42136(12).)
* ‘sqlite3.OptimizedUnicode’ has been undocumented and obsolete since
Python 3.3, when it was made an alias to *note str: 447. It is now
deprecated, scheduled for removal in Python 3.12. (Contributed by
Erlend E. Aasland in bpo-42264(13).)
* The undocumented built-in function ‘sqlite3.enable_shared_cache’ is
now deprecated, scheduled for removal in Python 3.12. Its use is
strongly discouraged by the SQLite3 documentation. See the SQLite3
docs(14) for more details. If a shared cache must be used, open
the database in URI mode using the ‘cache=shared’ query parameter.
(Contributed by Erlend E. Aasland in bpo-24464(15).)
* The following ‘threading’ methods are now deprecated:
* ‘threading.currentThread’ => *note threading.current_thread():
303.
* ‘threading.activeCount’ => *note threading.active_count():
304.
* ‘threading.Condition.notifyAll’ => *note
threading.Condition.notify_all(): 2fd.
* ‘threading.Event.isSet’ => *note threading.Event.is_set():
2fe.
* ‘threading.Thread.setName’ => *note threading.Thread.name:
302.
* ‘threading.thread.getName’ => *note threading.Thread.name:
302.
* ‘threading.Thread.isDaemon’ => *note threading.Thread.daemon:
300.
* ‘threading.Thread.setDaemon’ => *note threading.Thread.daemon:
300.
(Contributed by Jelle Zijlstra in gh-87889(16).)
* ‘pathlib.Path.link_to()’ is deprecated and slated for removal in
Python 3.12. Use *note pathlib.Path.hardlink_to(): 81d. instead.
(Contributed by Barney Gale in bpo-39950(17).)
* ‘cgi.log()’ is deprecated and slated for removal in Python 3.12.
(Contributed by Inada Naoki in bpo-41139(18).)
* The following *note ssl: d0. features have been deprecated since
Python 3.6, Python 3.7, or OpenSSL 1.1.0 and will be removed in
3.11:
* ‘OP_NO_SSLv2’, ‘OP_NO_SSLv3’, ‘OP_NO_TLSv1’, ‘OP_NO_TLSv1_1’,
‘OP_NO_TLSv1_2’, and ‘OP_NO_TLSv1_3’ are replaced by *note
minimum_version: 86d. and *note maximum_version: 86e.
* ‘PROTOCOL_SSLv2’, ‘PROTOCOL_SSLv3’, ‘PROTOCOL_SSLv23’,
‘PROTOCOL_TLSv1’, ‘PROTOCOL_TLSv1_1’, ‘PROTOCOL_TLSv1_2’, and
‘PROTOCOL_TLS’ are deprecated in favor of *note
PROTOCOL_TLS_CLIENT: 86f. and *note PROTOCOL_TLS_SERVER: 870.
* ‘wrap_socket()’ is replaced by *note
ssl.SSLContext.wrap_socket(): 520.
* ‘match_hostname()’
* ‘RAND_pseudo_bytes()’, ‘RAND_egd()’
* NPN features like *note ssl.SSLSocket.selected_npn_protocol():
871. and *note ssl.SSLContext.set_npn_protocols(): 2fc. are
replaced by ALPN.
* The threading debug (‘PYTHONTHREADDEBUG’ environment variable) is
deprecated in Python 3.10 and will be removed in Python 3.12. This
feature requires a *note debug build of Python: 1fb. (Contributed
by Victor Stinner in bpo-44584(19).)
* Importing from the ‘typing.io’ and ‘typing.re’ submodules will now
emit *note DeprecationWarning: 1a5. These submodules will be
removed in a future version of Python. Anything belonging to these
submodules should be imported directly from *note typing: 104.
instead. (Contributed by Sebastian Rittau in bpo-38291(20).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43833
(2) https://bugs.python.org/issue?@action=redirect&bpo=37319
(3) https://bugs.python.org/issue?@action=redirect&bpo=26131
(4) https://bugs.python.org/issue?@action=redirect&bpo=26131
(5) https://bugs.python.org/issue?@action=redirect&bpo=26131
(6) https://bugs.python.org/issue?@action=redirect&bpo=42134
(7) https://bugs.python.org/issue?@action=redirect&bpo=43672
(8) https://bugs.python.org/issue?@action=redirect&bpo=42135
(9) https://bugs.python.org/issue?@action=redirect&bpo=42135
(10) https://bugs.python.org/issue?@action=redirect&bpo=43720
(11) https://bugs.python.org/issue?@action=redirect&bpo=42137
(12) https://bugs.python.org/issue?@action=redirect&bpo=42136
(13) https://bugs.python.org/issue?@action=redirect&bpo=42264
(14) https://sqlite.org/c3ref/enable_shared_cache.html
(15) https://bugs.python.org/issue?@action=redirect&bpo=24464
(16) https://github.com/python/cpython/issues/87889
(17) https://bugs.python.org/issue?@action=redirect&bpo=39950
(18) https://bugs.python.org/issue?@action=redirect&bpo=41139
(19) https://bugs.python.org/issue?@action=redirect&bpo=44584
(20) https://bugs.python.org/issue?@action=redirect&bpo=38291
File: python.info, Node: Removed<5>, Next: Porting to Python 3 10, Prev: Deprecated<5>, Up: What’s New In Python 3 10
1.4.9 Removed
-------------
* Removed special methods ‘__int__’, ‘__float__’, ‘__floordiv__’,
‘__mod__’, ‘__divmod__’, ‘__rfloordiv__’, ‘__rmod__’ and
‘__rdivmod__’ of the *note complex: 2f2. class. They always raised
a *note TypeError: 534. (Contributed by Serhiy Storchaka in
bpo-41974(1).)
* The ‘ParserBase.error()’ method from the private and undocumented
‘_markupbase’ module has been removed. *note
html.parser.HTMLParser: 874. is the only subclass of ‘ParserBase’
and its ‘error()’ implementation was already removed in Python 3.5.
(Contributed by Berker Peksag in bpo-31844(2).)
* Removed the ‘unicodedata.ucnhash_CAPI’ attribute which was an
internal PyCapsule object. The related private
‘_PyUnicode_Name_CAPI’ structure was moved to the internal C API.
(Contributed by Victor Stinner in bpo-42157(3).)
* Removed the ‘parser’ module, which was deprecated in 3.9 due to the
switch to the new PEG parser, as well as all the C source and
header files that were only being used by the old parser, including
‘node.h’, ‘parser.h’, ‘graminit.h’ and ‘grammar.h’.
* Removed the Public C API functions
‘PyParser_SimpleParseStringFlags’,
‘PyParser_SimpleParseStringFlagsFilename’,
‘PyParser_SimpleParseFileFlags’ and ‘PyNode_Compile’ that were
deprecated in 3.9 due to the switch to the new PEG parser.
* Removed the ‘formatter’ module, which was deprecated in Python 3.4.
It is somewhat obsolete, little used, and not tested. It was
originally scheduled to be removed in Python 3.6, but such removals
were delayed until after Python 2.7 EOL. Existing users should copy
whatever classes they use into their code. (Contributed by Donghee
Na and Terry J. Reedy in bpo-42299(4).)
* Removed the ‘PyModule_GetWarningsModule()’ function that was
useless now due to the ‘_warnings’ module was converted to a
builtin module in 2.6. (Contributed by Hai Shi in bpo-42599(5).)
* Remove deprecated aliases to *note Collections Abstract Base
Classes: 875. from the *note collections: 1d. module. (Contributed
by Victor Stinner in bpo-37324(6).)
* The ‘loop’ parameter has been removed from most of *note asyncio:
a.‘s *note high-level API: 876. following deprecation in Python
3.8. The motivation behind this change is multifold:
1. This simplifies the high-level API.
2. The functions in the high-level API have been implicitly
getting the current thread’s running event loop since Python
3.7. There isn’t a need to pass the event loop to the API in
most normal use cases.
3. Event loop passing is error-prone especially when dealing with
loops running in different threads.
Note that the low-level API will still accept ‘loop’. See *note
Changes in the Python API: 877. for examples of how to replace
existing code.
(Contributed by Yurii Karabas, Andrew Svetlov, Yury Selivanov and
Kyle Stanley in bpo-42392(7).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41974
(2) https://bugs.python.org/issue?@action=redirect&bpo=31844
(3) https://bugs.python.org/issue?@action=redirect&bpo=42157
(4) https://bugs.python.org/issue?@action=redirect&bpo=42299
(5) https://bugs.python.org/issue?@action=redirect&bpo=42599
(6) https://bugs.python.org/issue?@action=redirect&bpo=37324
(7) https://bugs.python.org/issue?@action=redirect&bpo=42392
File: python.info, Node: Porting to Python 3 10, Next: CPython bytecode changes<3>, Prev: Removed<5>, Up: What’s New In Python 3 10
1.4.10 Porting to Python 3.10
-----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in the Python syntax::
* Changes in the Python API: Changes in the Python API<3>.
* Changes in the C API: Changes in the C API<2>.
File: python.info, Node: Changes in the Python syntax, Next: Changes in the Python API<3>, Up: Porting to Python 3 10
1.4.10.1 Changes in the Python syntax
.....................................
* Deprecation warning is now emitted when compiling previously valid
syntax if the numeric literal is immediately followed by a keyword
(like in ‘0in x’). In future releases it will be changed to syntax
warning, and finally to a syntax error. To get rid of the warning
and make the code compatible with future releases just add a space
between the numeric literal and the following keyword.
(Contributed by Serhiy Storchaka in bpo-43833(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=43833
File: python.info, Node: Changes in the Python API<3>, Next: Changes in the C API<2>, Prev: Changes in the Python syntax, Up: Porting to Python 3 10
1.4.10.2 Changes in the Python API
..................................
* The 'etype' parameters of the *note format_exception(): 849, *note
format_exception_only(): 84a, and *note print_exception(): 84b.
functions in the *note traceback: fe. module have been renamed to
'exc'. (Contributed by Zackery Spytz and Matthias Bussonnier in
bpo-26389(1).)
* *note atexit: c.: At Python exit, if a callback registered with
*note atexit.register(): 87b. fails, its exception is now logged.
Previously, only some exceptions were logged, and the last
exception was always silently ignored. (Contributed by Victor
Stinner in bpo-42639(2).)
* *note collections.abc.Callable: 7e5. generic now flattens type
parameters, similar to what *note typing.Callable: 7c0. currently
does. This means that ‘collections.abc.Callable[[int, str], str]’
will have ‘__args__’ of ‘(int, str, str)’; previously this was
‘([int, str], str)’. Code which accesses the arguments via *note
typing.get_args(): 87c. or ‘__args__’ need to account for this
change. Furthermore, *note TypeError: 534. may be raised for
invalid forms of parameterizing *note collections.abc.Callable:
7e5. which may have passed silently in Python 3.9. (Contributed by
Ken Jin in bpo-42195(3).)
* *note socket.htons(): 87d. and *note socket.ntohs(): 87e. now raise
*note OverflowError: 87f. instead of *note DeprecationWarning: 1a5.
if the given parameter will not fit in a 16-bit unsigned integer.
(Contributed by Erlend E. Aasland in bpo-42393(4).)
* The ‘loop’ parameter has been removed from most of *note asyncio:
a.‘s *note high-level API: 876. following deprecation in Python
3.8.
A coroutine that currently looks like this:
async def foo(loop):
await asyncio.sleep(1, loop=loop)
Should be replaced with this:
async def foo():
await asyncio.sleep(1)
If ‘foo()’ was specifically designed 'not' to run in the current
thread’s running event loop (e.g. running in another thread’s
event loop), consider using *note
asyncio.run_coroutine_threadsafe(): 880. instead.
(Contributed by Yurii Karabas, Andrew Svetlov, Yury Selivanov and
Kyle Stanley in bpo-42392(5).)
* The *note types.FunctionType: 881. constructor now inherits the
current builtins if the 'globals' dictionary has no
‘"__builtins__"’ key, rather than using ‘{"None": None}’ as
builtins: same behavior as *note eval(): 180. and *note exec():
17f. functions. Defining a function with ‘def function(...): ...’
in Python is not affected, globals cannot be overridden with this
syntax: it also inherits the current builtins. (Contributed by
Victor Stinner in bpo-42990(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26389
(2) https://bugs.python.org/issue?@action=redirect&bpo=42639
(3) https://bugs.python.org/issue?@action=redirect&bpo=42195
(4) https://bugs.python.org/issue?@action=redirect&bpo=42393
(5) https://bugs.python.org/issue?@action=redirect&bpo=42392
(6) https://bugs.python.org/issue?@action=redirect&bpo=42990
File: python.info, Node: Changes in the C API<2>, Prev: Changes in the Python API<3>, Up: Porting to Python 3 10
1.4.10.3 Changes in the C API
.............................
* The C API functions ‘PyParser_SimpleParseStringFlags’,
‘PyParser_SimpleParseStringFlagsFilename’,
‘PyParser_SimpleParseFileFlags’, ‘PyNode_Compile’ and the type used
by these functions, ‘struct _node’, were removed due to the switch
to the new PEG parser.
Source should be now be compiled directly to a code object using,
for example, *note Py_CompileString(): 883. The resulting code
object can then be evaluated using, for example, *note
PyEval_EvalCode(): 884.
Specifically:
* A call to ‘PyParser_SimpleParseStringFlags’ followed by
‘PyNode_Compile’ can be replaced by calling *note
Py_CompileString(): 883.
* There is no direct replacement for
‘PyParser_SimpleParseFileFlags’. To compile code from a ‘FILE
*’ argument, you will need to read the file in C and pass the
resulting buffer to *note Py_CompileString(): 883.
* To compile a file given a ‘char *’ filename, explicitly open
the file, read it and compile the result. One way to do this
is using the *note io: 7f. module with *note
PyImport_ImportModule(): 3ba, *note PyObject_CallMethod():
39b, *note PyBytes_AsString(): 885. and *note
Py_CompileString(): 883, as sketched below. (Declarations and
error handling are omitted.)
io_module = Import_ImportModule("io");
fileobject = PyObject_CallMethod(io_module, "open", "ss", filename, "rb");
source_bytes_object = PyObject_CallMethod(fileobject, "read", "");
result = PyObject_CallMethod(fileobject, "close", "");
source_buf = PyBytes_AsString(source_bytes_object);
code = Py_CompileString(source_buf, filename, Py_file_input);
* For ‘FrameObject’ objects, the *note f_lasti: 886. member now
represents a wordcode offset instead of a simple offset into
the bytecode string. This means that this number needs to be
multiplied by 2 to be used with APIs that expect a byte offset
instead (like *note PyCode_Addr2Line(): 887. for example).
Notice as well that the ‘f_lasti’ member of ‘FrameObject’
objects is not considered stable: please use *note
PyFrame_GetLineNumber(): 786. instead.
File: python.info, Node: CPython bytecode changes<3>, Next: Build Changes<4>, Prev: Porting to Python 3 10, Up: What’s New In Python 3 10
1.4.11 CPython bytecode changes
-------------------------------
* The ‘MAKE_FUNCTION’ instruction now accepts either a dict or a
tuple of strings as the function’s annotations. (Contributed by
Yurii Karabas and Inada Naoki in bpo-42202(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42202
File: python.info, Node: Build Changes<4>, Next: C API Changes<4>, Prev: CPython bytecode changes<3>, Up: What’s New In Python 3 10
1.4.12 Build Changes
--------------------
* PEP 644(1): Python now requires OpenSSL 1.1.1 or newer. OpenSSL
1.0.2 is no longer supported. (Contributed by Christian Heimes in
bpo-43669(2).)
* The C99 functions ‘snprintf()’ and ‘vsnprintf()’ are now required
to build Python. (Contributed by Victor Stinner in bpo-36020(3).)
* *note sqlite3: cf. requires SQLite 3.7.15 or higher. (Contributed
by Sergey Fedoseev and Erlend E. Aasland in bpo-40744(4) and
bpo-40810(5).)
* The *note atexit: c. module must now always be built as a built-in
module. (Contributed by Victor Stinner in bpo-42639(6).)
* Add *note -disable-test-modules: 88a. option to the ‘configure’
script: don’t build nor install test modules. (Contributed by
Xavier de Gaye, Thomas Petazzoni and Peixing Xin in bpo-27640(7).)
* Add *note -with-wheel-pkg-dir=PATH option: 88b. to the
‘./configure’ script. If specified, the *note ensurepip: 55.
module looks for ‘setuptools’ and ‘pip’ wheel packages in this
directory: if both are present, these wheel packages are used
instead of ensurepip bundled wheel packages.
Some Linux distribution packaging policies recommend against
bundling dependencies. For example, Fedora installs wheel packages
in the ‘/usr/share/python-wheels/’ directory and don’t install the
‘ensurepip._bundled’ package.
(Contributed by Victor Stinner in bpo-42856(8).)
* Add a new *note configure -without-static-libpython option: 88c. to
not build the ‘libpythonMAJOR.MINOR.a’ static library and not
install the ‘python.o’ object file.
(Contributed by Victor Stinner in bpo-43103(9).)
* The ‘configure’ script now uses the ‘pkg-config’ utility, if
available, to detect the location of Tcl/Tk headers and libraries.
As before, those locations can be explicitly specified with the
‘--with-tcltk-includes’ and ‘--with-tcltk-libs’ configuration
options. (Contributed by Manolis Stamatogiannakis in
bpo-42603(10).)
* Add *note -with-openssl-rpath: 88d. option to ‘configure’ script.
The option simplifies building Python with a custom OpenSSL
installation, e.g. ‘./configure --with-openssl=/path/to/openssl
--with-openssl-rpath=auto’. (Contributed by Christian Heimes in
bpo-43466(11).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0644/
(2) https://bugs.python.org/issue?@action=redirect&bpo=43669
(3) https://bugs.python.org/issue?@action=redirect&bpo=36020
(4) https://bugs.python.org/issue?@action=redirect&bpo=40744
(5) https://bugs.python.org/issue?@action=redirect&bpo=40810
(6) https://bugs.python.org/issue?@action=redirect&bpo=42639
(7) https://bugs.python.org/issue?@action=redirect&bpo=27640
(8) https://bugs.python.org/issue?@action=redirect&bpo=42856
(9) https://bugs.python.org/issue?@action=redirect&bpo=43103
(10) https://bugs.python.org/issue?@action=redirect&bpo=42603
(11) https://bugs.python.org/issue?@action=redirect&bpo=43466
File: python.info, Node: C API Changes<4>, Next: Notable security feature in 3 10 7, Prev: Build Changes<4>, Up: What’s New In Python 3 10
1.4.13 C API Changes
--------------------
* Menu:
* PEP 652; Maintaining the Stable ABI: PEP 652 Maintaining the Stable ABI.
* New Features: New Features<8>.
* Porting to Python 3.10: Porting to Python 3 10<2>.
* Deprecated: Deprecated<6>.
* Removed: Removed<6>.
File: python.info, Node: PEP 652 Maintaining the Stable ABI, Next: New Features<8>, Up: C API Changes<4>
1.4.13.1 PEP 652: Maintaining the Stable ABI
............................................
The Stable ABI (Application Binary Interface) for extension modules or
embedding Python is now explicitly defined. *note C API Stability: 550.
describes C API and ABI stability guarantees along with best practices
for using the Stable ABI.
(Contributed by Petr Viktorin in PEP 652(1) and bpo-43795(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0652/
(2) https://bugs.python.org/issue?@action=redirect&bpo=43795
File: python.info, Node: New Features<8>, Next: Porting to Python 3 10<2>, Prev: PEP 652 Maintaining the Stable ABI, Up: C API Changes<4>
1.4.13.2 New Features
.....................
* The result of *note PyNumber_Index(): 891. now always has exact
type *note int: 259. Previously, the result could have been an
instance of a subclass of ‘int’. (Contributed by Serhiy Storchaka
in bpo-40792(1).)
* Add a new *note orig_argv: 892. member to the *note PyConfig: 3a2.
structure: the list of the original command line arguments passed
to the Python executable. (Contributed by Victor Stinner in
bpo-23427(2).)
* The *note PyDateTime_DATE_GET_TZINFO(): 893. and *note
PyDateTime_TIME_GET_TZINFO(): 894. macros have been added for
accessing the ‘tzinfo’ attributes of *note datetime.datetime: 1cc.
and *note datetime.time: 1ce. objects. (Contributed by Zackery
Spytz in bpo-30155(3).)
* Add a *note PyCodec_Unregister(): 895. function to unregister a
codec search function. (Contributed by Hai Shi in bpo-41842(4).)
* The *note PyIter_Send(): 896. function was added to allow sending
value into iterator without raising ‘StopIteration’ exception.
(Contributed by Vladimir Matveev in bpo-41756(5).)
* Add *note PyUnicode_AsUTF8AndSize(): 897. to the limited C API.
(Contributed by Alex Gaynor in bpo-41784(6).)
* Add *note PyModule_AddObjectRef(): 360. function: similar to *note
PyModule_AddObject(): 361. but don’t steal a reference to the value
on success. (Contributed by Victor Stinner in bpo-1635741(7).)
* Add *note Py_NewRef(): 898. and *note Py_XNewRef(): 899. functions
to increment the reference count of an object and return the
object. (Contributed by Victor Stinner in bpo-42262(8).)
* The *note PyType_FromSpecWithBases(): 57b. and *note
PyType_FromModuleAndSpec(): 54e. functions now accept a single
class as the 'bases' argument. (Contributed by Serhiy Storchaka in
bpo-42423(9).)
* The *note PyType_FromModuleAndSpec(): 54e. function now accepts
NULL ‘tp_doc’ slot. (Contributed by Hai Shi in bpo-41832(10).)
* The *note PyType_GetSlot(): 89a. function can accept *note static
types: 77a. (Contributed by Hai Shi and Petr Viktorin in
bpo-41073(11).)
* Add a new *note PySet_CheckExact(): 89b. function to the C-API to
check if an object is an instance of *note set: 5d5. but not an
instance of a subtype. (Contributed by Pablo Galindo in
bpo-43277(12).)
* Add *note PyErr_SetInterruptEx(): 89c. which allows passing a
signal number to simulate. (Contributed by Antoine Pitrou in
bpo-43356(13).)
* The limited C API is now supported if *note Python is built in
debug mode: 1fb. (if the ‘Py_DEBUG’ macro is defined). In the
limited C API, the *note Py_INCREF(): 56b. and *note Py_DECREF():
56c. functions are now implemented as opaque function calls, rather
than accessing directly the *note PyObject.ob_refcnt: 89d. member,
if Python is built in debug mode and the ‘Py_LIMITED_API’ macro
targets Python 3.10 or newer. It became possible to support the
limited C API in debug mode because the *note PyObject: 334.
structure is the same in release and debug mode since Python 3.8
(see bpo-36465(14)).
The limited C API is still not supported in the *note
-with-trace-refs: 390. special build (‘Py_TRACE_REFS’ macro).
(Contributed by Victor Stinner in bpo-43688(15).)
* Add the *note Py_Is(x, y): 89e. function to test if the 'x' object
is the 'y' object, the same as ‘x is y’ in Python. Add also the
*note Py_IsNone(): 89f, *note Py_IsTrue(): 8a0, *note Py_IsFalse():
8a1. functions to test if an object is, respectively, the ‘None’
singleton, the ‘True’ singleton or the ‘False’ singleton.
(Contributed by Victor Stinner in bpo-43753(16).)
* Add new functions to control the garbage collector from C code:
*note PyGC_Enable(): 8a2, *note PyGC_Disable(): 8a3, *note
PyGC_IsEnabled(): 8a4. These functions allow to activate,
deactivate and query the state of the garbage collector from C code
without having to import the *note gc: 60. module.
* Add a new *note Py_TPFLAGS_DISALLOW_INSTANTIATION: 57f. type flag
to disallow creating type instances. (Contributed by Victor
Stinner in bpo-43916(17).)
* Add a new *note Py_TPFLAGS_IMMUTABLETYPE: 3be. type flag for
creating immutable type objects: type attributes cannot be set nor
deleted. (Contributed by Victor Stinner and Erlend E. Aasland in
bpo-43908(18).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40792
(2) https://bugs.python.org/issue?@action=redirect&bpo=23427
(3) https://bugs.python.org/issue?@action=redirect&bpo=30155
(4) https://bugs.python.org/issue?@action=redirect&bpo=41842
(5) https://bugs.python.org/issue?@action=redirect&bpo=41756
(6) https://bugs.python.org/issue?@action=redirect&bpo=41784
(7) https://bugs.python.org/issue?@action=redirect&bpo=1635741
(8) https://bugs.python.org/issue?@action=redirect&bpo=42262
(9) https://bugs.python.org/issue?@action=redirect&bpo=42423
(10) https://bugs.python.org/issue?@action=redirect&bpo=41832
(11) https://bugs.python.org/issue?@action=redirect&bpo=41073
(12) https://bugs.python.org/issue?@action=redirect&bpo=43277
(13) https://bugs.python.org/issue?@action=redirect&bpo=43356
(14) https://bugs.python.org/issue?@action=redirect&bpo=36465
(15) https://bugs.python.org/issue?@action=redirect&bpo=43688
(16) https://bugs.python.org/issue?@action=redirect&bpo=43753
(17) https://bugs.python.org/issue?@action=redirect&bpo=43916
(18) https://bugs.python.org/issue?@action=redirect&bpo=43908
File: python.info, Node: Porting to Python 3 10<2>, Next: Deprecated<6>, Prev: New Features<8>, Up: C API Changes<4>
1.4.13.3 Porting to Python 3.10
...............................
* The ‘PY_SSIZE_T_CLEAN’ macro must now be defined to use *note
PyArg_ParseTuple(): 56e. and *note Py_BuildValue(): 8a6. formats
which use ‘#’: ‘es#’, ‘et#’, ‘s#’, ‘u#’, ‘y#’, ‘z#’, ‘U#’ and ‘Z#’.
See *note Parsing arguments and building values: 8a7. and PEP
353(1). (Contributed by Victor Stinner in bpo-40943(2).)
* Since *note Py_REFCNT(): 8a8. is changed to the inline static
function, ‘Py_REFCNT(obj) = new_refcnt’ must be replaced with
‘Py_SET_REFCNT(obj, new_refcnt)’: see *note Py_SET_REFCNT(): 8a9.
(available since Python 3.9). For backward compatibility, this
macro can be used:
#if PY_VERSION_HEX < 0x030900A4
# define Py_SET_REFCNT(obj, refcnt) ((Py_REFCNT(obj) = (refcnt)), (void)0)
#endif
(Contributed by Victor Stinner in bpo-39573(3).)
* Calling *note PyDict_GetItem(): 382. without *note GIL: 159. held
had been allowed for historical reason. It is no longer allowed.
(Contributed by Victor Stinner in bpo-40839(4).)
* ‘PyUnicode_FromUnicode(NULL, size)’ and
‘PyUnicode_FromStringAndSize(NULL, size)’ raise
‘DeprecationWarning’ now. Use *note PyUnicode_New(): 8aa. to
allocate Unicode object without initial data. (Contributed by
Inada Naoki in bpo-36346(5).)
* The private ‘_PyUnicode_Name_CAPI’ structure of the PyCapsule API
‘unicodedata.ucnhash_CAPI’ has been moved to the internal C API.
(Contributed by Victor Stinner in bpo-42157(6).)
* *note Py_GetPath(): 3af, *note Py_GetPrefix(): 3b1, *note
Py_GetExecPrefix(): 3ad, *note Py_GetProgramFullPath(): 3b3, *note
Py_GetPythonHome(): 3b6. and *note Py_GetProgramName(): 3b5.
functions now return ‘NULL’ if called before *note Py_Initialize():
8ab. (before Python is initialized). Use the new *note Python
Initialization Configuration: 3a3. API to get the *note Python Path
Configuration: 8ac. (Contributed by Victor Stinner in
bpo-42260(7).)
* *note PyList_SET_ITEM(): 389, *note PyTuple_SET_ITEM(): 388. and
*note PyCell_SET(): 8ad. macros can no longer be used as l-value or
r-value. For example, ‘x = PyList_SET_ITEM(a, b, c)’ and
‘PyList_SET_ITEM(a, b, c) = x’ now fail with a compiler error. It
prevents bugs like ‘if (PyList_SET_ITEM (a, b, c) < 0) ...’ test.
(Contributed by Zackery Spytz and Victor Stinner in bpo-30459(8).)
* The non-limited API files ‘odictobject.h’, ‘parser_interface.h’,
‘picklebufobject.h’, ‘pyarena.h’, ‘pyctype.h’, ‘pydebug.h’,
‘pyfpe.h’, and ‘pytime.h’ have been moved to the ‘Include/cpython’
directory. These files must not be included directly, as they are
already included in ‘Python.h’; see *note Include Files: 77f. If
they have been included directly, consider including ‘Python.h’
instead. (Contributed by Nicholas Sim in bpo-35134(9).)
* Use the *note Py_TPFLAGS_IMMUTABLETYPE: 3be. type flag to create
immutable type objects. Do not rely on *note Py_TPFLAGS_HEAPTYPE:
8ae. to decide if a type object is mutable or not; check if *note
Py_TPFLAGS_IMMUTABLETYPE: 3be. is set instead. (Contributed by
Victor Stinner and Erlend E. Aasland in bpo-43908(10).)
* The undocumented function ‘Py_FrozenMain’ has been removed from the
limited API. The function is mainly useful for custom builds of
Python. (Contributed by Petr Viktorin in bpo-26241(11).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0353/
(2) https://bugs.python.org/issue?@action=redirect&bpo=40943
(3) https://bugs.python.org/issue?@action=redirect&bpo=39573
(4) https://bugs.python.org/issue?@action=redirect&bpo=40839
(5) https://bugs.python.org/issue?@action=redirect&bpo=36346
(6) https://bugs.python.org/issue?@action=redirect&bpo=42157
(7) https://bugs.python.org/issue?@action=redirect&bpo=42260
(8) https://bugs.python.org/issue?@action=redirect&bpo=30459
(9) https://bugs.python.org/issue?@action=redirect&bpo=35134
(10) https://bugs.python.org/issue?@action=redirect&bpo=43908
(11) https://bugs.python.org/issue?@action=redirect&bpo=26241
File: python.info, Node: Deprecated<6>, Next: Removed<6>, Prev: Porting to Python 3 10<2>, Up: C API Changes<4>
1.4.13.4 Deprecated
...................
* The ‘PyUnicode_InternImmortal()’ function is now deprecated and
will be removed in Python 3.12: use *note
PyUnicode_InternInPlace(): 8b0. instead. (Contributed by Victor
Stinner in bpo-41692(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41692
File: python.info, Node: Removed<6>, Prev: Deprecated<6>, Up: C API Changes<4>
1.4.13.5 Removed
................
* Removed ‘Py_UNICODE_str*’ functions manipulating ‘Py_UNICODE*’
strings. (Contributed by Inada Naoki in bpo-41123(1).)
* ‘Py_UNICODE_strlen’: use *note PyUnicode_GetLength(): 8b2. or
*note PyUnicode_GET_LENGTH: 8b3.
* ‘Py_UNICODE_strcat’: use *note PyUnicode_CopyCharacters():
8b4. or *note PyUnicode_FromFormat(): 385.
* ‘Py_UNICODE_strcpy’, ‘Py_UNICODE_strncpy’: use *note
PyUnicode_CopyCharacters(): 8b4. or *note
PyUnicode_Substring(): 8b5.
* ‘Py_UNICODE_strcmp’: use *note PyUnicode_Compare(): 8b6.
* ‘Py_UNICODE_strncmp’: use *note PyUnicode_Tailmatch(): 8b7.
* ‘Py_UNICODE_strchr’, ‘Py_UNICODE_strrchr’: use *note
PyUnicode_FindChar(): 8b8.
* Removed ‘PyUnicode_GetMax()’. Please migrate to new ( PEP 393(2))
APIs. (Contributed by Inada Naoki in bpo-41103(3).)
* Removed ‘PyLong_FromUnicode()’. Please migrate to *note
PyLong_FromUnicodeObject(): 8b9. (Contributed by Inada Naoki in
bpo-41103(4).)
* Removed ‘PyUnicode_AsUnicodeCopy()’. Please use *note
PyUnicode_AsUCS4Copy(): 8ba. or *note PyUnicode_AsWideCharString():
8bb. (Contributed by Inada Naoki in bpo-41103(5).)
* Removed ‘_Py_CheckRecursionLimit’ variable: it has been replaced by
‘ceval.recursion_limit’ of the *note PyInterpreterState: 8bc.
structure. (Contributed by Victor Stinner in bpo-41834(6).)
* Removed undocumented macros ‘Py_ALLOW_RECURSION’ and
‘Py_END_ALLOW_RECURSION’ and the ‘recursion_critical’ field of the
*note PyInterpreterState: 8bc. structure. (Contributed by Serhiy
Storchaka in bpo-41936(7).)
* Removed the undocumented ‘PyOS_InitInterrupts()’ function.
Initializing Python already implicitly installs signal handlers:
see *note PyConfig.install_signal_handlers: 8bd. (Contributed by
Victor Stinner in bpo-41713(8).)
* Remove the ‘PyAST_Validate()’ function. It is no longer possible
to build a AST object (‘mod_ty’ type) with the public C API. The
function was already excluded from the limited C API ( PEP 384(9)).
(Contributed by Victor Stinner in bpo-43244(10).)
* Remove the ‘symtable.h’ header file and the undocumented functions:
* ‘PyST_GetScope()’
* ‘PySymtable_Build()’
* ‘PySymtable_BuildObject()’
* ‘PySymtable_Free()’
* ‘Py_SymtableString()’
* ‘Py_SymtableStringObject()’
The ‘Py_SymtableString()’ function was part the stable ABI by
mistake but it could not be used, because the ‘symtable.h’ header
file was excluded from the limited C API.
Use Python *note symtable: d8. module instead. (Contributed by
Victor Stinner in bpo-43244(11).)
* Remove *note PyOS_ReadlineFunctionPointer(): 583. from the limited
C API headers and from ‘python3.dll’, the library that provides the
stable ABI on Windows. Since the function takes a ‘FILE*’
argument, its ABI stability cannot be guaranteed. (Contributed by
Petr Viktorin in bpo-43868(12).)
* Remove ‘ast.h’, ‘asdl.h’, and ‘Python-ast.h’ header files. These
functions were undocumented and excluded from the limited C API.
Most names defined by these header files were not prefixed by ‘Py’
and so could create names conflicts. For example, ‘Python-ast.h’
defined a ‘Yield’ macro which was conflict with the ‘Yield’ name
used by the Windows ‘’ header. Use the Python *note
ast: 8. module instead. (Contributed by Victor Stinner in
bpo-43244(13).)
* Remove the compiler and parser functions using ‘struct _mod’ type,
because the public AST C API was removed:
* ‘PyAST_Compile()’
* ‘PyAST_CompileEx()’
* ‘PyAST_CompileObject()’
* ‘PyFuture_FromAST()’
* ‘PyFuture_FromASTObject()’
* ‘PyParser_ASTFromFile()’
* ‘PyParser_ASTFromFileObject()’
* ‘PyParser_ASTFromFilename()’
* ‘PyParser_ASTFromString()’
* ‘PyParser_ASTFromStringObject()’
These functions were undocumented and excluded from the limited C
API. (Contributed by Victor Stinner in bpo-43244(14).)
* Remove the ‘pyarena.h’ header file with functions:
* ‘PyArena_New()’
* ‘PyArena_Free()’
* ‘PyArena_Malloc()’
* ‘PyArena_AddPyObject()’
These functions were undocumented, excluded from the limited C API,
and were only used internally by the compiler. (Contributed by
Victor Stinner in bpo-43244(15).)
* The ‘PyThreadState.use_tracing’ member has been removed to optimize
Python. (Contributed by Mark Shannon in bpo-43760(16).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41123
(2) https://peps.python.org/pep-0393/
(3) https://bugs.python.org/issue?@action=redirect&bpo=41103
(4) https://bugs.python.org/issue?@action=redirect&bpo=41103
(5) https://bugs.python.org/issue?@action=redirect&bpo=41103
(6) https://bugs.python.org/issue?@action=redirect&bpo=41834
(7) https://bugs.python.org/issue?@action=redirect&bpo=41936
(8) https://bugs.python.org/issue?@action=redirect&bpo=41713
(9) https://peps.python.org/pep-0384/
(10) https://bugs.python.org/issue?@action=redirect&bpo=43244
(11) https://bugs.python.org/issue?@action=redirect&bpo=43244
(12) https://bugs.python.org/issue?@action=redirect&bpo=43868
(13) https://bugs.python.org/issue?@action=redirect&bpo=43244
(14) https://bugs.python.org/issue?@action=redirect&bpo=43244
(15) https://bugs.python.org/issue?@action=redirect&bpo=43244
(16) https://bugs.python.org/issue?@action=redirect&bpo=43760
File: python.info, Node: Notable security feature in 3 10 7, Next: Notable security feature in 3 10 8, Prev: C API Changes<4>, Up: What’s New In Python 3 10
1.4.14 Notable security feature in 3.10.7
-----------------------------------------
Converting between *note int: 259. and *note str: 447. in bases other
than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10
(decimal) now raises a *note ValueError: 204. if the number of digits in
string form is above a limit to avoid potential denial of service
attacks due to the algorithmic complexity. This is a mitigation for CVE
2020-10735(1). This limit can be configured or disabled by environment
variable, command line flag, or *note sys: d9. APIs. See the *note
integer string conversion length limitation: 5f1. documentation. The
default limit is 4300 digits in string form.
---------- Footnotes ----------
(1) https://www.cve.org/CVERecord?id=CVE-2020-10735
File: python.info, Node: Notable security feature in 3 10 8, Next: Notable changes in 3 10 12, Prev: Notable security feature in 3 10 7, Up: What’s New In Python 3 10
1.4.15 Notable security feature in 3.10.8
-----------------------------------------
The deprecated ‘mailcap’ module now refuses to inject unsafe text
(filenames, MIME types, parameters) into shell commands. Instead of
using such text, it will warn and act as if a match was not found (or
for test commands, as if the test failed). (Contributed by Petr
Viktorin in gh-98966(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/98966
File: python.info, Node: Notable changes in 3 10 12, Prev: Notable security feature in 3 10 8, Up: What’s New In Python 3 10
1.4.16 Notable changes in 3.10.12
---------------------------------
* Menu:
* tarfile: tarfile<3>.
File: python.info, Node: tarfile<3>, Up: Notable changes in 3 10 12
1.4.16.1 tarfile
................
* The extraction methods in *note tarfile: de, and *note
shutil.unpack_archive(): 467, have a new a 'filter' argument that
allows limiting tar features than may be surprising or dangerous,
such as creating files outside the destination directory. See
*note Extraction filters: 468. for details. In Python 3.12, use
without the 'filter' argument will show a *note DeprecationWarning:
1a5. In Python 3.14, the default will switch to ‘'data'’.
(Contributed by Petr Viktorin in PEP 706(1).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0706/
File: python.info, Node: What’s New In Python 3 9, Next: What’s New In Python 3 8, Prev: What’s New In Python 3 10, Up: What’s New in Python
1.5 What’s New In Python 3.9
============================
Editor: Łukasz Langa
This article explains the new features in Python 3.9, compared to 3.8.
Python 3.9 was released on October 5, 2020. For full details, see the
*note changelog: 13c.
See also
........
PEP 596(1) - Python 3.9 Release Schedule
* Menu:
* Summary – Release highlights: Summary – Release highlights<4>.
* You should check for DeprecationWarning in your code::
* New Features: New Features<9>.
* Other Language Changes: Other Language Changes<5>.
* New Modules: New Modules<5>.
* Improved Modules: Improved Modules<5>.
* Optimizations: Optimizations<5>.
* Deprecated: Deprecated<7>.
* Removed: Removed<7>.
* Porting to Python 3.9: Porting to Python 3 9.
* Build Changes: Build Changes<5>.
* C API Changes: C API Changes<5>.
* Notable changes in Python 3.9.1: Notable changes in Python 3 9 1.
* Notable changes in Python 3.9.2: Notable changes in Python 3 9 2.
* Notable changes in Python 3.9.3: Notable changes in Python 3 9 3.
* Notable changes in Python 3.9.5: Notable changes in Python 3 9 5.
* Notable security feature in 3.9.14: Notable security feature in 3 9 14.
* Notable changes in 3.9.17: Notable changes in 3 9 17.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0596/
File: python.info, Node: Summary – Release highlights<4>, Next: You should check for DeprecationWarning in your code, Up: What’s New In Python 3 9
1.5.1 Summary – Release highlights
----------------------------------
New syntax features:
* PEP 584(1), union operators added to ‘dict’;
* PEP 585(2), type hinting generics in standard collections;
* PEP 614(3), relaxed grammar restrictions on decorators.
New built-in features:
* PEP 616(4), string methods to remove prefixes and suffixes.
New features in the standard library:
* PEP 593(5), flexible function and variable annotations;
* *note os.pidfd_open(): 476. added that allows process management
without races and signals.
Interpreter improvements:
* PEP 573(6), fast access to module state from methods of C extension
types;
* PEP 617(7), CPython now uses a new parser based on PEG;
* a number of Python builtins (range, tuple, set, frozenset, list,
dict) are now sped up using PEP 590(8) vectorcall;
* garbage collection does not block on resurrected objects;
* a number of Python modules (‘_abc’, ‘audioop’, ‘_bz2’, ‘_codecs’,
‘_contextvars’, ‘_crypt’, ‘_functools’, ‘_json’, ‘_locale’, *note
math: 8e, *note operator: 9f, *note resource: bc, *note time: ee,
‘_weakref’) now use multiphase initialization as defined by PEP
489;
* a number of standard library modules (‘audioop’, *note ast: 8,
*note grp: 66, ‘_hashlib’, *note pwd: b2, ‘_posixsubprocess’, *note
random: b8, *note select: c1, *note struct: d5, *note termios: e1,
*note zlib: 133.) are now using the stable ABI defined by PEP 384.
New library modules:
* PEP 615(9), the IANA Time Zone Database is now present in the
standard library in the *note zoneinfo: 134. module;
* an implementation of a topological sort of a graph is now provided
in the new *note graphlib: 65. module.
Release process changes:
* PEP 602(10), CPython adopts an annual release cycle.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0584/
(2) https://peps.python.org/pep-0585/
(3) https://peps.python.org/pep-0614/
(4) https://peps.python.org/pep-0616/
(5) https://peps.python.org/pep-0593/
(6) https://peps.python.org/pep-0573/
(7) https://peps.python.org/pep-0617/
(8) https://peps.python.org/pep-0590/
(9) https://peps.python.org/pep-0615/
(10) https://peps.python.org/pep-0602/
File: python.info, Node: You should check for DeprecationWarning in your code, Next: New Features<9>, Prev: Summary – Release highlights<4>, Up: What’s New In Python 3 9
1.5.2 You should check for DeprecationWarning in your code
----------------------------------------------------------
When Python 2.7 was still supported, a lot of functionality in Python 3
was kept for backward compatibility with Python 2.7. With the end of
Python 2 support, these backward compatibility layers have been removed,
or will be removed soon. Most of them emitted a *note
DeprecationWarning: 1a5. warning for several years. For example, using
‘collections.Mapping’ instead of ‘collections.abc.Mapping’ emits a *note
DeprecationWarning: 1a5. since Python 3.3, released in 2012.
Test your application with the *note -W: 8c6. ‘default’ command-line
option to see *note DeprecationWarning: 1a5. and *note
PendingDeprecationWarning: 8c7, or even with *note -W: 8c6. ‘error’ to
treat them as errors. *note Warnings Filter: 8c8. can be used to ignore
warnings from third-party code.
Python 3.9 is the last version providing those Python 2 backward
compatibility layers, to give more time to Python projects maintainers
to organize the removal of the Python 2 support and add support for
Python 3.9.
Aliases to *note Abstract Base Classes: 875. in the *note collections:
1d. module, like ‘collections.Mapping’ alias to *note
collections.abc.Mapping: 8c9, are kept for one last release for backward
compatibility. They will be removed from Python 3.10.
More generally, try to run your tests in the *note Python Development
Mode: 1fa. which helps to prepare your code to make it compatible with
the next Python version.
Note: a number of pre-existing deprecations were removed in this version
of Python as well. Consult the *note Removed: 8ca. section.
File: python.info, Node: New Features<9>, Next: Other Language Changes<5>, Prev: You should check for DeprecationWarning in your code, Up: What’s New In Python 3 9
1.5.3 New Features
------------------
* Menu:
* Dictionary Merge & Update Operators::
* New String Methods to Remove Prefixes and Suffixes::
* Type Hinting Generics in Standard Collections::
* New Parser::
File: python.info, Node: Dictionary Merge & Update Operators, Next: New String Methods to Remove Prefixes and Suffixes, Up: New Features<9>
1.5.3.1 Dictionary Merge & Update Operators
...........................................
Merge (‘|’) and update (‘|=’) operators have been added to the built-in
*note dict: 258. class. Those complement the existing ‘dict.update’ and
‘{**d1, **d2}’ methods of merging dictionaries.
Example:
>>> x = {"key1": "value1 from x", "key2": "value2 from x"}
>>> y = {"key2": "value2 from y", "key3": "value3 from y"}
>>> x | y
{'key1': 'value1 from x', 'key2': 'value2 from y', 'key3': 'value3 from y'}
>>> y | x
{'key2': 'value2 from x', 'key3': 'value3 from y', 'key1': 'value1 from x'}
See PEP 584(1) for a full description. (Contributed by Brandt Bucher in
bpo-36144(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0584/
(2) https://bugs.python.org/issue?@action=redirect&bpo=36144
File: python.info, Node: New String Methods to Remove Prefixes and Suffixes, Next: Type Hinting Generics in Standard Collections, Prev: Dictionary Merge & Update Operators, Up: New Features<9>
1.5.3.2 New String Methods to Remove Prefixes and Suffixes
..........................................................
*note str.removeprefix(prefix): 8ce. and *note str.removesuffix(suffix):
8cf. have been added to easily remove an unneeded prefix or a suffix
from a string. Corresponding ‘bytes’, ‘bytearray’, and
‘collections.UserString’ methods have also been added. See PEP 616(1)
for a full description. (Contributed by Dennis Sweeney in
bpo-39939(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0616/
(2) https://bugs.python.org/issue?@action=redirect&bpo=39939
File: python.info, Node: Type Hinting Generics in Standard Collections, Next: New Parser, Prev: New String Methods to Remove Prefixes and Suffixes, Up: New Features<9>
1.5.3.3 Type Hinting Generics in Standard Collections
.....................................................
In type annotations you can now use built-in collection types such as
‘list’ and ‘dict’ as generic types instead of importing the
corresponding capitalized types (e.g. ‘List’ or ‘Dict’) from ‘typing’.
Some other types in the standard library are also now generic, for
example ‘queue.Queue’.
Example:
def greet_all(names: list[str]) -> None:
for name in names:
print("Hello", name)
See PEP 585(1) for more details. (Contributed by Guido van Rossum,
Ethan Smith, and Batuhan Taşkaya in bpo-39481(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
(2) https://bugs.python.org/issue?@action=redirect&bpo=39481
File: python.info, Node: New Parser, Prev: Type Hinting Generics in Standard Collections, Up: New Features<9>
1.5.3.4 New Parser
..................
Python 3.9 uses a new parser, based on PEG(1) instead of LL(1)(2). The
new parser’s performance is roughly comparable to that of the old
parser, but the PEG formalism is more flexible than LL(1) when it comes
to designing new language features. We’ll start using this flexibility
in Python 3.10 and later.
The *note ast: 8. module uses the new parser and produces the same AST
as the old parser.
In Python 3.10, the old parser will be deleted and so will all
functionality that depends on it (primarily the ‘parser’ module, which
has long been deprecated). In Python 3.9 'only', you can switch back to
the LL(1) parser using a command line switch (‘-X oldparser’) or an
environment variable (‘PYTHONOLDPARSER=1’).
See PEP 617(3) for more details. (Contributed by Guido van Rossum,
Pablo Galindo and Lysandros Nikolaou in bpo-40334(4).)
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/Parsing_expression_grammar
(2) https://en.wikipedia.org/wiki/LL_parser
(3) https://peps.python.org/pep-0617/
(4) https://bugs.python.org/issue?@action=redirect&bpo=40334
File: python.info, Node: Other Language Changes<5>, Next: New Modules<5>, Prev: New Features<9>, Up: What’s New In Python 3 9
1.5.4 Other Language Changes
----------------------------
* *note __import__(): 8d3. now raises *note ImportError: 415. instead
of *note ValueError: 204, which used to occur when a relative
import went past its top-level package. (Contributed by Ngalim
Siregar in bpo-37444(1).)
* Python now gets the absolute path of the script filename specified
on the command line (ex: ‘python3 script.py’): the ‘__file__’
attribute of the *note __main__: 1. module became an absolute path,
rather than a relative path. These paths now remain valid after
the current directory is changed by *note os.chdir(): 609. As a
side effect, the traceback also displays the absolute path for
*note __main__: 1. module frames in this case. (Contributed by
Victor Stinner in bpo-20443(2).)
* In the *note Python Development Mode: 1fa. and in *note debug
build: 1fb, the 'encoding' and 'errors' arguments are now checked
for string encoding and decoding operations. Examples: *note
open(): 517, *note str.encode(): 8d4. and *note bytes.decode():
8d5.
By default, for best performance, the 'errors' argument is only
checked at the first encoding/decoding error and the 'encoding'
argument is sometimes ignored for empty strings. (Contributed by
Victor Stinner in bpo-37388(3).)
* ‘"".replace("", s, n)’ now returns ‘s’ instead of an empty string
for all non-zero ‘n’. It is now consistent with ‘"".replace("",
s)’. There are similar changes for *note bytes: 1c2. and *note
bytearray: 53a. objects. (Contributed by Serhiy Storchaka in
bpo-28029(4).)
* Any valid expression can now be used as a *note decorator: 72c.
Previously, the grammar was much more restrictive. See PEP 614(5)
for details. (Contributed by Brandt Bucher in bpo-39702(6).)
* Improved help for the *note typing: 104. module. Docstrings are
now shown for all special forms and special generic aliases (like
‘Union’ and ‘List’). Using *note help(): 8d6. with generic alias
like ‘List[int]’ will show the help for the correspondent concrete
type (‘list’ in this case). (Contributed by Serhiy Storchaka in
bpo-40257(7).)
* Parallel running of *note aclose(): 8d7. / *note asend(): 8d8. /
*note athrow(): 4f5. is now prohibited, and ‘ag_running’ now
reflects the actual running status of the async generator.
(Contributed by Yury Selivanov in bpo-30773(8).)
* Unexpected errors in calling the ‘__iter__’ method are no longer
masked by ‘TypeError’ in the *note in: 2ee. operator and functions
*note contains(): 8d9, *note indexOf(): 8da. and *note countOf():
8db. of the *note operator: 9f. module. (Contributed by Serhiy
Storchaka in bpo-40824(9).)
* Unparenthesized lambda expressions can no longer be the expression
part in an ‘if’ clause in comprehensions and generator expressions.
See bpo-41848(10) and bpo-43755(11) for details.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37444
(2) https://bugs.python.org/issue?@action=redirect&bpo=20443
(3) https://bugs.python.org/issue?@action=redirect&bpo=37388
(4) https://bugs.python.org/issue?@action=redirect&bpo=28029
(5) https://peps.python.org/pep-0614/
(6) https://bugs.python.org/issue?@action=redirect&bpo=39702
(7) https://bugs.python.org/issue?@action=redirect&bpo=40257
(8) https://bugs.python.org/issue?@action=redirect&bpo=30773
(9) https://bugs.python.org/issue?@action=redirect&bpo=40824
(10) https://bugs.python.org/issue?@action=redirect&bpo=41848
(11) https://bugs.python.org/issue?@action=redirect&bpo=43755
File: python.info, Node: New Modules<5>, Next: Improved Modules<5>, Prev: Other Language Changes<5>, Up: What’s New In Python 3 9
1.5.5 New Modules
-----------------
* Menu:
* zoneinfo::
* graphlib::
File: python.info, Node: zoneinfo, Next: graphlib, Up: New Modules<5>
1.5.5.1 zoneinfo
................
The *note zoneinfo: 134. module brings support for the IANA time zone
database to the standard library. It adds *note zoneinfo.ZoneInfo: 8de,
a concrete *note datetime.tzinfo: 5da. implementation backed by the
system’s time zone data.
Example:
>>> from zoneinfo import ZoneInfo
>>> from datetime import datetime, timedelta
>>> # Daylight saving time
>>> dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles"))
>>> print(dt)
2020-10-31 12:00:00-07:00
>>> dt.tzname()
'PDT'
>>> # Standard time
>>> dt += timedelta(days=7)
>>> print(dt)
2020-11-07 12:00:00-08:00
>>> print(dt.tzname())
PST
As a fall-back source of data for platforms that don’t ship the IANA
database, the tzdata(1) module was released as a first-party package –
distributed via PyPI and maintained by the CPython core team.
See also
........
PEP 615(2) – Support for the IANA Time Zone Database in the Standard Library
PEP written and implemented by Paul Ganssle
---------- Footnotes ----------
(1) https://pypi.org/project/tzdata/
(2) https://peps.python.org/pep-0615/
File: python.info, Node: graphlib, Prev: zoneinfo, Up: New Modules<5>
1.5.5.2 graphlib
................
A new module, *note graphlib: 65, was added that contains the *note
graphlib.TopologicalSorter: 8e0. class to offer functionality to perform
topological sorting of graphs. (Contributed by Pablo Galindo, Tim
Peters and Larry Hastings in bpo-17005(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17005
File: python.info, Node: Improved Modules<5>, Next: Optimizations<5>, Prev: New Modules<5>, Up: What’s New In Python 3 9
1.5.6 Improved Modules
----------------------
* Menu:
* ast: ast<2>.
* asyncio: asyncio<5>.
* compileall: compileall<2>.
* concurrent.futures: concurrent futures<2>.
* curses: curses<2>.
* datetime: datetime<2>.
* distutils: distutils<3>.
* fcntl: fcntl<2>.
* ftplib: ftplib<2>.
* gc: gc<2>.
* hashlib: hashlib<4>.
* http::
* IDLE and idlelib: IDLE and idlelib<3>.
* imaplib::
* importlib: importlib<3>.
* inspect: inspect<4>.
* ipaddress: ipaddress<2>.
* math: math<4>.
* multiprocessing: multiprocessing<2>.
* nntplib::
* os: os<5>.
* pathlib: pathlib<6>.
* pdb: pdb<3>.
* poplib::
* pprint: pprint<2>.
* pydoc::
* random: random<3>.
* signal::
* smtplib::
* socket: socket<3>.
* time: time<3>.
* sys: sys<6>.
* tracemalloc::
* typing: typing<6>.
* unicodedata: unicodedata<4>.
* venv: venv<3>.
* xml: xml<3>.
File: python.info, Node: ast<2>, Next: asyncio<5>, Up: Improved Modules<5>
1.5.6.1 ast
...........
Added the 'indent' option to *note dump(): 8e3. which allows it to
produce a multiline indented output. (Contributed by Serhiy Storchaka
in bpo-37995(1).)
Added *note ast.unparse(): 8e4. as a function in the *note ast: 8.
module that can be used to unparse an *note ast.AST: 1a6. object and
produce a string with code that would produce an equivalent *note
ast.AST: 1a6. object when parsed. (Contributed by Pablo Galindo and
Batuhan Taskaya in bpo-38870(2).)
Added docstrings to AST nodes that contains the ASDL signature used to
construct that node. (Contributed by Batuhan Taskaya in bpo-39638(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37995
(2) https://bugs.python.org/issue?@action=redirect&bpo=38870
(3) https://bugs.python.org/issue?@action=redirect&bpo=39638
File: python.info, Node: asyncio<5>, Next: compileall<2>, Prev: ast<2>, Up: Improved Modules<5>
1.5.6.2 asyncio
...............
Due to significant security concerns, the 'reuse_address' parameter of
*note asyncio.loop.create_datagram_endpoint(): 72e. is no longer
supported. This is because of the behavior of the socket option
‘SO_REUSEADDR’ in UDP. For more details, see the documentation for
‘loop.create_datagram_endpoint()’. (Contributed by Kyle Stanley,
Antoine Pitrou, and Yury Selivanov in bpo-37228(1).)
Added a new *note coroutine: 48d. *note shutdown_default_executor():
8e6. that schedules a shutdown for the default executor that waits on
the *note ThreadPoolExecutor: 73d. to finish closing. Also, *note
asyncio.run(): 478. has been updated to use the new *note coroutine:
48d. (Contributed by Kyle Stanley in bpo-34037(2).)
Added *note asyncio.PidfdChildWatcher: 475, a Linux-specific child
watcher implementation that polls process file descriptors.
(bpo-38692(3))
Added a new *note coroutine: 48d. *note asyncio.to_thread(): 8e7. It is
mainly used for running IO-bound functions in a separate thread to avoid
blocking the event loop, and essentially works as a high-level version
of *note run_in_executor(): 8e8. that can directly take keyword
arguments. (Contributed by Kyle Stanley and Yury Selivanov in
bpo-32309(4).)
When cancelling the task due to a timeout, *note asyncio.wait_for():
5fb. will now wait until the cancellation is complete also in the case
when 'timeout' is <= 0, like it does with positive timeouts.
(Contributed by Elvis Pranskevichus in bpo-32751(5).)
*note asyncio: a. now raises *note TypeError: 534. when calling
incompatible methods with an *note ssl.SSLSocket: 8e9. socket.
(Contributed by Ido Michael in bpo-37404(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37228
(2) https://bugs.python.org/issue?@action=redirect&bpo=34037
(3) https://bugs.python.org/issue?@action=redirect&bpo=38692
(4) https://bugs.python.org/issue?@action=redirect&bpo=32309
(5) https://bugs.python.org/issue?@action=redirect&bpo=32751
(6) https://bugs.python.org/issue?@action=redirect&bpo=37404
File: python.info, Node: compileall<2>, Next: concurrent futures<2>, Prev: asyncio<5>, Up: Improved Modules<5>
1.5.6.3 compileall
..................
Added new possibility to use hardlinks for duplicated ‘.pyc’ files:
'hardlink_dupes' parameter and –hardlink-dupes command line option.
(Contributed by Lumír ‘Frenzy’ Balhar in bpo-40495(1).)
Added new options for path manipulation in resulting ‘.pyc’ files:
'stripdir', 'prependdir', 'limit_sl_dest' parameters and -s, -p, -e
command line options. Added the possibility to specify the option for
an optimization level multiple times. (Contributed by Lumír ‘Frenzy’
Balhar in bpo-38112(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40495
(2) https://bugs.python.org/issue?@action=redirect&bpo=38112
File: python.info, Node: concurrent futures<2>, Next: curses<2>, Prev: compileall<2>, Up: Improved Modules<5>
1.5.6.4 concurrent.futures
..........................
Added a new 'cancel_futures' parameter to *note
concurrent.futures.Executor.shutdown(): 8ec. that cancels all pending
futures which have not started running, instead of waiting for them to
complete before shutting down the executor. (Contributed by Kyle
Stanley in bpo-39349(1).)
Removed daemon threads from *note ThreadPoolExecutor: 73d. and *note
ProcessPoolExecutor: 8ed. This improves compatibility with
subinterpreters and predictability in their shutdown processes.
(Contributed by Kyle Stanley in bpo-39812(2).)
Workers in *note ProcessPoolExecutor: 8ed. are now spawned on demand,
only when there are no available idle workers to reuse. This optimizes
startup overhead and reduces the amount of lost CPU time to idle
workers. (Contributed by Kyle Stanley in bpo-39207(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39349
(2) https://bugs.python.org/issue?@action=redirect&bpo=39812
(3) https://bugs.python.org/issue?@action=redirect&bpo=39207
File: python.info, Node: curses<2>, Next: datetime<2>, Prev: concurrent futures<2>, Up: Improved Modules<5>
1.5.6.5 curses
..............
Added *note curses.get_escdelay(): 8ef, *note curses.set_escdelay():
8f0, *note curses.get_tabsize(): 8f1, and *note curses.set_tabsize():
8f2. functions. (Contributed by Anthony Sottile in bpo-38312(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38312
File: python.info, Node: datetime<2>, Next: distutils<3>, Prev: curses<2>, Up: Improved Modules<5>
1.5.6.6 datetime
................
The *note isocalendar(): 8f4. of *note datetime.date: 1cd. and *note
isocalendar(): 8f5. of *note datetime.datetime: 1cc. methods now returns
a *note namedtuple(): 1ca. instead of a *note tuple: 36b. (Contributed
by Donghee Na in bpo-24416(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24416
File: python.info, Node: distutils<3>, Next: fcntl<2>, Prev: datetime<2>, Up: Improved Modules<5>
1.5.6.7 distutils
.................
The ‘upload’ command now creates SHA2-256 and Blake2b-256 hash digests.
It skips MD5 on platforms that block MD5 digest. (Contributed by
Christian Heimes in bpo-40698(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40698
File: python.info, Node: fcntl<2>, Next: ftplib<2>, Prev: distutils<3>, Up: Improved Modules<5>
1.5.6.8 fcntl
.............
Added constants ‘fcntl.F_OFD_GETLK’, ‘fcntl.F_OFD_SETLK’ and
‘fcntl.F_OFD_SETLKW’. (Contributed by Donghee Na in bpo-38602(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38602
File: python.info, Node: ftplib<2>, Next: gc<2>, Prev: fcntl<2>, Up: Improved Modules<5>
1.5.6.9 ftplib
..............
*note FTP: 8f9. and *note FTP_TLS: 8fa. now raise a *note ValueError:
204. if the given timeout for their constructor is zero to prevent the
creation of a non-blocking socket. (Contributed by Donghee Na in
bpo-39259(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39259
File: python.info, Node: gc<2>, Next: hashlib<4>, Prev: ftplib<2>, Up: Improved Modules<5>
1.5.6.10 gc
...........
When the garbage collector makes a collection in which some objects
resurrect (they are reachable from outside the isolated cycles after the
finalizers have been executed), do not block the collection of all
objects that are still unreachable. (Contributed by Pablo Galindo and
Tim Peters in bpo-38379(1).)
Added a new function *note gc.is_finalized(): 8fc. to check if an object
has been finalized by the garbage collector. (Contributed by Pablo
Galindo in bpo-39322(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38379
(2) https://bugs.python.org/issue?@action=redirect&bpo=39322
File: python.info, Node: hashlib<4>, Next: http, Prev: gc<2>, Up: Improved Modules<5>
1.5.6.11 hashlib
................
The *note hashlib: 68. module can now use SHA3 hashes and SHAKE XOF from
OpenSSL when available. (Contributed by Christian Heimes in
bpo-37630(1).)
Builtin hash modules can now be disabled with ‘./configure
--without-builtin-hashlib-hashes’ or selectively enabled with e.g.
‘./configure --with-builtin-hashlib-hashes=sha3,blake2’ to force use of
OpenSSL based implementation. (Contributed by Christian Heimes in
bpo-40479(2))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37630
(2) https://bugs.python.org/issue?@action=redirect&bpo=40479
File: python.info, Node: http, Next: IDLE and idlelib<3>, Prev: hashlib<4>, Up: Improved Modules<5>
1.5.6.12 http
.............
HTTP status codes ‘103 EARLY_HINTS’, ‘418 IM_A_TEAPOT’ and ‘425
TOO_EARLY’ are added to *note http.HTTPStatus: 8ff. (Contributed by
Donghee Na in bpo-39509(1) and Ross Rhodes in bpo-39507(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39509
(2) https://bugs.python.org/issue?@action=redirect&bpo=39507
File: python.info, Node: IDLE and idlelib<3>, Next: imaplib, Prev: http, Up: Improved Modules<5>
1.5.6.13 IDLE and idlelib
.........................
Added option to toggle cursor blink off. (Contributed by Zackery Spytz
in bpo-4603(1).)
Escape key now closes IDLE completion windows. (Contributed by Johnny
Najera in bpo-38944(2).)
Added keywords to module name completion list. (Contributed by Terry J.
Reedy in bpo-37765(3).)
New in 3.9 maintenance releases
Make IDLE invoke *note sys.excepthook(): 807. (when started without
‘-n’). User hooks were previously ignored. (Contributed by Ken Hilton
in bpo-43008(4).)
The changes above have been backported to 3.8 maintenance releases.
Rearrange the settings dialog. Split the General tab into Windows and
Shell/Ed tabs. Move help sources, which extend the Help menu, to the
Extensions tab. Make space for new options and shorten the dialog. The
latter makes the dialog better fit small screens. (Contributed by Terry
Jan Reedy in bpo-40468(5).) Move the indent space setting from the Font
tab to the new Windows tab. (Contributed by Mark Roseman and Terry Jan
Reedy in bpo-33962(6).)
Apply syntax highlighting to ‘.pyi’ files. (Contributed by Alex Waygood
and Terry Jan Reedy in bpo-45447(7).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4603
(2) https://bugs.python.org/issue?@action=redirect&bpo=38944
(3) https://bugs.python.org/issue?@action=redirect&bpo=37765
(4) https://bugs.python.org/issue?@action=redirect&bpo=43008
(5) https://bugs.python.org/issue?@action=redirect&bpo=40468
(6) https://bugs.python.org/issue?@action=redirect&bpo=33962
(7) https://bugs.python.org/issue?@action=redirect&bpo=45447
File: python.info, Node: imaplib, Next: importlib<3>, Prev: IDLE and idlelib<3>, Up: Improved Modules<5>
1.5.6.14 imaplib
................
*note IMAP4: 902. and *note IMAP4_SSL: 903. now have an optional
'timeout' parameter for their constructors. Also, the *note open():
904. method now has an optional 'timeout' parameter with this change.
The overridden methods of *note IMAP4_SSL: 903. and *note IMAP4_stream:
905. were applied to this change. (Contributed by Donghee Na in
bpo-38615(1).)
*note imaplib.IMAP4.unselect(): 906. is added. *note
imaplib.IMAP4.unselect(): 906. frees server’s resources associated with
the selected mailbox and returns the server to the authenticated state.
This command performs the same actions as *note imaplib.IMAP4.close():
907, except that no messages are permanently removed from the currently
selected mailbox. (Contributed by Donghee Na in bpo-40375(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38615
(2) https://bugs.python.org/issue?@action=redirect&bpo=40375
File: python.info, Node: importlib<3>, Next: inspect<4>, Prev: imaplib, Up: Improved Modules<5>
1.5.6.15 importlib
..................
To improve consistency with import statements, *note
importlib.util.resolve_name(): 909. now raises *note ImportError: 415.
instead of *note ValueError: 204. for invalid relative import attempts.
(Contributed by Ngalim Siregar in bpo-37444(1).)
Import loaders which publish immutable module objects can now publish
immutable packages in addition to individual modules. (Contributed by
Dino Viehland in bpo-39336(2).)
Added *note importlib.resources.files(): 48a. function with support for
subdirectories in package data, matching backport in
‘importlib_resources’ version 1.5. (Contributed by Jason R. Coombs in
bpo-39791(3).)
Refreshed ‘importlib.metadata’ from ‘importlib_metadata’ version 1.6.1.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37444
(2) https://bugs.python.org/issue?@action=redirect&bpo=39336
(3) https://bugs.python.org/issue?@action=redirect&bpo=39791
File: python.info, Node: inspect<4>, Next: ipaddress<2>, Prev: importlib<3>, Up: Improved Modules<5>
1.5.6.16 inspect
................
*note inspect.BoundArguments.arguments: 90b. is changed from
‘OrderedDict’ to regular dict. (Contributed by Inada Naoki in
bpo-36350(1) and bpo-39775(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36350
(2) https://bugs.python.org/issue?@action=redirect&bpo=39775
File: python.info, Node: ipaddress<2>, Next: math<4>, Prev: inspect<4>, Up: Improved Modules<5>
1.5.6.17 ipaddress
..................
*note ipaddress: 80. now supports IPv6 Scoped Addresses (IPv6 address
with suffix ‘%’).
Scoped IPv6 addresses can be parsed using *note ipaddress.IPv6Address:
1ff. If present, scope zone ID is available through the *note scope_id:
90d. attribute. (Contributed by Oleksandr Pavliuk in bpo-34788(1).)
Starting with Python 3.9.5 the *note ipaddress: 80. module no longer
accepts any leading zeros in IPv4 address strings. (Contributed by
Christian Heimes in bpo-36384(2)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34788
(2) https://bugs.python.org/issue?@action=redirect&bpo=36384
File: python.info, Node: math<4>, Next: multiprocessing<2>, Prev: ipaddress<2>, Up: Improved Modules<5>
1.5.6.18 math
.............
Expanded the *note math.gcd(): 90f. function to handle multiple
arguments. Formerly, it only supported two arguments. (Contributed by
Serhiy Storchaka in bpo-39648(1).)
Added *note math.lcm(): 910.: return the least common multiple of
specified arguments. (Contributed by Mark Dickinson, Ananthakrishnan
and Serhiy Storchaka in bpo-39479(2) and bpo-39648(3).)
Added *note math.nextafter(): 494.: return the next floating-point value
after 'x' towards 'y'. (Contributed by Victor Stinner in bpo-39288(4).)
Added *note math.ulp(): 911.: return the value of the least significant
bit of a float. (Contributed by Victor Stinner in bpo-39310(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39648
(2) https://bugs.python.org/issue?@action=redirect&bpo=39479
(3) https://bugs.python.org/issue?@action=redirect&bpo=39648
(4) https://bugs.python.org/issue?@action=redirect&bpo=39288
(5) https://bugs.python.org/issue?@action=redirect&bpo=39310
File: python.info, Node: multiprocessing<2>, Next: nntplib, Prev: math<4>, Up: Improved Modules<5>
1.5.6.19 multiprocessing
........................
The *note multiprocessing.SimpleQueue: 913. class has a new *note
close(): 914. method to explicitly close the queue. (Contributed by
Victor Stinner in bpo-30966(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30966
File: python.info, Node: nntplib, Next: os<5>, Prev: multiprocessing<2>, Up: Improved Modules<5>
1.5.6.20 nntplib
................
‘NNTP’ and ‘NNTP_SSL’ now raise a *note ValueError: 204. if the given
timeout for their constructor is zero to prevent the creation of a
non-blocking socket. (Contributed by Donghee Na in bpo-39259(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39259
File: python.info, Node: os<5>, Next: pathlib<6>, Prev: nntplib, Up: Improved Modules<5>
1.5.6.21 os
...........
Added *note CLD_KILLED: 917. and *note CLD_STOPPED: 918. for ‘si_code’.
(Contributed by Donghee Na in bpo-38493(1).)
Exposed the Linux-specific *note os.pidfd_open(): 476. (bpo-38692(2))
and *note os.P_PIDFD: 919. (bpo-38713(3)) for process management with
file descriptors.
The *note os.unsetenv(): 91a. function is now also available on Windows.
(Contributed by Victor Stinner in bpo-39413(4).)
The *note os.putenv(): 91b. and *note os.unsetenv(): 91a. functions are
now always available. (Contributed by Victor Stinner in bpo-39395(5).)
Added *note os.waitstatus_to_exitcode(): 91c. function: convert a wait
status to an exit code. (Contributed by Victor Stinner in
bpo-40094(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38493
(2) https://bugs.python.org/issue?@action=redirect&bpo=38692
(3) https://bugs.python.org/issue?@action=redirect&bpo=38713
(4) https://bugs.python.org/issue?@action=redirect&bpo=39413
(5) https://bugs.python.org/issue?@action=redirect&bpo=39395
(6) https://bugs.python.org/issue?@action=redirect&bpo=40094
File: python.info, Node: pathlib<6>, Next: pdb<3>, Prev: os<5>, Up: Improved Modules<5>
1.5.6.22 pathlib
................
Added *note pathlib.Path.readlink(): 91e. which acts similarly to *note
os.readlink(): 91f. (Contributed by Girts Folkmanis in bpo-30618(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30618
File: python.info, Node: pdb<3>, Next: poplib, Prev: pathlib<6>, Up: Improved Modules<5>
1.5.6.23 pdb
............
On Windows now *note Pdb: 921. supports ‘~/.pdbrc’. (Contributed by Tim
Hopper and Dan Lidral-Porter in bpo-20523(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20523
File: python.info, Node: poplib, Next: pprint<2>, Prev: pdb<3>, Up: Improved Modules<5>
1.5.6.24 poplib
...............
*note POP3: 923. and *note POP3_SSL: 924. now raise a *note ValueError:
204. if the given timeout for their constructor is zero to prevent the
creation of a non-blocking socket. (Contributed by Donghee Na in
bpo-39259(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39259
File: python.info, Node: pprint<2>, Next: pydoc, Prev: poplib, Up: Improved Modules<5>
1.5.6.25 pprint
...............
*note pprint: ae. can now pretty-print *note types.SimpleNamespace: 1d1.
(Contributed by Carl Bordum Hansen in bpo-37376(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37376
File: python.info, Node: pydoc, Next: random<3>, Prev: pprint<2>, Up: Improved Modules<5>
1.5.6.26 pydoc
..............
The documentation string is now shown not only for class, function,
method etc, but for any object that has its own *note __doc__: 927.
attribute. (Contributed by Serhiy Storchaka in bpo-40257(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40257
File: python.info, Node: random<3>, Next: signal, Prev: pydoc, Up: Improved Modules<5>
1.5.6.27 random
...............
Added a new *note random.Random.randbytes(): 929. method: generate
random bytes. (Contributed by Victor Stinner in bpo-40286(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40286
File: python.info, Node: signal, Next: smtplib, Prev: random<3>, Up: Improved Modules<5>
1.5.6.28 signal
...............
Exposed the Linux-specific *note signal.pidfd_send_signal(): 92b. for
sending to signals to a process using a file descriptor instead of a
pid. (bpo-38712(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38712
File: python.info, Node: smtplib, Next: socket<3>, Prev: signal, Up: Improved Modules<5>
1.5.6.29 smtplib
................
*note SMTP: 92d. and *note SMTP_SSL: 92e. now raise a *note ValueError:
204. if the given timeout for their constructor is zero to prevent the
creation of a non-blocking socket. (Contributed by Donghee Na in
bpo-39259(1).)
*note LMTP: 92f. constructor now has an optional 'timeout' parameter.
(Contributed by Donghee Na in bpo-39329(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39259
(2) https://bugs.python.org/issue?@action=redirect&bpo=39329
File: python.info, Node: socket<3>, Next: time<3>, Prev: smtplib, Up: Improved Modules<5>
1.5.6.30 socket
...............
The *note socket: cc. module now exports the *note CAN_RAW_JOIN_FILTERS:
931. constant on Linux 4.1 and greater. (Contributed by Stefan
Tatschner and Zackery Spytz in bpo-25780(1).)
The socket module now supports the *note CAN_J1939: 932. protocol on
platforms that support it. (Contributed by Karl Ding in bpo-40291(2).)
The socket module now has the *note socket.send_fds(): 933. and *note
socket.recv_fds(): 934. functions. (Contributed by Joannah Nanjekye,
Shinya Okano and Victor Stinner in bpo-28724(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25780
(2) https://bugs.python.org/issue?@action=redirect&bpo=40291
(3) https://bugs.python.org/issue?@action=redirect&bpo=28724
File: python.info, Node: time<3>, Next: sys<6>, Prev: socket<3>, Up: Improved Modules<5>
1.5.6.31 time
.............
On AIX, *note thread_time(): 936. is now implemented with
‘thread_cputime()’ which has nanosecond resolution, rather than
‘clock_gettime(CLOCK_THREAD_CPUTIME_ID)’ which has a resolution of 10
milliseconds. (Contributed by Batuhan Taskaya in bpo-40192(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40192
File: python.info, Node: sys<6>, Next: tracemalloc, Prev: time<3>, Up: Improved Modules<5>
1.5.6.32 sys
............
Added a new *note sys.platlibdir: 938. attribute: name of the
platform-specific library directory. It is used to build the path of
standard library and the paths of installed extension modules. It is
equal to ‘"lib"’ on most platforms. On Fedora and SuSE, it is equal to
‘"lib64"’ on 64-bit platforms. (Contributed by Jan Matějek, Matěj Cepl,
Charalampos Stratakis and Victor Stinner in bpo-1294959(1).)
Previously, *note sys.stderr: 939. was block-buffered when
non-interactive. Now ‘stderr’ defaults to always being line-buffered.
(Contributed by Jendrik Seipp in bpo-13601(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1294959
(2) https://bugs.python.org/issue?@action=redirect&bpo=13601
File: python.info, Node: tracemalloc, Next: typing<6>, Prev: sys<6>, Up: Improved Modules<5>
1.5.6.33 tracemalloc
....................
Added *note tracemalloc.reset_peak(): 93b. to set the peak size of
traced memory blocks to the current size, to measure the peak of
specific pieces of code. (Contributed by Huon Wilson in bpo-40630(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=40630
File: python.info, Node: typing<6>, Next: unicodedata<4>, Prev: tracemalloc, Up: Improved Modules<5>
1.5.6.34 typing
...............
PEP 593(1) introduced an *note typing.Annotated: 93d. type to decorate
existing types with context-specific metadata and new ‘include_extras’
parameter to *note typing.get_type_hints(): 6ad. to access the metadata
at runtime. (Contributed by Till Varoquaux and Konstantin Kashin.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0593/
File: python.info, Node: unicodedata<4>, Next: venv<3>, Prev: typing<6>, Up: Improved Modules<5>
1.5.6.35 unicodedata
....................
The Unicode database has been updated to version 13.0.0.
(bpo-39926(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39926
File: python.info, Node: venv<3>, Next: xml<3>, Prev: unicodedata<4>, Up: Improved Modules<5>
1.5.6.36 venv
.............
The activation scripts provided by *note venv: 111. now all specify
their prompt customization consistently by always using the value
specified by ‘__VENV_PROMPT__’. Previously some scripts unconditionally
used ‘__VENV_PROMPT__’, others only if it happened to be set (which was
the default case), and one used ‘__VENV_NAME__’ instead. (Contributed
by Brett Cannon in bpo-37663(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37663
File: python.info, Node: xml<3>, Prev: venv<3>, Up: Improved Modules<5>
1.5.6.37 xml
............
White space characters within attributes are now preserved when
serializing *note xml.etree.ElementTree: 125. to XML file. EOLNs are no
longer normalized to “n”. This is the result of discussion about how to
interpret section 2.11 of XML spec. (Contributed by Mefistotelis in
bpo-39011(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39011
File: python.info, Node: Optimizations<5>, Next: Deprecated<7>, Prev: Improved Modules<5>, Up: What’s New In Python 3 9
1.5.7 Optimizations
-------------------
* Optimized the idiom for assignment a temporary variable in
comprehensions. Now ‘for y in [expr]’ in comprehensions is as fast
as a simple assignment ‘y = expr’. For example:
sums = [s for s in [0] for x in data for s in [s + x]]
Unlike the ‘:=’ operator this idiom does not leak a variable to the
outer scope.
(Contributed by Serhiy Storchaka in bpo-32856(1).)
* Optimized signal handling in multithreaded applications. If a
thread different than the main thread gets a signal, the bytecode
evaluation loop is no longer interrupted at each bytecode
instruction to check for pending signals which cannot be handled.
Only the main thread of the main interpreter can handle signals.
Previously, the bytecode evaluation loop was interrupted at each
instruction until the main thread handles signals. (Contributed by
Victor Stinner in bpo-40010(2).)
* Optimized the *note subprocess: d6. module on FreeBSD using
‘closefrom()’. (Contributed by Ed Maste, Conrad Meyer, Kyle Evans,
Kubilay Kocak and Victor Stinner in bpo-38061(3).)
* *note PyLong_FromDouble(): 942. is now up to 1.87x faster for
values that fit into long. (Contributed by Sergey Fedoseev in
bpo-37986(4).)
* A number of Python builtins (*note range: 943, *note tuple: 36b,
*note set: 5d5, *note frozenset: 5d6, *note list: 60d, *note dict:
258.) are now sped up by using PEP 590(5) vectorcall protocol.
(Contributed by Donghee Na, Mark Shannon, Jeroen Demeyer and Petr
Viktorin in bpo-37207(6).)
* Optimized ‘set.difference_update()’ for the case when the other set
is much larger than the base set. (Suggested by Evgeny Kapun with
code contributed by Michele Orrù in bpo-8425(7).)
* Python’s small object allocator (‘obmalloc.c’) now allows (no more
than) one empty arena to remain available for immediate reuse,
without returning it to the OS. This prevents thrashing in simple
loops where an arena could be created and destroyed anew on each
iteration. (Contributed by Tim Peters in bpo-37257(8).)
* *note floor division: 944. of float operation now has a better
performance. Also the message of *note ZeroDivisionError: 945. for
this operation is updated. (Contributed by Donghee Na in
bpo-39434(9).)
* Decoding short ASCII strings with UTF-8 and ascii codecs is now
about 15% faster. (Contributed by Inada Naoki in bpo-37348(10).)
Here’s a summary of performance improvements from Python 3.4 through
Python 3.9:
Python version 3.4 3.5 3.6 3.7 3.8 3.9
-------------- --- --- --- --- --- ---
Variable and attribute read access:
read_local 7.1 7.1 5.4 5.1 3.9 3.9
read_nonlocal 7.1 8.1 5.8 5.4 4.4 4.5
read_global 15.5 19.0 14.3 13.6 7.6 7.8
read_builtin 21.1 21.6 18.5 19.0 7.5 7.8
read_classvar_from_class 25.6 26.5 20.7 19.5 18.4 17.9
read_classvar_from_instance 22.8 23.5 18.8 17.1 16.4 16.9
read_instancevar 32.4 33.1 28.0 26.3 25.4 25.3
read_instancevar_slots 27.8 31.3 20.8 20.8 20.2 20.5
read_namedtuple 73.8 57.5 45.0 46.8 18.4 18.7
read_boundmethod 37.6 37.9 29.6 26.9 27.7 41.1
Variable and attribute write access:
write_local 8.7 9.3 5.5 5.3 4.3 4.3
write_nonlocal 10.5 11.1 5.6 5.5 4.7 4.8
write_global 19.7 21.2 18.0 18.0 15.8 16.7
write_classvar 92.9 96.0 104.6 102.1 39.2 39.8
write_instancevar 44.6 45.8 40.0 38.9 35.5 37.4
write_instancevar_slots 35.6 36.1 27.3 26.6 25.7 25.8
Data structure read access:
read_list 24.2 24.5 20.8 20.8 19.0 19.5
read_deque 24.7 25.5 20.2 20.6 19.8 20.2
read_dict 24.3 25.7 22.3 23.0 21.0 22.4
read_strdict 22.6 24.3 19.5 21.2 18.9 21.5
Data structure write access:
write_list 27.1 28.5 22.5 21.6 20.0 20.0
write_deque 28.7 30.1 22.7 21.8 23.5 21.7
write_dict 31.4 33.3 29.3 29.2 24.7 25.4
write_strdict 28.4 29.9 27.5 25.2 23.1 24.5
Stack (or queue) operations:
list_append_pop 93.4 112.7 75.4 74.2 50.8 50.6
deque_append_pop 43.5 57.0 49.4 49.2 42.5 44.2
deque_append_popleft 43.7 57.3 49.7 49.7 42.8 46.4
Timing loop:
loop_overhead 0.5 0.6 0.4 0.3 0.3 0.3
These results were generated from the variable access benchmark script
at: ‘Tools/scripts/var_access_benchmark.py’. The benchmark script
displays timings in nanoseconds. The benchmarks were measured on an
Intel® Core™ i7-4960HQ processor(11) running the macOS 64-bit builds
found at python.org(12).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32856
(2) https://bugs.python.org/issue?@action=redirect&bpo=40010
(3) https://bugs.python.org/issue?@action=redirect&bpo=38061
(4) https://bugs.python.org/issue?@action=redirect&bpo=37986
(5) https://peps.python.org/pep-0590/
(6) https://bugs.python.org/issue?@action=redirect&bpo=37207
(7) https://bugs.python.org/issue?@action=redirect&bpo=8425
(8) https://bugs.python.org/issue?@action=redirect&bpo=37257
(9) https://bugs.python.org/issue?@action=redirect&bpo=39434
(10) https://bugs.python.org/issue?@action=redirect&bpo=37348
(11)
https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html
(12) https://www.python.org/downloads/macos/
File: python.info, Node: Deprecated<7>, Next: Removed<7>, Prev: Optimizations<5>, Up: What’s New In Python 3 9
1.5.8 Deprecated
----------------
* The distutils ‘bdist_msi’ command is now deprecated, use
‘bdist_wheel’ (wheel packages) instead. (Contributed by Hugo van
Kemenade in bpo-39586(1).)
* Currently *note math.factorial(): 947. accepts *note float: 2f1.
instances with non-negative integer values (like ‘5.0’). It raises
a *note ValueError: 204. for non-integral and negative floats. It
is now deprecated. In future Python versions it will raise a *note
TypeError: 534. for all floats. (Contributed by Serhiy Storchaka
in bpo-37315(2).)
* The ‘parser’ and ‘symbol’ modules are deprecated and will be
removed in future versions of Python. For the majority of use
cases, users can leverage the Abstract Syntax Tree (AST) generation
and compilation stage, using the *note ast: 8. module.
* The Public C API functions ‘PyParser_SimpleParseStringFlags()’,
‘PyParser_SimpleParseStringFlagsFilename()’,
‘PyParser_SimpleParseFileFlags()’ and ‘PyNode_Compile()’ are
deprecated and will be removed in Python 3.10 together with the old
parser.
* Using *note NotImplemented: 7cd. in a boolean context has been
deprecated, as it is almost exclusively the result of incorrect
rich comparator implementations. It will be made a *note
TypeError: 534. in a future version of Python. (Contributed by
Josh Rosenberg in bpo-35712(3).)
* The *note random: b8. module currently accepts any hashable type as
a possible seed value. Unfortunately, some of those types are not
guaranteed to have a deterministic hash value. After Python 3.9,
the module will restrict its seeds to *note None: 671, *note int:
259, *note float: 2f1, *note str: 447, *note bytes: 1c2, and *note
bytearray: 53a.
* Opening the *note GzipFile: 416. file for writing without
specifying the 'mode' argument is deprecated. In future Python
versions it will always be opened for reading by default. Specify
the 'mode' argument for opening it for writing and silencing a
warning. (Contributed by Serhiy Storchaka in bpo-28286(4).)
* Deprecated the ‘split()’ method of ‘_tkinter.TkappType’ in favour
of the ‘splitlist()’ method which has more consistent and
predictable behavior. (Contributed by Serhiy Storchaka in
bpo-38371(5).)
* The explicit passing of coroutine objects to *note asyncio.wait():
47b. has been deprecated and will be removed in version 3.11.
(Contributed by Yury Selivanov and Kyle Stanley in bpo-34790(6).)
* binhex4 and hexbin4 standards are now deprecated. The ‘binhex’
module and the following *note binascii: 10. functions are now
deprecated:
* ‘b2a_hqx()’, ‘a2b_hqx()’
* ‘rlecode_hqx()’, ‘rledecode_hqx()’
(Contributed by Victor Stinner in bpo-39353(7).)
* *note ast: 8. classes ‘slice’, ‘Index’ and ‘ExtSlice’ are
considered deprecated and will be removed in future Python
versions. ‘value’ itself should be used instead of ‘Index(value)’.
‘Tuple(slices, Load())’ should be used instead of
‘ExtSlice(slices)’. (Contributed by Serhiy Storchaka in
bpo-34822(8).)
* *note ast: 8. classes ‘Suite’, ‘Param’, ‘AugLoad’ and ‘AugStore’
are considered deprecated and will be removed in future Python
versions. They were not generated by the parser and not accepted
by the code generator in Python 3. (Contributed by Batuhan Taskaya
in bpo-39639(9) and bpo-39969(10) and Serhiy Storchaka in
bpo-39988(11).)
* The ‘PyEval_InitThreads()’ and ‘PyEval_ThreadsInitialized()’
functions are now deprecated and will be removed in Python 3.11.
Calling ‘PyEval_InitThreads()’ now does nothing. The *note GIL:
159. is initialized by *note Py_Initialize(): 8ab. since Python
3.7. (Contributed by Victor Stinner in bpo-39877(12).)
* Passing ‘None’ as the first argument to the *note shlex.split():
538. function has been deprecated. (Contributed by Zackery Spytz
in bpo-33262(13).)
* ‘smtpd.MailmanProxy()’ is now deprecated as it is unusable without
an external module, ‘mailman’. (Contributed by Samuel Colvin in
bpo-35800(14).)
* The ‘lib2to3’ module now emits a *note PendingDeprecationWarning:
8c7. Python 3.9 switched to a PEG parser (see PEP 617(15)), and
Python 3.10 may include new language syntax that is not parsable by
lib2to3’s LL(1) parser. The ‘lib2to3’ module may be removed from
the standard library in a future Python version. Consider
third-party alternatives such as LibCST(16) or parso(17).
(Contributed by Carl Meyer in bpo-40360(18).)
* The 'random' parameter of *note random.shuffle(): 742. has been
deprecated. (Contributed by Raymond Hettinger in bpo-40465(19))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39586
(2) https://bugs.python.org/issue?@action=redirect&bpo=37315
(3) https://bugs.python.org/issue?@action=redirect&bpo=35712
(4) https://bugs.python.org/issue?@action=redirect&bpo=28286
(5) https://bugs.python.org/issue?@action=redirect&bpo=38371
(6) https://bugs.python.org/issue?@action=redirect&bpo=34790
(7) https://bugs.python.org/issue?@action=redirect&bpo=39353
(8) https://bugs.python.org/issue?@action=redirect&bpo=34822
(9) https://bugs.python.org/issue?@action=redirect&bpo=39639
(10) https://bugs.python.org/issue?@action=redirect&bpo=39969
(11) https://bugs.python.org/issue?@action=redirect&bpo=39988
(12) https://bugs.python.org/issue?@action=redirect&bpo=39877
(13) https://bugs.python.org/issue?@action=redirect&bpo=33262
(14) https://bugs.python.org/issue?@action=redirect&bpo=35800
(15) https://peps.python.org/pep-0617/
(16) https://libcst.readthedocs.io/
(17) https://parso.readthedocs.io/
(18) https://bugs.python.org/issue?@action=redirect&bpo=40360
(19) https://bugs.python.org/issue?@action=redirect&bpo=40465
File: python.info, Node: Removed<7>, Next: Porting to Python 3 9, Prev: Deprecated<7>, Up: What’s New In Python 3 9
1.5.9 Removed
-------------
* The erroneous version at ‘unittest.mock.__version__’ has been
removed.
* ‘nntplib.NNTP’: ‘xpath()’ and ‘xgtitle()’ methods have been
removed. These methods are deprecated since Python 3.3.
Generally, these extensions are not supported or not enabled by
NNTP server administrators. For ‘xgtitle()’, please use
‘nntplib.NNTP.descriptions()’ or ‘nntplib.NNTP.description()’
instead. (Contributed by Donghee Na in bpo-39366(1).)
* *note array.array: 1a0.: ‘tostring()’ and ‘fromstring()’ methods
have been removed. They were aliases to ‘tobytes()’ and
‘frombytes()’, deprecated since Python 3.2. (Contributed by Victor
Stinner in bpo-38916(2).)
* The undocumented ‘sys.callstats()’ function has been removed.
Since Python 3.7, it was deprecated and always returned *note None:
671. It required a special build option ‘CALL_PROFILE’ which was
already removed in Python 3.7. (Contributed by Victor Stinner in
bpo-37414(3).)
* The ‘sys.getcheckinterval()’ and ‘sys.setcheckinterval()’ functions
have been removed. They were deprecated since Python 3.2. Use
*note sys.getswitchinterval(): 949. and *note
sys.setswitchinterval(): 94a. instead. (Contributed by Victor
Stinner in bpo-37392(4).)
* The C function ‘PyImport_Cleanup()’ has been removed. It was
documented as: “Empty the module table. For internal use only.”
(Contributed by Victor Stinner in bpo-36710(5).)
* ‘_dummy_thread’ and ‘dummy_threading’ modules have been removed.
These modules were deprecated since Python 3.7 which requires
threading support. (Contributed by Victor Stinner in
bpo-37312(6).)
* ‘aifc.openfp()’ alias to ‘aifc.open()’, ‘sunau.openfp()’ alias to
‘sunau.open()’, and ‘wave.openfp()’ alias to *note wave.open():
94b. have been removed. They were deprecated since Python 3.7.
(Contributed by Victor Stinner in bpo-37320(7).)
* The ‘isAlive()’ method of *note threading.Thread: 94c. has been
removed. It was deprecated since Python 3.8. Use *note
is_alive(): 94d. instead. (Contributed by Donghee Na in
bpo-37804(8).)
* Methods ‘getchildren()’ and ‘getiterator()’ of classes *note
ElementTree: 94e. and *note Element: 30a. in the *note ElementTree:
125. module have been removed. They were deprecated in Python 3.2.
Use ‘iter(x)’ or ‘list(x)’ instead of ‘x.getchildren()’ and
‘x.iter()’ or ‘list(x.iter())’ instead of ‘x.getiterator()’.
(Contributed by Serhiy Storchaka in bpo-36543(9).)
* The old *note plistlib: ab. API has been removed, it was deprecated
since Python 3.4. Use the *note load(): 94f, *note loads(): 950,
*note dump(): 951, and *note dumps(): 952. functions.
Additionally, the 'use_builtin_types' parameter was removed,
standard *note bytes: 1c2. objects are always used instead.
(Contributed by Jon Janzen in bpo-36409(10).)
* The C function ‘PyGen_NeedsFinalizing’ has been removed. It was
not documented, tested, or used anywhere within CPython after the
implementation of PEP 442(11). Patch by Joannah Nanjekye.
(Contributed by Joannah Nanjekye in bpo-15088(12))
* ‘base64.encodestring()’ and ‘base64.decodestring()’, aliases
deprecated since Python 3.1, have been removed: use *note
base64.encodebytes(): 953. and *note base64.decodebytes(): 954.
instead. (Contributed by Victor Stinner in bpo-39351(13).)
* ‘fractions.gcd()’ function has been removed, it was deprecated
since Python 3.5 (bpo-22486(14)): use *note math.gcd(): 90f.
instead. (Contributed by Victor Stinner in bpo-39350(15).)
* The 'buffering' parameter of *note bz2.BZ2File: 863. has been
removed. Since Python 3.0, it was ignored and using it emitted a
*note DeprecationWarning: 1a5. Pass an open file object to control
how the file is opened. (Contributed by Victor Stinner in
bpo-39357(16).)
* The 'encoding' parameter of *note json.loads(): 955. has been
removed. As of Python 3.1, it was deprecated and ignored; using it
has emitted a *note DeprecationWarning: 1a5. since Python 3.8.
(Contributed by Inada Naoki in bpo-39377(17))
* ‘with (await asyncio.lock):’ and ‘with (yield from asyncio.lock):’
statements are not longer supported, use ‘async with lock’ instead.
The same is correct for ‘asyncio.Condition’ and
‘asyncio.Semaphore’. (Contributed by Andrew Svetlov in
bpo-34793(18).)
* The ‘sys.getcounts()’ function, the ‘-X showalloccount’ command
line option and the ‘show_alloc_count’ field of the C structure
*note PyConfig: 3a2. have been removed. They required a special
Python build by defining ‘COUNT_ALLOCS’ macro. (Contributed by
Victor Stinner in bpo-39489(19).)
* The ‘_field_types’ attribute of the *note typing.NamedTuple: 2b2.
class has been removed. It was deprecated since Python 3.8. Use
the ‘__annotations__’ attribute instead. (Contributed by Serhiy
Storchaka in bpo-40182(20).)
* The ‘symtable.SymbolTable.has_exec()’ method has been removed. It
was deprecated since 2006, and only returning ‘False’ when it’s
called. (Contributed by Batuhan Taskaya in bpo-40208(21))
* The ‘asyncio.Task.current_task()’ and ‘asyncio.Task.all_tasks()’
have been removed. They were deprecated since Python 3.7 and you
can use *note asyncio.current_task(): 479. and *note
asyncio.all_tasks(): 956. instead. (Contributed by Rémi Lapeyre in
bpo-40967(22))
* The ‘unescape()’ method in the *note html.parser.HTMLParser: 874.
class has been removed (it was deprecated since Python 3.4). *note
html.unescape(): 957. should be used for converting character
references to the corresponding unicode characters.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39366
(2) https://bugs.python.org/issue?@action=redirect&bpo=38916
(3) https://bugs.python.org/issue?@action=redirect&bpo=37414
(4) https://bugs.python.org/issue?@action=redirect&bpo=37392
(5) https://bugs.python.org/issue?@action=redirect&bpo=36710
(6) https://bugs.python.org/issue?@action=redirect&bpo=37312
(7) https://bugs.python.org/issue?@action=redirect&bpo=37320
(8) https://bugs.python.org/issue?@action=redirect&bpo=37804
(9) https://bugs.python.org/issue?@action=redirect&bpo=36543
(10) https://bugs.python.org/issue?@action=redirect&bpo=36409
(11) https://peps.python.org/pep-0442/
(12) https://bugs.python.org/issue?@action=redirect&bpo=15088
(13) https://bugs.python.org/issue?@action=redirect&bpo=39351
(14) https://bugs.python.org/issue?@action=redirect&bpo=22486
(15) https://bugs.python.org/issue?@action=redirect&bpo=39350
(16) https://bugs.python.org/issue?@action=redirect&bpo=39357
(17) https://bugs.python.org/issue?@action=redirect&bpo=39377
(18) https://bugs.python.org/issue?@action=redirect&bpo=34793
(19) https://bugs.python.org/issue?@action=redirect&bpo=39489
(20) https://bugs.python.org/issue?@action=redirect&bpo=40182
(21) https://bugs.python.org/issue?@action=redirect&bpo=40208
(22) https://bugs.python.org/issue?@action=redirect&bpo=40967
File: python.info, Node: Porting to Python 3 9, Next: Build Changes<5>, Prev: Removed<7>, Up: What’s New In Python 3 9
1.5.10 Porting to Python 3.9
----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in the Python API: Changes in the Python API<4>.
* Changes in the C API: Changes in the C API<3>.
* CPython bytecode changes: CPython bytecode changes<4>.
File: python.info, Node: Changes in the Python API<4>, Next: Changes in the C API<3>, Up: Porting to Python 3 9
1.5.10.1 Changes in the Python API
..................................
* *note __import__(): 8d3. and *note importlib.util.resolve_name():
909. now raise *note ImportError: 415. where it previously raised
*note ValueError: 204. Callers catching the specific exception
type and supporting both Python 3.9 and earlier versions will need
to catch both using ‘except (ImportError, ValueError):’.
* The *note venv: 111. activation scripts no longer special-case when
‘__VENV_PROMPT__’ is set to ‘""’.
* The *note select.epoll.unregister(): 95a. method no longer ignores
the *note EBADF: 95b. error. (Contributed by Victor Stinner in
bpo-39239(1).)
* The 'compresslevel' parameter of *note bz2.BZ2File: 863. became
keyword-only, since the 'buffering' parameter has been removed.
(Contributed by Victor Stinner in bpo-39357(2).)
* Simplified AST for subscription. Simple indices will be
represented by their value, extended slices will be represented as
tuples. ‘Index(value)’ will return a ‘value’ itself,
‘ExtSlice(slices)’ will return ‘Tuple(slices, Load())’.
(Contributed by Serhiy Storchaka in bpo-34822(3).)
* The *note importlib: 77. module now ignores the *note PYTHONCASEOK:
95c. environment variable when the *note -E: 95d. or *note -I: 95e.
command line options are being used.
* The 'encoding' parameter has been added to the classes *note
ftplib.FTP: 8f9. and *note ftplib.FTP_TLS: 8fa. as a keyword-only
parameter, and the default encoding is changed from Latin-1 to
UTF-8 to follow RFC 2640(4).
* *note asyncio.loop.shutdown_default_executor(): 8e6. has been added
to *note AbstractEventLoop: 95f, meaning alternative event loops
that inherit from it should have this method defined. (Contributed
by Kyle Stanley in bpo-34037(5).)
* The constant values of future flags in the *note __future__: 0.
module is updated in order to prevent collision with compiler
flags. Previously ‘PyCF_ALLOW_TOP_LEVEL_AWAIT’ was clashing with
‘CO_FUTURE_DIVISION’. (Contributed by Batuhan Taskaya in
bpo-39562(6))
* ‘array('u')’ now uses ‘wchar_t’ as C type instead of ‘Py_UNICODE’.
This change doesn’t affect to its behavior because ‘Py_UNICODE’ is
alias of ‘wchar_t’ since Python 3.3. (Contributed by Inada Naoki
in bpo-34538(7).)
* The *note logging.getLogger(): 960. API now returns the root logger
when passed the name ‘'root'’, whereas previously it returned a
non-root logger named ‘'root'’. This could affect cases where user
code explicitly wants a non-root logger named ‘'root'’, or
instantiates a logger using ‘logging.getLogger(__name__)’ in some
top-level module called ‘'root.py'’. (Contributed by Vinay Sajip
in bpo-37742(8).)
* Division handling of *note PurePath: 4a2. now returns *note
NotImplemented: 7cd. instead of raising a *note TypeError: 534.
when passed something other than an instance of ‘str’ or *note
PurePath: 4a2. This allows creating compatible classes that don’t
inherit from those mentioned types. (Contributed by Roger Aiudi in
bpo-34775(9)).
* Starting with Python 3.9.5 the *note ipaddress: 80. module no
longer accepts any leading zeros in IPv4 address strings. Leading
zeros are ambiguous and interpreted as octal notation by some
libraries. For example the legacy function *note
socket.inet_aton(): 961. treats leading zeros as octal notatation.
glibc implementation of modern *note inet_pton(): 962. does not
accept any leading zeros. (Contributed by Christian Heimes in
bpo-36384(10)).
* *note codecs.lookup(): 963. now normalizes the encoding name the
same way as *note encodings.normalize_encoding(): 7f7, except that
*note codecs.lookup(): 963. also converts the name to lower case.
For example, ‘"latex+latin1"’ encoding name is now normalized to
‘"latex_latin1"’. (Contributed by Jordon Xu in bpo-37751(11).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=39239
(2) https://bugs.python.org/issue?@action=redirect&bpo=39357
(3) https://bugs.python.org/issue?@action=redirect&bpo=34822
(4) https://datatracker.ietf.org/doc/html/rfc2640.html
(5) https://bugs.python.org/issue?@action=redirect&bpo=34037
(6) https://bugs.python.org/issue?@action=redirect&bpo=39562
(7) https://bugs.python.org/issue?@action=redirect&bpo=34538
(8) https://bugs.python.org/issue?@action=redirect&bpo=37742
(9) https://bugs.python.org/issue?@action=redirect&bpo=34775
(10) https://bugs.python.org/issue?@action=redirect&bpo=36384
(11) https://bugs.python.org/issue?@action=redirect&bpo=37751
File: python.info, Node: Changes in the C API<3>, Next: CPython bytecode changes<4>, Prev: Changes in the Python API<4>, Up: Porting to Python 3 9
1.5.10.2 Changes in the C API
.............................
* Instances of *note heap-allocated types: 965. (such as those
created with *note PyType_FromSpec(): 57a. and similar APIs) hold a
reference to their type object since Python 3.8. As indicated in
the “Changes in the C API” of Python 3.8, for the vast majority of
cases, there should be no side effect but for types that have a
custom *note tp_traverse: 779. function, ensure that all custom
‘tp_traverse’ functions of heap-allocated types visit the object’s
type.
Example:
int
foo_traverse(foo_struct *self, visitproc visit, void *arg) {
// Rest of the traverse function
#if PY_VERSION_HEX >= 0x03090000
// This was not needed before Python 3.9 (Python issue 35810 and 40217)
Py_VISIT(Py_TYPE(self));
#endif
}
If your traverse function delegates to ‘tp_traverse’ of its base
class (or another type), ensure that ‘Py_TYPE(self)’ is visited
only once. Note that only *note heap type: 965. are expected to
visit the type in ‘tp_traverse’.
For example, if your ‘tp_traverse’ function includes:
base->tp_traverse(self, visit, arg)
then add:
#if PY_VERSION_HEX >= 0x03090000
// This was not needed before Python 3.9 (bpo-35810 and bpo-40217)
if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) {
// a heap type's tp_traverse already visited Py_TYPE(self)
} else {
Py_VISIT(Py_TYPE(self));
}
#else
(See bpo-35810(1) and bpo-40217(2) for more information.)
* The functions ‘PyEval_CallObject’, ‘PyEval_CallFunction’,
‘PyEval_CallMethod’ and ‘PyEval_CallObjectWithKeywords’ are
deprecated. Use *note PyObject_Call(): 398. and its variants
instead. (See more details in bpo-29548(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35810
(2) https://bugs.python.org/issue?@action=redirect&bpo=40217
(3) https://bugs.python.org/issue?@action=redirect&bpo=29548
File: python.info, Node: CPython bytecode changes<4>, Prev: Changes in the C API<3>, Up: Porting to Python 3 9
1.5.10.3 CPython bytecode changes
.................................
* The *note LOAD_ASSERTION_ERROR: 967. opcode was added for handling
the *note assert: 968. statement. Previously, the assert statement
would not work correctly if the *note AssertionError: 6a5.
exception was being shadowed. (Contributed by Zackery Spytz in
bpo-34880(1).)
* The *note COMPARE_OP: 969. opcode was split into four distinct
instructions:
* ‘COMPARE_OP’ for rich comparisons
* ‘IS_OP’ for ‘is’ and ‘is not’ tests
* ‘CONTAINS_OP’ for ‘in’ and ‘not in’ tests
* ‘JUMP_IF_NOT_EXC_MATCH’ for checking exceptions in
‘try-except’ statements.
(Contributed by Mark Shannon in bpo-39156(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34880
(2) https://bugs.python.org/issue?@action=redirect&bpo=39156
File: python.info, Node: Build Changes<5>, Next: C API Changes<5>, Prev: Porting to Python 3 9, Up: What’s New In Python 3 9
1.5.11 Build Changes
--------------------
* Added ‘--with-platlibdir’ option to the ‘configure’ script: name of
the platform-specific library directory, stored in the new *note
sys.platlibdir: 938. attribute. See *note sys.platlibdir: 938.
attribute for more information. (Contributed by Jan Matějek, Matěj
Cepl, Charalampos Stratakis and Victor Stinner in bpo-1294959(1).)
* The ‘COUNT_ALLOCS’ special build macro has been removed.
(Contributed by Victor Stinner in bpo-39489(2).)
* On non-Windows platforms, the ‘setenv()’ and ‘unsetenv()’ functions
are now required to build Python. (Contributed by Victor Stinner
in bpo-39395(3).)
* On non-Windows platforms, creating ‘bdist_wininst’ installers is
now officially unsupported. (See bpo-10945(4) for more details.)
* When building Python on macOS from source, ‘_tkinter’ now links
with non-system Tcl and Tk frameworks if they are installed in
‘/Library/Frameworks’, as had been the case on older releases of
macOS. If a macOS SDK is explicitly configured, by using *note
-enable-universalsdk: 96b. or ‘-isysroot’, only the SDK itself is
searched. The default behavior can still be overridden with
‘--with-tcltk-includes’ and ‘--with-tcltk-libs’. (Contributed by
Ned Deily in bpo-34956(5).)
* Python can now be built for Windows 10 ARM64. (Contributed by
Steve Dower in bpo-33125(6).)
* Some individual tests are now skipped when ‘--pgo’ is used. The
tests in question increased the PGO task time significantly and
likely didn’t help improve optimization of the final executable.
This speeds up the task by a factor of about 15x. Running the full
unit test suite is slow. This change may result in a slightly less
optimized build since not as many code branches will be executed.
If you are willing to wait for the much slower build, the old
behavior can be restored using ‘./configure [..] PROFILE_TASK="-m
test --pgo-extended"’. We make no guarantees as to which PGO task
set produces a faster build. Users who care should run their own
relevant benchmarks as results can depend on the environment,
workload, and compiler tool chain. (See bpo-36044(7) and
bpo-37707(8) for more details.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1294959
(2) https://bugs.python.org/issue?@action=redirect&bpo=39489
(3) https://bugs.python.org/issue?@action=redirect&bpo=39395
(4) https://bugs.python.org/issue?@action=redirect&bpo=10945
(5) https://bugs.python.org/issue?@action=redirect&bpo=34956
(6) https://bugs.python.org/issue?@action=redirect&bpo=33125
(7) https://bugs.python.org/issue?@action=redirect&bpo=36044
(8) https://bugs.python.org/issue?@action=redirect&bpo=37707
File: python.info, Node: C API Changes<5>, Next: Notable changes in Python 3 9 1, Prev: Build Changes<5>, Up: What’s New In Python 3 9
1.5.12 C API Changes
--------------------
* Menu:
* New Features: New Features<10>.
* Porting to Python 3.9: Porting to Python 3 9<2>.
* Removed: Removed<8>.
File: python.info, Node: New Features<10>, Next: Porting to Python 3 9<2>, Up: C API Changes<5>
1.5.12.1 New Features
.....................
* PEP 573(1): Added *note PyType_FromModuleAndSpec(): 54e. to
associate a module with a class; *note PyType_GetModule(): 96e. and
*note PyType_GetModuleState(): 96f. to retrieve the module and its
state; and *note PyCMethod: 75f. and *note METH_METHOD: 970. to
allow a method to access the class it was defined in. (Contributed
by Marcel Plch and Petr Viktorin in bpo-38787(2).)
* Added *note PyFrame_GetCode(): 785. function: get a frame code.
Added *note PyFrame_GetBack(): 781. function: get the frame next
outer frame. (Contributed by Victor Stinner in bpo-40421(3).)
* Added *note PyFrame_GetLineNumber(): 786. to the limited C API.
(Contributed by Victor Stinner in bpo-40421(4).)
* Added *note PyThreadState_GetInterpreter(): 971. and *note
PyInterpreterState_Get(): 3a9. functions to get the interpreter.
Added *note PyThreadState_GetFrame(): 789. function to get the
current frame of a Python thread state. Added *note
PyThreadState_GetID(): 972. function: get the unique identifier of
a Python thread state. (Contributed by Victor Stinner in
bpo-39947(5).)
* Added a new public *note PyObject_CallNoArgs(): 397. function to
the C API, which calls a callable Python object without any
arguments. It is the most efficient way to call a callable Python
object without any argument. (Contributed by Victor Stinner in
bpo-37194(6).)
* Changes in the limited C API (if ‘Py_LIMITED_API’ macro is
defined):
* Provide *note Py_EnterRecursiveCall(): 973. and *note
Py_LeaveRecursiveCall(): 974. as regular functions for the
limited API. Previously, there were defined as macros, but
these macros didn’t compile with the limited C API which
cannot access ‘PyThreadState.recursion_depth’ field (the
structure is opaque in the limited C API).
* ‘PyObject_INIT()’ and ‘PyObject_INIT_VAR()’ become regular
“opaque” function to hide implementation details.
(Contributed by Victor Stinner in bpo-38644(7) and bpo-39542(8).)
* The *note PyModule_AddType(): 975. function is added to help adding
a type to a module. (Contributed by Donghee Na in bpo-40024(9).)
* Added the functions *note PyObject_GC_IsTracked(): 976. and *note
PyObject_GC_IsFinalized(): 977. to the public API to allow to query
if Python objects are being currently tracked or have been already
finalized by the garbage collector respectively. (Contributed by
Pablo Galindo Salgado in bpo-40241(10).)
* Added ‘_PyObject_FunctionStr()’ to get a user-friendly string
representation of a function-like object. (Patch by Jeroen Demeyer
in bpo-37645(11).)
* Added *note PyObject_CallOneArg(): 978. for calling an object with
one positional argument (Patch by Jeroen Demeyer in bpo-37483(12).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0573/
(2) https://bugs.python.org/issue?@action=redirect&bpo=38787
(3) https://bugs.python.org/issue?@action=redirect&bpo=40421
(4) https://bugs.python.org/issue?@action=redirect&bpo=40421
(5) https://bugs.python.org/issue?@action=redirect&bpo=39947
(6) https://bugs.python.org/issue?@action=redirect&bpo=37194
(7) https://bugs.python.org/issue?@action=redirect&bpo=38644
(8) https://bugs.python.org/issue?@action=redirect&bpo=39542
(9) https://bugs.python.org/issue?@action=redirect&bpo=40024
(10) https://bugs.python.org/issue?@action=redirect&bpo=40241
(11) https://bugs.python.org/issue?@action=redirect&bpo=37645
(12) https://bugs.python.org/issue?@action=redirect&bpo=37483
File: python.info, Node: Porting to Python 3 9<2>, Next: Removed<8>, Prev: New Features<10>, Up: C API Changes<5>
1.5.12.2 Porting to Python 3.9
..............................
* ‘PyInterpreterState.eval_frame’ ( PEP 523(1)) now requires a new
mandatory 'tstate' parameter (‘PyThreadState*’). (Contributed by
Victor Stinner in bpo-38500(2).)
* Extension modules: *note m_traverse: 97a, *note m_clear: 97b. and
*note m_free: 97c. functions of *note PyModuleDef: 97d. are no
longer called if the module state was requested but is not
allocated yet. This is the case immediately after the module is
created and before the module is executed (*note Py_mod_exec: 97e.
function). More precisely, these functions are not called if *note
m_size: 97f. is greater than 0 and the module state (as returned by
*note PyModule_GetState(): 980.) is ‘NULL’.
Extension modules without module state (‘m_size <= 0’) are not
affected.
* If *note Py_AddPendingCall(): 981. is called in a subinterpreter,
the function is now scheduled to be called from the subinterpreter,
rather than being called from the main interpreter. Each
subinterpreter now has its own list of scheduled calls.
(Contributed by Victor Stinner in bpo-39984(3).)
* The Windows registry is no longer used to initialize *note
sys.path: 3b0. when the ‘-E’ option is used (if *note
PyConfig.use_environment: 3d5. is set to ‘0’). This is significant
when embedding Python on Windows. (Contributed by Zackery Spytz in
bpo-8901(4).)
* The global variable *note PyStructSequence_UnnamedField: 982. is
now a constant and refers to a constant string. (Contributed by
Serhiy Storchaka in bpo-38650(5).)
* The ‘PyGC_Head’ structure is now opaque. It is only defined in the
internal C API (‘pycore_gc.h’). (Contributed by Victor Stinner in
bpo-40241(6).)
* The ‘Py_UNICODE_COPY’, ‘Py_UNICODE_FILL’, ‘PyUnicode_WSTR_LENGTH’,
‘PyUnicode_FromUnicode()’, ‘PyUnicode_AsUnicode()’,
‘_PyUnicode_AsUnicode’, and ‘PyUnicode_AsUnicodeAndSize()’ are
marked as deprecated in C. They have been deprecated by PEP 393(7)
since Python 3.3. (Contributed by Inada Naoki in bpo-36346(8).)
* The *note Py_FatalError(): 983. function is replaced with a macro
which logs automatically the name of the current function, unless
the ‘Py_LIMITED_API’ macro is defined. (Contributed by Victor
Stinner in bpo-39882(9).)
* The vectorcall protocol now requires that the caller passes only
strings as keyword names. (See bpo-37540(10) for more
information.)
* Implementation details of a number of macros and functions are now
hidden:
* *note PyObject_IS_GC(): 984. macro was converted to a
function.
* The ‘PyObject_NEW()’ macro becomes an alias to the *note
PyObject_New: 985. macro, and the ‘PyObject_NEW_VAR()’ macro
becomes an alias to the *note PyObject_NewVar: 986. macro.
They no longer access directly the *note
PyTypeObject.tp_basicsize: 987. member.
* ‘PyObject_GET_WEAKREFS_LISTPTR()’ macro was converted to a
function: the macro accessed directly the *note
PyTypeObject.tp_weaklistoffset: 988. member.
* *note PyObject_CheckBuffer(): 394. macro was converted to a
function: the macro accessed directly the *note
PyTypeObject.tp_as_buffer: 989. member.
* *note PyIndex_Check(): 98a. is now always declared as an
opaque function to hide implementation details: removed the
‘PyIndex_Check()’ macro. The macro accessed directly the
*note PyTypeObject.tp_as_number: 98b. member.
(See bpo-40170(11) for more details.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0523/
(2) https://bugs.python.org/issue?@action=redirect&bpo=38500
(3) https://bugs.python.org/issue?@action=redirect&bpo=39984
(4) https://bugs.python.org/issue?@action=redirect&bpo=8901
(5) https://bugs.python.org/issue?@action=redirect&bpo=38650
(6) https://bugs.python.org/issue?@action=redirect&bpo=40241
(7) https://peps.python.org/pep-0393/
(8) https://bugs.python.org/issue?@action=redirect&bpo=36346
(9) https://bugs.python.org/issue?@action=redirect&bpo=39882
(10) https://bugs.python.org/issue?@action=redirect&bpo=37540
(11) https://bugs.python.org/issue?@action=redirect&bpo=40170
File: python.info, Node: Removed<8>, Prev: Porting to Python 3 9<2>, Up: C API Changes<5>
1.5.12.3 Removed
................
* Excluded ‘PyFPE_START_PROTECT()’ and ‘PyFPE_END_PROTECT()’ macros
of ‘pyfpe.h’ from the limited C API. (Contributed by Victor Stinner
in bpo-38835(1).)
* The ‘tp_print’ slot of *note PyTypeObject: 98d. has been removed.
It was used for printing objects to files in Python 2.7 and before.
Since Python 3.0, it has been ignored and unused. (Contributed by
Jeroen Demeyer in bpo-36974(2).)
* Changes in the limited C API (if ‘Py_LIMITED_API’ macro is
defined):
* Excluded the following functions from the limited C API:
* ‘PyThreadState_DeleteCurrent()’ (Contributed by Joannah
Nanjekye in bpo-37878(3).)
* ‘_Py_CheckRecursionLimit’
* ‘_Py_NewReference()’
* ‘_Py_ForgetReference()’
* ‘_PyTraceMalloc_NewReference()’
* ‘_Py_GetRefTotal()’
* The trashcan mechanism which never worked in the limited
C API.
* ‘PyTrash_UNWIND_LEVEL’
* ‘Py_TRASHCAN_BEGIN_CONDITION’
* ‘Py_TRASHCAN_BEGIN’
* ‘Py_TRASHCAN_END’
* ‘Py_TRASHCAN_SAFE_BEGIN’
* ‘Py_TRASHCAN_SAFE_END’
* Moved following functions and definitions to the internal C
API:
* ‘_PyDebug_PrintTotalRefs()’
* ‘_Py_PrintReferences()’
* ‘_Py_PrintReferenceAddresses()’
* ‘_Py_tracemalloc_config’
* ‘_Py_AddToAllObjects()’ (specific to ‘Py_TRACE_REFS’
build)
(Contributed by Victor Stinner in bpo-38644(4) and bpo-39542(5).)
* Removed ‘_PyRuntime.getframe’ hook and removed
‘_PyThreadState_GetFrame’ macro which was an alias to
‘_PyRuntime.getframe’. They were only exposed by the internal C
API. Removed also ‘PyThreadFrameGetter’ type. (Contributed by
Victor Stinner in bpo-39946(6).)
* Removed the following functions from the C API. Call *note
PyGC_Collect(): 98e. explicitly to clear all free lists.
(Contributed by Inada Naoki and Victor Stinner in bpo-37340(7),
bpo-38896(8) and bpo-40428(9).)
* ‘PyAsyncGen_ClearFreeLists()’
* ‘PyContext_ClearFreeList()’
* ‘PyDict_ClearFreeList()’
* ‘PyFloat_ClearFreeList()’
* ‘PyFrame_ClearFreeList()’
* ‘PyList_ClearFreeList()’
* ‘PyMethod_ClearFreeList()’ and ‘PyCFunction_ClearFreeList()’:
the free lists of bound method objects have been removed.
* ‘PySet_ClearFreeList()’: the set free list has been removed in
Python 3.4.
* ‘PyTuple_ClearFreeList()’
* ‘PyUnicode_ClearFreeList()’: the Unicode free list has been
removed in Python 3.3.
* Removed ‘_PyUnicode_ClearStaticStrings()’ function. (Contributed
by Victor Stinner in bpo-39465(10).)
* Removed ‘Py_UNICODE_MATCH’. It has been deprecated by PEP 393(11),
and broken since Python 3.3. The *note PyUnicode_Tailmatch(): 8b7.
function can be used instead. (Contributed by Inada Naoki in
bpo-36346(12).)
* Cleaned header files of interfaces defined but with no
implementation. The public API symbols being removed are:
‘_PyBytes_InsertThousandsGroupingLocale’,
‘_PyBytes_InsertThousandsGrouping’, ‘_Py_InitializeFromArgs’,
‘_Py_InitializeFromWideArgs’, ‘_PyFloat_Repr’, ‘_PyFloat_Digits’,
‘_PyFloat_DigitsInit’, ‘PyFrame_ExtendStack’,
‘_PyAIterWrapper_Type’, ‘PyNullImporter_Type’, ‘PyCmpWrapper_Type’,
‘PySortWrapper_Type’, ‘PyNoArgsFunction’. (Contributed by Pablo
Galindo Salgado in bpo-39372(13).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=38835
(2) https://bugs.python.org/issue?@action=redirect&bpo=36974
(3) https://bugs.python.org/issue?@action=redirect&bpo=37878
(4) https://bugs.python.org/issue?@action=redirect&bpo=38644
(5) https://bugs.python.org/issue?@action=redirect&bpo=39542
(6) https://bugs.python.org/issue?@action=redirect&bpo=39946
(7) https://bugs.python.org/issue?@action=redirect&bpo=37340
(8) https://bugs.python.org/issue?@action=redirect&bpo=38896
(9) https://bugs.python.org/issue?@action=redirect&bpo=40428
(10) https://bugs.python.org/issue?@action=redirect&bpo=39465
(11) https://peps.python.org/pep-0393/
(12) https://bugs.python.org/issue?@action=redirect&bpo=36346
(13) https://bugs.python.org/issue?@action=redirect&bpo=39372
File: python.info, Node: Notable changes in Python 3 9 1, Next: Notable changes in Python 3 9 2, Prev: C API Changes<5>, Up: What’s New In Python 3 9
1.5.13 Notable changes in Python 3.9.1
--------------------------------------
* Menu:
* typing: typing<7>.
* macOS 11.0 (Big Sur) and Apple Silicon Mac support: macOS 11 0 Big Sur and Apple Silicon Mac support.
File: python.info, Node: typing<7>, Next: macOS 11 0 Big Sur and Apple Silicon Mac support, Up: Notable changes in Python 3 9 1
1.5.13.1 typing
...............
The behavior of *note typing.Literal: 851. was changed to conform with
PEP 586(1) and to match the behavior of static type checkers specified
in the PEP.
1. ‘Literal’ now de-duplicates parameters.
2. Equality comparisons between ‘Literal’ objects are now order
independent.
3. ‘Literal’ comparisons now respect types. For example, ‘Literal[0]
== Literal[False]’ previously evaluated to ‘True’. It is now
‘False’. To support this change, the internally used type cache
now supports differentiating types.
4. ‘Literal’ objects will now raise a *note TypeError: 534. exception
during equality comparisons if any of their parameters are not
*note hashable: 60c. Note that declaring ‘Literal’ with mutable
parameters will not throw an error:
>>> from typing import Literal
>>> Literal[{0}]
>>> Literal[{0}] == Literal[{False}]
Traceback (most recent call last):
File "", line 1, in
TypeError: unhashable type: 'set'
(Contributed by Yurii Karabas in bpo-42345(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0586/
(2) https://bugs.python.org/issue?@action=redirect&bpo=42345
File: python.info, Node: macOS 11 0 Big Sur and Apple Silicon Mac support, Prev: typing<7>, Up: Notable changes in Python 3 9 1
1.5.13.2 macOS 11.0 (Big Sur) and Apple Silicon Mac support
...........................................................
As of 3.9.1, Python now fully supports building and running on macOS
11.0 (Big Sur) and on Apple Silicon Macs (based on the ‘ARM64’
architecture). A new universal build variant, ‘universal2’, is now
available to natively support both ‘ARM64’ and ‘Intel 64’ in one set of
executables. Binaries can also now be built on current versions of
macOS to be deployed on a range of older macOS versions (tested to 10.9)
while making some newer OS functions and options conditionally available
based on the operating system version in use at runtime (“weaklinking”).
(Contributed by Ronald Oussoren and Lawrence D’Anna in bpo-41100(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=41100
File: python.info, Node: Notable changes in Python 3 9 2, Next: Notable changes in Python 3 9 3, Prev: Notable changes in Python 3 9 1, Up: What’s New In Python 3 9
1.5.14 Notable changes in Python 3.9.2
--------------------------------------
* Menu:
* collections.abc: collections abc<2>.
* urllib.parse: urllib parse<2>.
File: python.info, Node: collections abc<2>, Next: urllib parse<2>, Up: Notable changes in Python 3 9 2
1.5.14.1 collections.abc
........................
*note collections.abc.Callable: 7e5. generic now flattens type
parameters, similar to what *note typing.Callable: 7c0. currently does.
This means that ‘collections.abc.Callable[[int, str], str]’ will have
‘__args__’ of ‘(int, str, str)’; previously this was ‘([int, str],
str)’. To allow this change, *note types.GenericAlias: 7e6. can now be
subclassed, and a subclass will be returned when subscripting the *note
collections.abc.Callable: 7e5. type. Code which accesses the arguments
via *note typing.get_args(): 87c. or ‘__args__’ need to account for this
change. A *note DeprecationWarning: 1a5. may be emitted for invalid
forms of parameterizing *note collections.abc.Callable: 7e5. which may
have passed silently in Python 3.9.1. This *note DeprecationWarning:
1a5. will become a *note TypeError: 534. in Python 3.10. (Contributed
by Ken Jin in bpo-42195(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42195
File: python.info, Node: urllib parse<2>, Prev: collections abc<2>, Up: Notable changes in Python 3 9 2
1.5.14.2 urllib.parse
.....................
Earlier Python versions allowed using both ‘;’ and ‘&’ as query
parameter separators in *note urllib.parse.parse_qs(): 27c. and *note
urllib.parse.parse_qsl(): 27b. Due to security concerns, and to conform
with newer W3C recommendations, this has been changed to allow only a
single separator key, with ‘&’ as the default. This change also affects
‘cgi.parse()’ and ‘cgi.parse_multipart()’ as they use the affected
functions internally. For more details, please see their respective
documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and
Ken Jin in bpo-42967(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42967
File: python.info, Node: Notable changes in Python 3 9 3, Next: Notable changes in Python 3 9 5, Prev: Notable changes in Python 3 9 2, Up: What’s New In Python 3 9
1.5.15 Notable changes in Python 3.9.3
--------------------------------------
A security fix alters the *note ftplib.FTP: 8f9. behavior to not trust
the IPv4 address sent from the remote server when setting up a passive
data channel. We reuse the ftp server IP address instead. For unusual
code requiring the old behavior, set a ‘trust_server_pasv_ipv4_address’
attribute on your FTP instance to ‘True’. (See gh-87451(1))
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/87451
File: python.info, Node: Notable changes in Python 3 9 5, Next: Notable security feature in 3 9 14, Prev: Notable changes in Python 3 9 3, Up: What’s New In Python 3 9
1.5.16 Notable changes in Python 3.9.5
--------------------------------------
* Menu:
* urllib.parse: urllib parse<3>.
File: python.info, Node: urllib parse<3>, Up: Notable changes in Python 3 9 5
1.5.16.1 urllib.parse
.....................
The presence of newline or tab characters in parts of a URL allows for
some forms of attacks. Following the WHATWG specification that updates
RFC 3986(1), ASCII newline ‘\n’, ‘\r’ and tab ‘\t’ characters are
stripped from the URL by the parser in *note urllib.parse: 10a.
preventing such attacks. The removal characters are controlled by a new
module level variable ‘urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE’. (See
gh-88048(2))
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc3986.html
(2) https://github.com/python/cpython/issues/88048
File: python.info, Node: Notable security feature in 3 9 14, Next: Notable changes in 3 9 17, Prev: Notable changes in Python 3 9 5, Up: What’s New In Python 3 9
1.5.17 Notable security feature in 3.9.14
-----------------------------------------
Converting between *note int: 259. and *note str: 447. in bases other
than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10
(decimal) now raises a *note ValueError: 204. if the number of digits in
string form is above a limit to avoid potential denial of service
attacks due to the algorithmic complexity. This is a mitigation for CVE
2020-10735(1). This limit can be configured or disabled by environment
variable, command line flag, or *note sys: d9. APIs. See the *note
integer string conversion length limitation: 5f1. documentation. The
default limit is 4300 digits in string form.
---------- Footnotes ----------
(1) https://www.cve.org/CVERecord?id=CVE-2020-10735
File: python.info, Node: Notable changes in 3 9 17, Prev: Notable security feature in 3 9 14, Up: What’s New In Python 3 9
1.5.18 Notable changes in 3.9.17
--------------------------------
* Menu:
* tarfile: tarfile<4>.
File: python.info, Node: tarfile<4>, Up: Notable changes in 3 9 17
1.5.18.1 tarfile
................
* The extraction methods in *note tarfile: de, and *note
shutil.unpack_archive(): 467, have a new a 'filter' argument that
allows limiting tar features than may be surprising or dangerous,
such as creating files outside the destination directory. See
*note Extraction filters: 468. for details. In Python 3.12, use
without the 'filter' argument will show a *note DeprecationWarning:
1a5. In Python 3.14, the default will switch to ‘'data'’.
(Contributed by Petr Viktorin in PEP 706(1).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0706/
File: python.info, Node: What’s New In Python 3 8, Next: What’s New In Python 3 7, Prev: What’s New In Python 3 9, Up: What’s New in Python
1.6 What’s New In Python 3.8
============================
Editor: Raymond Hettinger
This article explains the new features in Python 3.8, compared to 3.7.
Python 3.8 was released on October 14, 2019. For full details, see the
*note changelog: 13c.
* Menu:
* Summary – Release highlights: Summary – Release highlights<5>.
* New Features: New Features<11>.
* Other Language Changes: Other Language Changes<6>.
* New Modules: New Modules<6>.
* Improved Modules: Improved Modules<6>.
* Optimizations: Optimizations<6>.
* Build and C API Changes::
* Deprecated: Deprecated<8>.
* API and Feature Removals::
* Porting to Python 3.8: Porting to Python 3 8.
* Notable changes in Python 3.8.1: Notable changes in Python 3 8 1.
* Notable changes in Python 3.8.2: Notable changes in Python 3 8 2.
* Notable changes in Python 3.8.3: Notable changes in Python 3 8 3.
* Notable changes in Python 3.8.8: Notable changes in Python 3 8 8.
* Notable changes in Python 3.8.9: Notable changes in Python 3 8 9.
* Notable changes in Python 3.8.10: Notable changes in Python 3 8 10.
* Notable changes in Python 3.8.10: Notable changes in Python 3 8 10<2>.
* Notable changes in Python 3.8.12: Notable changes in Python 3 8 12.
* Notable security feature in 3.8.14: Notable security feature in 3 8 14.
* Notable changes in 3.8.17: Notable changes in 3 8 17.
File: python.info, Node: Summary – Release highlights<5>, Next: New Features<11>, Up: What’s New In Python 3 8
1.6.1 Summary – Release highlights
----------------------------------
File: python.info, Node: New Features<11>, Next: Other Language Changes<6>, Prev: Summary – Release highlights<5>, Up: What’s New In Python 3 8
1.6.2 New Features
------------------
* Menu:
* Assignment expressions::
* Positional-only parameters::
* Parallel filesystem cache for compiled bytecode files::
* Debug build uses the same ABI as release build::
* f-strings support = for self-documenting expressions and debugging::
* PEP 578; Python Runtime Audit Hooks: PEP 578 Python Runtime Audit Hooks.
* PEP 587; Python Initialization Configuration: PEP 587 Python Initialization Configuration.
* PEP 590; Vectorcall; a fast calling protocol for CPython: PEP 590 Vectorcall a fast calling protocol for CPython.
* Pickle protocol 5 with out-of-band data buffers::
File: python.info, Node: Assignment expressions, Next: Positional-only parameters, Up: New Features<11>
1.6.2.1 Assignment expressions
..............................
There is new syntax ‘:=’ that assigns values to variables as part of a
larger expression. It is affectionately known as “the walrus operator”
due to its resemblance to the eyes and tusks of a walrus(1).
In this example, the assignment expression helps avoid calling *note
len(): 62a. twice:
if (n := len(a)) > 10:
print(f"List is too long ({n} elements, expected <= 10)")
A similar benefit arises during regular expression matching where match
objects are needed twice, once to test whether a match occurred and
another to extract a subgroup:
discount = 0.0
if (mo := re.search(r'(\d+)% discount', advertisement)):
discount = float(mo.group(1)) / 100.0
The operator is also useful with while-loops that compute a value to
test loop termination and then need that same value again in the body of
the loop:
# Loop over fixed length blocks
while (block := f.read(256)) != '':
process(block)
Another motivating use case arises in list comprehensions where a value
computed in a filtering condition is also needed in the expression body:
[clean_name.title() for name in names
if (clean_name := normalize('NFC', name)) in allowed_names]
Try to limit use of the walrus operator to clean cases that reduce
complexity and improve readability.
See PEP 572(2) for a full description.
(Contributed by Emily Morehouse in bpo-35224(3).)
---------- Footnotes ----------
(1)
https://en.wikipedia.org/wiki/Walrus#/media/File:Pacific_Walrus_-_Bull_(8247646168).jpg
(2) https://peps.python.org/pep-0572/
(3) https://bugs.python.org/issue?@action=redirect&bpo=35224
File: python.info, Node: Positional-only parameters, Next: Parallel filesystem cache for compiled bytecode files, Prev: Assignment expressions, Up: New Features<11>
1.6.2.2 Positional-only parameters
..................................
There is a new function parameter syntax ‘/’ to indicate that some
function parameters must be specified positionally and cannot be used as
keyword arguments. This is the same notation shown by ‘help()’ for C
functions annotated with Larry Hastings’ Argument Clinic(1) tool.
In the following example, parameters 'a' and 'b' are positional-only,
while 'c' or 'd' can be positional or keyword, and 'e' or 'f' are
required to be keywords:
def f(a, b, /, c, d, *, e, f):
print(a, b, c, d, e, f)
The following is a valid call:
f(10, 20, 30, d=40, e=50, f=60)
However, these are invalid calls:
f(10, b=20, c=30, d=40, e=50, f=60) # b cannot be a keyword argument
f(10, 20, 30, 40, 50, f=60) # e must be a keyword argument
One use case for this notation is that it allows pure Python functions
to fully emulate behaviors of existing C coded functions. For example,
the built-in *note divmod(): 9a1. function does not accept keyword
arguments:
def divmod(a, b, /):
"Emulate the built in divmod() function"
return (a // b, a % b)
Another use case is to preclude keyword arguments when the parameter
name is not helpful. For example, the builtin *note len(): 62a.
function has the signature ‘len(obj, /)’. This precludes awkward calls
such as:
len(obj='hello') # The "obj" keyword argument impairs readability
A further benefit of marking a parameter as positional-only is that it
allows the parameter name to be changed in the future without risk of
breaking client code. For example, in the *note statistics: d2. module,
the parameter name 'dist' may be changed in the future. This was made
possible with the following function specification:
def quantiles(dist, /, *, n=4, method='exclusive')
...
Since the parameters to the left of ‘/’ are not exposed as possible
keywords, the parameters names remain available for use in ‘**kwargs’:
>>> def f(a, b, /, **kwargs):
... print(a, b, kwargs)
...
>>> f(10, 20, a=1, b=2, c=3) # a and b are used in two ways
10 20 {'a': 1, 'b': 2, 'c': 3}
This greatly simplifies the implementation of functions and methods that
need to accept arbitrary keyword arguments. For example, here is an
excerpt from code in the *note collections: 1d. module:
class Counter(dict):
def __init__(self, iterable=None, /, **kwds):
# Note "iterable" is a possible keyword argument
See PEP 570(2) for a full description.
(Contributed by Pablo Galindo in bpo-36540(3).)
---------- Footnotes ----------
(1) https://devguide.python.org/development-tools/clinic/
(2) https://peps.python.org/pep-0570/
(3) https://bugs.python.org/issue?@action=redirect&bpo=36540
File: python.info, Node: Parallel filesystem cache for compiled bytecode files, Next: Debug build uses the same ABI as release build, Prev: Positional-only parameters, Up: New Features<11>
1.6.2.3 Parallel filesystem cache for compiled bytecode files
.............................................................
The new *note PYTHONPYCACHEPREFIX: 9a3. setting (also available as *note
-X: 176. ‘pycache_prefix’) configures the implicit bytecode cache to use
a separate parallel filesystem tree, rather than the default
‘__pycache__’ subdirectories within each source directory.
The location of the cache is reported in *note sys.pycache_prefix: 9a4.
(*note None: 671. indicates the default location in ‘__pycache__’
subdirectories).
(Contributed by Carl Meyer in bpo-33499(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=33499
File: python.info, Node: Debug build uses the same ABI as release build, Next: f-strings support = for self-documenting expressions and debugging, Prev: Parallel filesystem cache for compiled bytecode files, Up: New Features<11>
1.6.2.4 Debug build uses the same ABI as release build
......................................................
Python now uses the same ABI whether it’s built in release or debug
mode. On Unix, when Python is built in debug mode, it is now possible
to load C extensions built in release mode and C extensions built using
the stable ABI.
Release builds and *note debug builds: 1fb. are now ABI compatible:
defining the ‘Py_DEBUG’ macro no longer implies the ‘Py_TRACE_REFS’
macro, which introduces the only ABI incompatibility. The
‘Py_TRACE_REFS’ macro, which adds the *note sys.getobjects(): 423.
function and the *note PYTHONDUMPREFS: 9a6. environment variable, can be
set using the new *note ./configure -with-trace-refs: 390. build option.
(Contributed by Victor Stinner in bpo-36465(1).)
On Unix, C extensions are no longer linked to libpython except on
Android and Cygwin. It is now possible for a statically linked Python
to load a C extension built using a shared library Python. (Contributed
by Victor Stinner in bpo-21536(2).)
On Unix, when Python is built in debug mode, import now also looks for C
extensions compiled in release mode and for C extensions compiled with
the stable ABI. (Contributed by Victor Stinner in bpo-36722(3).)
To embed Python into an application, a new ‘--embed’ option must be
passed to ‘python3-config --libs --embed’ to get ‘-lpython3.8’ (link the
application to libpython). To support both 3.8 and older, try
‘python3-config --libs --embed’ first and fallback to ‘python3-config
--libs’ (without ‘--embed’) if the previous command fails.
Add a pkg-config ‘python-3.8-embed’ module to embed Python into an
application: ‘pkg-config python-3.8-embed --libs’ includes
‘-lpython3.8’. To support both 3.8 and older, try ‘pkg-config
python-X.Y-embed --libs’ first and fallback to ‘pkg-config python-X.Y
--libs’ (without ‘--embed’) if the previous command fails (replace ‘X.Y’
with the Python version).
On the other hand, ‘pkg-config python3.8 --libs’ no longer contains
‘-lpython3.8’. C extensions must not be linked to libpython (except on
Android and Cygwin, whose cases are handled by the script); this change
is backward incompatible on purpose. (Contributed by Victor Stinner in
bpo-36721(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36465
(2) https://bugs.python.org/issue?@action=redirect&bpo=21536
(3) https://bugs.python.org/issue?@action=redirect&bpo=36722
(4) https://bugs.python.org/issue?@action=redirect&bpo=36721
File: python.info, Node: f-strings support = for self-documenting expressions and debugging, Next: PEP 578 Python Runtime Audit Hooks, Prev: Debug build uses the same ABI as release build, Up: New Features<11>
1.6.2.5 f-strings support ‘=’ for self-documenting expressions and debugging
............................................................................
Added an ‘=’ specifier to *note f-string: 431.s. An f-string such as
‘f'{expr=}'’ will expand to the text of the expression, an equal sign,
then the representation of the evaluated expression. For example:
>>> user = 'eric_idle'
>>> member_since = date(1975, 7, 31)
>>> f'{user=} {member_since=}'
"user='eric_idle' member_since=datetime.date(1975, 7, 31)"
The usual *note f-string format specifiers: 9a9. allow more control over
how the result of the expression is displayed:
>>> delta = date.today() - member_since
>>> f'{user=!s} {delta.days=:,d}'
'user=eric_idle delta.days=16,075'
The ‘=’ specifier will display the whole expression so that calculations
can be shown:
>>> print(f'{theta=} {cos(radians(theta))=:.3f}')
theta=30 cos(radians(theta))=0.866
(Contributed by Eric V. Smith and Larry Hastings in bpo-36817(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36817
File: python.info, Node: PEP 578 Python Runtime Audit Hooks, Next: PEP 587 Python Initialization Configuration, Prev: f-strings support = for self-documenting expressions and debugging, Up: New Features<11>
1.6.2.6 PEP 578: Python Runtime Audit Hooks
...........................................
The PEP adds an Audit Hook and Verified Open Hook. Both are available
from Python and native code, allowing applications and frameworks
written in pure Python code to take advantage of extra notifications,
while also allowing embedders or system administrators to deploy builds
of Python where auditing is always enabled.
See PEP 578(1) for full details.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0578/
File: python.info, Node: PEP 587 Python Initialization Configuration, Next: PEP 590 Vectorcall a fast calling protocol for CPython, Prev: PEP 578 Python Runtime Audit Hooks, Up: New Features<11>
1.6.2.7 PEP 587: Python Initialization Configuration
....................................................
The PEP 587(1) adds a new C API to configure the Python Initialization
providing finer control on the whole configuration and better error
reporting.
New structures:
* *note PyConfig: 3a2.
* *note PyPreConfig: 9ac.
* *note PyStatus: 9ad.
* *note PyWideStringList: 9ae.
New functions:
* *note PyConfig_Clear(): 9af.
* *note PyConfig_InitIsolatedConfig(): 9b0.
* *note PyConfig_InitPythonConfig(): 9b1.
* *note PyConfig_Read(): 78b.
* *note PyConfig_SetArgv(): 9b2.
* *note PyConfig_SetBytesArgv(): 9b3.
* *note PyConfig_SetBytesString(): 9b4.
* *note PyConfig_SetString(): 9b5.
* *note PyPreConfig_InitIsolatedConfig(): 9b6.
* *note PyPreConfig_InitPythonConfig(): 9b7.
* *note PyStatus_Error(): 9b8.
* *note PyStatus_Exception(): 9b9.
* *note PyStatus_Exit(): 9ba.
* *note PyStatus_IsError(): 9bb.
* *note PyStatus_IsExit(): 9bc.
* *note PyStatus_NoMemory(): 9bd.
* *note PyStatus_Ok(): 9be.
* *note PyWideStringList_Append(): 9bf.
* *note PyWideStringList_Insert(): 9c0.
* *note Py_BytesMain(): 9c1.
* *note Py_ExitStatusException(): 9c2.
* *note Py_InitializeFromConfig(): 3c1.
* *note Py_PreInitialize(): 3e7.
* *note Py_PreInitializeFromArgs(): 9c3.
* *note Py_PreInitializeFromBytesArgs(): 9c4.
* *note Py_RunMain(): 9c5.
This PEP also adds ‘_PyRuntimeState.preconfig’ (*note PyPreConfig: 9ac.
type) and ‘PyInterpreterState.config’ (*note PyConfig: 3a2. type) fields
to these internal structures. ‘PyInterpreterState.config’ becomes the
new reference configuration, replacing global configuration variables
and other private variables.
See *note Python Initialization Configuration: 3a3. for the
documentation.
See PEP 587(2) for a full description.
(Contributed by Victor Stinner in bpo-36763(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0587/
(2) https://peps.python.org/pep-0587/
(3) https://bugs.python.org/issue?@action=redirect&bpo=36763
File: python.info, Node: PEP 590 Vectorcall a fast calling protocol for CPython, Next: Pickle protocol 5 with out-of-band data buffers, Prev: PEP 587 Python Initialization Configuration, Up: New Features<11>
1.6.2.8 PEP 590: Vectorcall: a fast calling protocol for CPython
................................................................
*note The Vectorcall Protocol: 54f. is added to the Python/C API. It is
meant to formalize existing optimizations which were already done for
various classes. Any *note static type: 77a. implementing a callable
can use this protocol.
This is currently provisional. The aim is to make it fully public in
Python 3.9.
See PEP 590(1) for a full description.
(Contributed by Jeroen Demeyer, Mark Shannon and Petr Viktorin in
bpo-36974(2).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0590/
(2) https://bugs.python.org/issue?@action=redirect&bpo=36974
File: python.info, Node: Pickle protocol 5 with out-of-band data buffers, Prev: PEP 590 Vectorcall a fast calling protocol for CPython, Up: New Features<11>
1.6.2.9 Pickle protocol 5 with out-of-band data buffers
.......................................................
When *note pickle: a6. is used to transfer large data between Python
processes in order to take advantage of multi-core or multi-machine
processing, it is important to optimize the transfer by reducing memory
copies, and possibly by applying custom techniques such as
data-dependent compression.
The *note pickle: a6. protocol 5 introduces support for out-of-band
buffers where PEP 3118(1)-compatible data can be transmitted separately
from the main pickle stream, at the discretion of the communication
layer.
See PEP 574(2) for a full description.
(Contributed by Antoine Pitrou in bpo-36785(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-3118/
(2) https://peps.python.org/pep-0574/
(3) https://bugs.python.org/issue?@action=redirect&bpo=36785
File: python.info, Node: Other Language Changes<6>, Next: New Modules<6>, Prev: New Features<11>, Up: What’s New In Python 3 8
1.6.3 Other Language Changes
----------------------------
* A *note continue: 9c9. statement was illegal in the *note finally:
9ca. clause due to a problem with the implementation. In Python
3.8 this restriction was lifted. (Contributed by Serhiy Storchaka
in bpo-32489(1).)
* The *note bool: 463, *note int: 259, and *note fractions.Fraction:
1e9. types now have an *note as_integer_ratio(): 9cb. method like
that found in *note float: 2f1. and *note decimal.Decimal: 29f.
This minor API extension makes it possible to write ‘numerator,
denominator = x.as_integer_ratio()’ and have it work across
multiple numeric types. (Contributed by Lisa Roach in bpo-33073(2)
and Raymond Hettinger in bpo-37819(3).)
* Constructors of *note int: 259, *note float: 2f1. and *note
complex: 2f2. will now use the *note __index__(): 718. special
method, if available and the corresponding method *note __int__():
717, *note __float__(): 9cc. or *note __complex__(): 5e3. is not
available. (Contributed by Serhiy Storchaka in bpo-20092(4).)
* Added support of ‘\N{NAME}’ escapes in *note regular expressions:
b9.:
>>> notice = 'Copyright © 2019'
>>> copyright_year_pattern = re.compile(r'\N{copyright sign}\s*(\d{4})')
>>> int(copyright_year_pattern.search(notice).group(1))
2019
(Contributed by Jonathan Eunice and Serhiy Storchaka in
bpo-30688(5).)
* Dict and dictviews are now iterable in reversed insertion order
using *note reversed(): 862. (Contributed by Rémi Lapeyre in
bpo-33462(6).)
* The syntax allowed for keyword names in function calls was further
restricted. In particular, ‘f((keyword)=arg)’ is no longer
allowed. It was never intended to permit more than a bare name on
the left-hand side of a keyword argument assignment term.
(Contributed by Benjamin Peterson in bpo-34641(7).)
* Generalized iterable unpacking in *note yield: 9cd. and *note
return: 9ce. statements no longer requires enclosing parentheses.
This brings the 'yield' and 'return' syntax into better agreement
with normal assignment syntax:
>>> def parse(family):
lastname, *members = family.split()
return lastname.upper(), *members
>>> parse('simpsons homer marge bart lisa maggie')
('SIMPSONS', 'homer', 'marge', 'bart', 'lisa', 'maggie')
(Contributed by David Cuthbert and Jordan Chapman in bpo-32117(8).)
* When a comma is missed in code such as ‘[(10, 20) (30, 40)]’, the
compiler displays a *note SyntaxWarning: 461. with a helpful
suggestion. This improves on just having a *note TypeError: 534.
indicating that the first tuple was not callable. (Contributed by
Serhiy Storchaka in bpo-15248(9).)
* Arithmetic operations between subclasses of *note datetime.date:
1cd. or *note datetime.datetime: 1cc. and *note datetime.timedelta:
9cf. objects now return an instance of the subclass, rather than
the base class. This also affects the return type of operations
whose implementation (directly or indirectly) uses *note
datetime.timedelta: 9cf. arithmetic, such as *note astimezone():
9d0. (Contributed by Paul Ganssle in bpo-32417(10).)
* When the Python interpreter is interrupted by Ctrl-C (SIGINT) and
the resulting *note KeyboardInterrupt: 9d1. exception is not
caught, the Python process now exits via a SIGINT signal or with
the correct exit code such that the calling process can detect that
it died due to a Ctrl-C. Shells on POSIX and Windows use this to
properly terminate scripts in interactive sessions. (Contributed
by Google via Gregory P. Smith in bpo-1054041(11).)
* Some advanced styles of programming require updating the *note
types.CodeType: 2e3. object for an existing function. Since code
objects are immutable, a new code object needs to be created, one
that is modeled on the existing code object. With 19 parameters,
this was somewhat tedious. Now, the new ‘replace()’ method makes
it possible to create a clone with a few altered parameters.
Here’s an example that alters the *note statistics.mean(): 6cd.
function to prevent the 'data' parameter from being used as a
keyword argument:
>>> from statistics import mean
>>> mean(data=[10, 20, 90])
40
>>> mean.__code__ = mean.__code__.replace(co_posonlyargcount=1)
>>> mean(data=[10, 20, 90])
Traceback (most recent call last):
...
TypeError: mean() got some positional-only arguments passed as keyword arguments: 'data'
(Contributed by Victor Stinner in bpo-37032(12).)
* For integers, the three-argument form of the *note pow(): 9d2.
function now permits the exponent to be negative in the case where
the base is relatively prime to the modulus. It then computes a
modular inverse to the base when the exponent is ‘-1’, and a
suitable power of that inverse for other negative exponents. For
example, to compute the modular multiplicative inverse(13) of 38
modulo 137, write:
>>> pow(38, -1, 137)
119
>>> 119 * 38 % 137
1
Modular inverses arise in the solution of linear Diophantine
equations(14). For example, to find integer solutions for ‘4258𝑥 +
147𝑦 = 369’, first rewrite as ‘4258𝑥 ≡ 369 (mod 147)’ then solve:
>>> x = 369 * pow(4258, -1, 147) % 147
>>> y = (4258 * x - 369) // -147
>>> 4258 * x + 147 * y
369
(Contributed by Mark Dickinson in bpo-36027(15).)
* Dict comprehensions have been synced-up with dict literals so that
the key is computed first and the value second:
>>> # Dict comprehension
>>> cast = {input('role? '): input('actor? ') for i in range(2)}
role? King Arthur
actor? Chapman
role? Black Knight
actor? Cleese
>>> # Dict literal
>>> cast = {input('role? '): input('actor? ')}
role? Sir Robin
actor? Eric Idle
The guaranteed execution order is helpful with assignment
expressions because variables assigned in the key expression will
be available in the value expression:
>>> names = ['Martin von Löwis', 'Łukasz Langa', 'Walter Dörwald']
>>> {(n := normalize('NFC', name)).casefold() : n for name in names}
{'martin von löwis': 'Martin von Löwis',
'łukasz langa': 'Łukasz Langa',
'walter dörwald': 'Walter Dörwald'}
(Contributed by Jörn Heissler in bpo-35224(16).)
* The *note object.__reduce__(): 9d3. method can now return a tuple
from two to six elements long. Formerly, five was the limit. The
new, optional sixth element is a callable with a ‘(obj, state)’
signature. This allows the direct control over the state-updating
behavior of a specific object. If not 'None', this callable will
have priority over the object’s ‘__setstate__()’ method.
(Contributed by Pierre Glaser and Olivier Grisel in bpo-35900(17).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32489
(2) https://bugs.python.org/issue?@action=redirect&bpo=33073
(3) https://bugs.python.org/issue?@action=redirect&bpo=37819
(4) https://bugs.python.org/issue?@action=redirect&bpo=20092
(5) https://bugs.python.org/issue?@action=redirect&bpo=30688
(6) https://bugs.python.org/issue?@action=redirect&bpo=33462
(7) https://bugs.python.org/issue?@action=redirect&bpo=34641
(8) https://bugs.python.org/issue?@action=redirect&bpo=32117
(9) https://bugs.python.org/issue?@action=redirect&bpo=15248
(10) https://bugs.python.org/issue?@action=redirect&bpo=32417
(11) https://bugs.python.org/issue?@action=redirect&bpo=1054041
(12) https://bugs.python.org/issue?@action=redirect&bpo=37032
(13) https://en.wikipedia.org/wiki/Modular_multiplicative_inverse
(14) https://en.wikipedia.org/wiki/Diophantine_equation
(15) https://bugs.python.org/issue?@action=redirect&bpo=36027
(16) https://bugs.python.org/issue?@action=redirect&bpo=35224
(17) https://bugs.python.org/issue?@action=redirect&bpo=35900
File: python.info, Node: New Modules<6>, Next: Improved Modules<6>, Prev: Other Language Changes<6>, Up: What’s New In Python 3 8
1.6.4 New Modules
-----------------
* The new *note importlib.metadata: 7a. module provides (provisional)
support for reading metadata from third-party packages. For
example, it can extract an installed package’s version number, list
of entry points, and more:
>>> # Note following example requires that the popular "requests"
>>> # package has been installed.
>>>
>>> from importlib.metadata import version, requires, files
>>> version('requests')
'2.22.0'
>>> list(requires('requests'))
['chardet (<3.1.0,>=3.0.2)']
>>> list(files('requests'))[:5]
[PackagePath('requests-2.22.0.dist-info/INSTALLER'),
PackagePath('requests-2.22.0.dist-info/LICENSE'),
PackagePath('requests-2.22.0.dist-info/METADATA'),
PackagePath('requests-2.22.0.dist-info/RECORD'),
PackagePath('requests-2.22.0.dist-info/WHEEL')]
(Contributed by Barry Warsaw and Jason R. Coombs in bpo-34632(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34632
File: python.info, Node: Improved Modules<6>, Next: Optimizations<6>, Prev: New Modules<6>, Up: What’s New In Python 3 8
1.6.5 Improved Modules
----------------------
* Menu:
* ast: ast<3>.
* asyncio: asyncio<6>.
* builtins: builtins<2>.
* collections::
* cProfile::
* csv: csv<2>.
* curses: curses<3>.
* ctypes: ctypes<2>.
* datetime: datetime<3>.
* functools: functools<2>.
* gc: gc<3>.
* gettext::
* gzip: gzip<3>.
* IDLE and idlelib: IDLE and idlelib<4>.
* inspect: inspect<5>.
* io: io<3>.
* itertools: itertools<4>.
* json.tool: json tool.
* logging: logging<2>.
* math: math<5>.
* mmap: mmap<2>.
* multiprocessing: multiprocessing<3>.
* os: os<6>.
* os.path: os path<5>.
* pathlib: pathlib<7>.
* pickle::
* plistlib::
* pprint: pprint<3>.
* py_compile: py_compile<2>.
* shlex::
* shutil: shutil<4>.
* socket: socket<4>.
* ssl: ssl<4>.
* statistics: statistics<4>.
* sys: sys<7>.
* tarfile: tarfile<5>.
* threading: threading<4>.
* tokenize: tokenize<2>.
* tkinter: tkinter<4>.
* time: time<4>.
* typing: typing<8>.
* unicodedata: unicodedata<5>.
* unittest: unittest<6>.
* venv: venv<4>.
* weakref::
* xml: xml<4>.
* xmlrpc::
File: python.info, Node: ast<3>, Next: asyncio<6>, Up: Improved Modules<6>
1.6.5.1 ast
...........
AST nodes now have ‘end_lineno’ and ‘end_col_offset’ attributes, which
give the precise location of the end of the node. (This only applies to
nodes that have ‘lineno’ and ‘col_offset’ attributes.)
New function *note ast.get_source_segment(): 9d7. returns the source
code for a specific AST node.
(Contributed by Ivan Levkivskyi in bpo-33416(1).)
The *note ast.parse(): 1a8. function has some new flags:
* ‘type_comments=True’ causes it to return the text of PEP 484(2) and
PEP 526(3) type comments associated with certain AST nodes;
* ‘mode='func_type'’ can be used to parse PEP 484(4) “signature type
comments” (returned for function definition AST nodes);
* ‘feature_version=(3, N)’ allows specifying an earlier Python 3
version. For example, ‘feature_version=(3, 4)’ will treat *note
async: 72d. and *note await: 1b9. as non-reserved words.
(Contributed by Guido van Rossum in bpo-35766(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=33416
(2) https://peps.python.org/pep-0484/
(3) https://peps.python.org/pep-0526/
(4) https://peps.python.org/pep-0484/
(5) https://bugs.python.org/issue?@action=redirect&bpo=35766
File: python.info, Node: asyncio<6>, Next: builtins<2>, Prev: ast<3>, Up: Improved Modules<6>
1.6.5.2 asyncio
...............
*note asyncio.run(): 478. has graduated from the provisional to stable
API. This function can be used to execute a *note coroutine: 48d. and
return the result while automatically managing the event loop. For
example:
import asyncio
async def main():
await asyncio.sleep(0)
return 42
asyncio.run(main())
This is 'roughly' equivalent to:
import asyncio
async def main():
await asyncio.sleep(0)
return 42
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(main())
finally:
asyncio.set_event_loop(None)
loop.close()
The actual implementation is significantly more complex. Thus, *note
asyncio.run(): 478. should be the preferred way of running asyncio
programs.
(Contributed by Yury Selivanov in bpo-32314(1).)
Running ‘python -m asyncio’ launches a natively async REPL. This allows
rapid experimentation with code that has a top-level *note await: 1b9.
There is no longer a need to directly call ‘asyncio.run()’ which would
spawn a new event loop on every invocation:
$ python -m asyncio
asyncio REPL 3.8.0
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(10, result='hello')
hello
(Contributed by Yury Selivanov in bpo-37028(2).)
The exception *note asyncio.CancelledError: 1b8. now inherits from *note
BaseException: 5b7. rather than *note Exception: 9d9. and no longer
inherits from *note concurrent.futures.CancelledError: 9da.
(Contributed by Yury Selivanov in bpo-32528(3).)
On Windows, the default event loop is now *note ProactorEventLoop: 605.
(Contributed by Victor Stinner in bpo-34687(4).)
*note ProactorEventLoop: 605. now also supports UDP. (Contributed by
Adam Meily and Andrew Svetlov in bpo-29883(5).)
*note ProactorEventLoop: 605. can now be interrupted by *note
KeyboardInterrupt: 9d1. (“CTRL+C”). (Contributed by Vladimir Matveev in
bpo-23057(6).)
Added *note asyncio.Task.get_coro(): 9db. for getting the wrapped
coroutine within an *note asyncio.Task: 1be. (Contributed by Alex
Grönholm in bpo-36999(7).)
Asyncio tasks can now be named, either by passing the ‘name’ keyword
argument to *note asyncio.create_task(): 1bf. or the *note
create_task(): 1c0. event loop method, or by calling the *note
set_name(): 9dc. method on the task object. The task name is visible in
the ‘repr()’ output of *note asyncio.Task: 1be. and can also be
retrieved using the *note get_name(): 9dd. method. (Contributed by Alex
Grönholm in bpo-34270(8).)
Added support for Happy Eyeballs(9) to *note
asyncio.loop.create_connection(): 5ff. To specify the behavior, two new
parameters have been added: 'happy_eyeballs_delay' and 'interleave'.
The Happy Eyeballs algorithm improves responsiveness in applications
that support IPv4 and IPv6 by attempting to simultaneously connect using
both. (Contributed by twisteroid ambassador in bpo-33530(10).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32314
(2) https://bugs.python.org/issue?@action=redirect&bpo=37028
(3) https://bugs.python.org/issue?@action=redirect&bpo=32528
(4) https://bugs.python.org/issue?@action=redirect&bpo=34687
(5) https://bugs.python.org/issue?@action=redirect&bpo=29883
(6) https://bugs.python.org/issue?@action=redirect&bpo=23057
(7) https://bugs.python.org/issue?@action=redirect&bpo=36999
(8) https://bugs.python.org/issue?@action=redirect&bpo=34270
(9) https://en.wikipedia.org/wiki/Happy_Eyeballs
(10) https://bugs.python.org/issue?@action=redirect&bpo=33530
File: python.info, Node: builtins<2>, Next: collections, Prev: asyncio<6>, Up: Improved Modules<6>
1.6.5.3 builtins
................
The *note compile(): 192. built-in has been improved to accept the
‘ast.PyCF_ALLOW_TOP_LEVEL_AWAIT’ flag. With this new flag passed, *note
compile(): 192. will allow top-level ‘await’, ‘async for’ and ‘async
with’ constructs that are usually considered invalid syntax.
Asynchronous code object marked with the ‘CO_COROUTINE’ flag may then be
returned. (Contributed by Matthias Bussonnier in bpo-34616(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34616
File: python.info, Node: collections, Next: cProfile, Prev: builtins<2>, Up: Improved Modules<6>
1.6.5.4 collections
...................
The *note _asdict(): 9e0. method for *note collections.namedtuple():
1ca. now returns a *note dict: 258. instead of a *note
collections.OrderedDict: 5d7. This works because regular dicts have
guaranteed ordering since Python 3.7. If the extra features of
‘OrderedDict’ are required, the suggested remediation is to cast the
result to the desired type: ‘OrderedDict(nt._asdict())’. (Contributed
by Raymond Hettinger in bpo-35864(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35864
File: python.info, Node: cProfile, Next: csv<2>, Prev: collections, Up: Improved Modules<6>
1.6.5.5 cProfile
................
The *note cProfile.Profile: 9e2. class can now be used as a context
manager. Profile a block of code by running:
import cProfile
with cProfile.Profile() as profiler:
# code to be profiled
...
(Contributed by Scott Sanderson in bpo-29235(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=29235
File: python.info, Node: csv<2>, Next: curses<3>, Prev: cProfile, Up: Improved Modules<6>
1.6.5.6 csv
...........
The *note csv.DictReader: 9e4. now returns instances of *note dict: 258.
instead of a *note collections.OrderedDict: 5d7. The tool is now faster
and uses less memory while still preserving the field order.
(Contributed by Michael Selik in bpo-34003(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34003
File: python.info, Node: curses<3>, Next: ctypes<2>, Prev: csv<2>, Up: Improved Modules<6>
1.6.5.7 curses
..............
Added a new variable holding structured version information for the
underlying ncurses library: *note ncurses_version: 9e6. (Contributed by
Serhiy Storchaka in bpo-31680(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31680
File: python.info, Node: ctypes<2>, Next: datetime<3>, Prev: curses<3>, Up: Improved Modules<6>
1.6.5.8 ctypes
..............
On Windows, *note CDLL: 9e8. and subclasses now accept a 'winmode'
parameter to specify flags for the underlying ‘LoadLibraryEx’ call. The
default flags are set to only load DLL dependencies from trusted
locations, including the path where the DLL is stored (if a full or
partial path is used to load the initial DLL) and paths added by *note
add_dll_directory(): 9e9. (Contributed by Steve Dower in bpo-36085(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36085
File: python.info, Node: datetime<3>, Next: functools<2>, Prev: ctypes<2>, Up: Improved Modules<6>
1.6.5.9 datetime
................
Added new alternate constructors *note datetime.date.fromisocalendar():
9eb. and *note datetime.datetime.fromisocalendar(): 9ec, which construct
*note date: 1cd. and *note datetime: 1cc. objects respectively from ISO
year, week number, and weekday; these are the inverse of each class’s
‘isocalendar’ method. (Contributed by Paul Ganssle in bpo-36004(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36004
File: python.info, Node: functools<2>, Next: gc<3>, Prev: datetime<3>, Up: Improved Modules<6>
1.6.5.10 functools
..................
*note functools.lru_cache(): 9ee. can now be used as a straight
decorator rather than as a function returning a decorator. So both of
these are now supported:
@lru_cache
def f(x):
...
@lru_cache(maxsize=256)
def f(x):
...
(Contributed by Raymond Hettinger in bpo-36772(1).)
Added a new *note functools.cached_property(): 53e. decorator, for
computed properties cached for the life of the instance.
import functools
import statistics
class Dataset:
def __init__(self, sequence_of_numbers):
self.data = sequence_of_numbers
@functools.cached_property
def variance(self):
return statistics.variance(self.data)
(Contributed by Carl Meyer in bpo-21145(2))
Added a new *note functools.singledispatchmethod(): 9ef. decorator that
converts methods into *note generic functions: 9f0. using *note single
dispatch: 9f1.:
from functools import singledispatchmethod
from contextlib import suppress
class TaskManager:
def __init__(self, tasks):
self.tasks = list(tasks)
@singledispatchmethod
def discard(self, value):
with suppress(ValueError):
self.tasks.remove(value)
@discard.register(list)
def _(self, tasks):
targets = set(tasks)
self.tasks = [x for x in self.tasks if x not in targets]
(Contributed by Ethan Smith in bpo-32380(3))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36772
(2) https://bugs.python.org/issue?@action=redirect&bpo=21145
(3) https://bugs.python.org/issue?@action=redirect&bpo=32380
File: python.info, Node: gc<3>, Next: gettext, Prev: functools<2>, Up: Improved Modules<6>
1.6.5.11 gc
...........
*note get_objects(): 7ff. can now receive an optional 'generation'
parameter indicating a generation to get objects from. (Contributed by
Pablo Galindo in bpo-36016(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36016
File: python.info, Node: gettext, Next: gzip<3>, Prev: gc<3>, Up: Improved Modules<6>
1.6.5.12 gettext
................
Added *note pgettext(): 9f4. and its variants. (Contributed by Franz
Glasner, Éric Araujo, and Cheryl Sabella in bpo-2504(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2504
File: python.info, Node: gzip<3>, Next: IDLE and idlelib<4>, Prev: gettext, Up: Improved Modules<6>
1.6.5.13 gzip
.............
Added the 'mtime' parameter to *note gzip.compress(): 63a. for
reproducible output. (Contributed by Guo Ci Teo in bpo-34898(1).)
A *note BadGzipFile: 9f6. exception is now raised instead of *note
OSError: 413. for certain types of invalid or corrupt gzip files.
(Contributed by Filip Gruszczyński, Michele Orrù, and Zackery Spytz in
bpo-6584(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34898
(2) https://bugs.python.org/issue?@action=redirect&bpo=6584
File: python.info, Node: IDLE and idlelib<4>, Next: inspect<5>, Prev: gzip<3>, Up: Improved Modules<6>
1.6.5.14 IDLE and idlelib
.........................
Output over N lines (50 by default) is squeezed down to a button. N can
be changed in the PyShell section of the General page of the Settings
dialog. Fewer, but possibly extra long, lines can be squeezed by right
clicking on the output. Squeezed output can be expanded in place by
double-clicking the button or into the clipboard or a separate window by
right-clicking the button. (Contributed by Tal Einat in
bpo-1529353(1).)
Add “Run Customized” to the Run menu to run a module with customized
settings. Any command line arguments entered are added to sys.argv.
They also re-appear in the box for the next customized run. One can
also suppress the normal Shell main module restart. (Contributed by
Cheryl Sabella, Terry Jan Reedy, and others in bpo-5680(2) and
bpo-37627(3).)
Added optional line numbers for IDLE editor windows. Windows open
without line numbers unless set otherwise in the General tab of the
configuration dialog. Line numbers for an existing window are shown and
hidden in the Options menu. (Contributed by Tal Einat and Saimadhav
Heblikar in bpo-17535(4).)
OS native encoding is now used for converting between Python strings and
Tcl objects. This allows IDLE to work with emoji and other non-BMP
characters. These characters can be displayed or copied and pasted to
or from the clipboard. Converting strings from Tcl to Python and back
now never fails. (Many people worked on this for eight years but the
problem was finally solved by Serhiy Storchaka in bpo-13153(5).)
New in 3.8.1:
Add option to toggle cursor blink off. (Contributed by Zackery Spytz in
bpo-4603(6).)
Escape key now closes IDLE completion windows. (Contributed by Johnny
Najera in bpo-38944(7).)
The changes above have been backported to 3.7 maintenance releases.
Add keywords to module name completion list. (Contributed by Terry J.
Reedy in bpo-37765(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1529353
(2) https://bugs.python.org/issue?@action=redirect&bpo=5680
(3) https://bugs.python.org/issue?@action=redirect&bpo=37627
(4) https://bugs.python.org/issue?@action=redirect&bpo=17535
(5) https://bugs.python.org/issue?@action=redirect&bpo=13153
(6) https://bugs.python.org/issue?@action=redirect&bpo=4603
(7) https://bugs.python.org/issue?@action=redirect&bpo=38944
(8) https://bugs.python.org/issue?@action=redirect&bpo=37765
File: python.info, Node: inspect<5>, Next: io<3>, Prev: IDLE and idlelib<4>, Up: Improved Modules<6>
1.6.5.15 inspect
................
The *note inspect.getdoc(): 9f9. function can now find docstrings for
‘__slots__’ if that attribute is a *note dict: 258. where the values are
docstrings. This provides documentation options similar to what we
already have for *note property(): 194, *note classmethod(): 166, and
*note staticmethod(): 412.:
class AudioClip:
__slots__ = {'bit_rate': 'expressed in kilohertz to one decimal place',
'duration': 'in seconds, rounded up to an integer'}
def __init__(self, bit_rate, duration):
self.bit_rate = round(bit_rate / 1000.0, 1)
self.duration = ceil(duration)
(Contributed by Raymond Hettinger in bpo-36326(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36326
File: python.info, Node: io<3>, Next: itertools<4>, Prev: inspect<5>, Up: Improved Modules<6>
1.6.5.16 io
...........
In development mode (*note -X: 176. ‘env’) and in *note debug build:
1fb, the *note io.IOBase: 1f7. finalizer now logs the exception if the
‘close()’ method fails. The exception is ignored silently by default in
release build. (Contributed by Victor Stinner in bpo-18748(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18748
File: python.info, Node: itertools<4>, Next: json tool, Prev: io<3>, Up: Improved Modules<6>
1.6.5.17 itertools
..................
The *note itertools.accumulate(): 9fc. function added an option
'initial' keyword argument to specify an initial value:
>>> from itertools import accumulate
>>> list(accumulate([10, 5, 30, 15], initial=1000))
[1000, 1010, 1015, 1045, 1060]
(Contributed by Lisa Roach in bpo-34659(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34659
File: python.info, Node: json tool, Next: logging<2>, Prev: itertools<4>, Up: Improved Modules<6>
1.6.5.18 json.tool
..................
Add option ‘--json-lines’ to parse every input line as a separate JSON
object. (Contributed by Weipeng Hong in bpo-31553(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31553
File: python.info, Node: logging<2>, Next: math<5>, Prev: json tool, Up: Improved Modules<6>
1.6.5.19 logging
................
Added a 'force' keyword argument to *note logging.basicConfig(): 9ff.
When set to true, any existing handlers attached to the root logger are
removed and closed before carrying out the configuration specified by
the other arguments.
This solves a long-standing problem. Once a logger or 'basicConfig()'
had been called, subsequent calls to 'basicConfig()' were silently
ignored. This made it difficult to update, experiment with, or teach
the various logging configuration options using the interactive prompt
or a Jupyter notebook.
(Suggested by Raymond Hettinger, implemented by Donghee Na, and reviewed
by Vinay Sajip in bpo-33897(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=33897
File: python.info, Node: math<5>, Next: mmap<2>, Prev: logging<2>, Up: Improved Modules<6>
1.6.5.20 math
.............
Added new function *note math.dist(): a01. for computing Euclidean
distance between two points. (Contributed by Raymond Hettinger in
bpo-33089(1).)
Expanded the *note math.hypot(): a02. function to handle multiple
dimensions. Formerly, it only supported the 2-D case. (Contributed by
Raymond Hettinger in bpo-33089(2).)
Added new function, *note math.prod(): a03, as analogous function to
*note sum(): 466. that returns the product of a ‘start’ value (default:
1) times an iterable of numbers:
>>> prior = 0.8
>>> likelihoods = [0.625, 0.84, 0.30]
>>> math.prod(likelihoods, start=prior)
0.126
(Contributed by Pablo Galindo in bpo-35606(3).)
Added two new combinatoric functions *note math.perm(): 6cc. and *note
math.comb(): 6cb.:
>>> math.perm(10, 3) # Permutations of 10 things taken 3 at a time
720
>>> math.comb(10, 3) # Combinations of 10 things taken 3 at a time
120
(Contributed by Yash Aggarwal, Keller Fuchs, Serhiy Storchaka, and
Raymond Hettinger in bpo-37128(4), bpo-37178(5), and bpo-35431(6).)
Added a new function *note math.isqrt(): a04. for computing accurate
integer square roots without conversion to floating point. The new
function supports arbitrarily large integers. It is faster than
‘floor(sqrt(n))’ but slower than *note math.sqrt(): a05.:
>>> r = 650320427
>>> s = r ** 2
>>> isqrt(s - 1) # correct
650320426
>>> floor(sqrt(s - 1)) # incorrect
650320427
(Contributed by Mark Dickinson in bpo-36887(7).)
The function *note math.factorial(): 947. no longer accepts arguments
that are not int-like. (Contributed by Pablo Galindo in bpo-33083(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=33089
(2) https://bugs.python.org/issue?@action=redirect&bpo=33089
(3) https://bugs.python.org/issue?@action=redirect&bpo=35606
(4) https://bugs.python.org/issue?@action=redirect&bpo=37128
(5) https://bugs.python.org/issue?@action=redirect&bpo=37178
(6) https://bugs.python.org/issue?@action=redirect&bpo=35431
(7) https://bugs.python.org/issue?@action=redirect&bpo=36887
(8) https://bugs.python.org/issue?@action=redirect&bpo=33083
File: python.info, Node: mmap<2>, Next: multiprocessing<3>, Prev: math<5>, Up: Improved Modules<6>
1.6.5.21 mmap
.............
The *note mmap.mmap: 20d. class now has an *note madvise(): a07. method
to access the ‘madvise()’ system call. (Contributed by Zackery Spytz in
bpo-32941(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32941
File: python.info, Node: multiprocessing<3>, Next: os<6>, Prev: mmap<2>, Up: Improved Modules<6>
1.6.5.22 multiprocessing
........................
Added new *note multiprocessing.shared_memory: 99. module. (Contributed
by Davin Potts in bpo-35813(1).)
On macOS, the 'spawn' start method is now used by default. (Contributed
by Victor Stinner in bpo-33725(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35813
(2) https://bugs.python.org/issue?@action=redirect&bpo=33725
File: python.info, Node: os<6>, Next: os path<5>, Prev: multiprocessing<3>, Up: Improved Modules<6>
1.6.5.23 os
...........
Added new function *note add_dll_directory(): 9e9. on Windows for
providing additional search paths for native dependencies when importing
extension modules or loading DLLs using *note ctypes: 2a. (Contributed
by Steve Dower in bpo-36085(1).)
A new *note os.memfd_create(): a0a. function was added to wrap the
‘memfd_create()’ syscall. (Contributed by Zackery Spytz and Christian
Heimes in bpo-26836(2).)
On Windows, much of the manual logic for handling reparse points
(including symlinks and directory junctions) has been delegated to the
operating system. Specifically, *note os.stat(): 49c. will now traverse
anything supported by the operating system, while *note os.lstat(): 49d.
will only open reparse points that identify as “name surrogates” while
others are opened as for *note os.stat(): 49c. In all cases,
‘stat_result.st_mode’ will only have ‘S_IFLNK’ set for symbolic links
and not other kinds of reparse points. To identify other kinds of
reparse point, check the new ‘stat_result.st_reparse_tag’ attribute.
On Windows, *note os.readlink(): 91f. is now able to read directory
junctions. Note that *note islink(): a0b. will return ‘False’ for
directory junctions, and so code that checks ‘islink’ first will
continue to treat junctions as directories, while code that handles
errors from *note os.readlink(): 91f. may now treat junctions as links.
(Contributed by Steve Dower in bpo-37834(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36085
(2) https://bugs.python.org/issue?@action=redirect&bpo=26836
(3) https://bugs.python.org/issue?@action=redirect&bpo=37834
File: python.info, Node: os path<5>, Next: pathlib<7>, Prev: os<6>, Up: Improved Modules<6>
1.6.5.24 os.path
................
*note os.path: a2. functions that return a boolean result like *note
exists(): a0d, *note lexists(): a0e, *note isdir(): a0f, *note isfile():
a10, *note islink(): a0b, and *note ismount(): a11. now return ‘False’
instead of raising *note ValueError: 204. or its subclasses *note
UnicodeEncodeError: 673. and *note UnicodeDecodeError: a12. for paths
that contain characters or bytes unrepresentable at the OS level.
(Contributed by Serhiy Storchaka in bpo-33721(1).)
*note expanduser(): a13. on Windows now prefers the ‘USERPROFILE’
environment variable and does not use ‘HOME’, which is not normally set
for regular user accounts. (Contributed by Anthony Sottile in
bpo-36264(2).)
*note isdir(): a0f. on Windows no longer returns ‘True’ for a link to a
non-existent directory.
*note realpath(): 227. on Windows now resolves reparse points, including
symlinks and directory junctions.
(Contributed by Steve Dower in bpo-37834(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=33721
(2) https://bugs.python.org/issue?@action=redirect&bpo=36264
(3) https://bugs.python.org/issue?@action=redirect&bpo=37834
File: python.info, Node: pathlib<7>, Next: pickle, Prev: os path<5>, Up: Improved Modules<6>
1.6.5.25 pathlib
................
*note pathlib.Path: 22b. methods that return a boolean result like *note
exists(): a15, *note is_dir(): 232, *note is_file(): 231, *note
is_mount(): a16, *note is_symlink(): a17, *note is_block_device(): a18,
*note is_char_device(): a19, *note is_fifo(): a1a, *note is_socket():
a1b. now return ‘False’ instead of raising *note ValueError: 204. or its
subclass *note UnicodeEncodeError: 673. for paths that contain
characters unrepresentable at the OS level. (Contributed by Serhiy
Storchaka in bpo-33721(1).)
Added ‘pathlib.Path.link_to()’ which creates a hard link pointing to a
path. (Contributed by Joannah Nanjekye in bpo-26978(2)) Note that
‘link_to’ was deprecated in 3.10 and removed in 3.12 in favor of a
‘hardlink_to’ method added in 3.10 which matches the semantics of the
existing ‘symlink_to’ method.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=33721
(2) https://bugs.python.org/issue?@action=redirect&bpo=26978
File: python.info, Node: pickle, Next: plistlib, Prev: pathlib<7>, Up: Improved Modules<6>
1.6.5.26 pickle
...............
*note pickle: a6. extensions subclassing the C-optimized *note Pickler:
a1d. can now override the pickling logic of functions and classes by
defining the special *note reducer_override(): a1e. method.
(Contributed by Pierre Glaser and Olivier Grisel in bpo-35900(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35900
File: python.info, Node: plistlib, Next: pprint<3>, Prev: pickle, Up: Improved Modules<6>
1.6.5.27 plistlib
.................
Added new *note plistlib.UID: a20. and enabled support for reading and
writing NSKeyedArchiver-encoded binary plists. (Contributed by Jon
Janzen in bpo-26707(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26707
File: python.info, Node: pprint<3>, Next: py_compile<2>, Prev: plistlib, Up: Improved Modules<6>
1.6.5.28 pprint
...............
The *note pprint: ae. module added a 'sort_dicts' parameter to several
functions. By default, those functions continue to sort dictionaries
before rendering or printing. However, if 'sort_dicts' is set to false,
the dictionaries retain the order that keys were inserted. This can be
useful for comparison to JSON inputs during debugging.
In addition, there is a convenience new function, *note pprint.pp():
a22. that is like *note pprint.pprint(): 824. but with 'sort_dicts'
defaulting to ‘False’:
>>> from pprint import pprint, pp
>>> d = dict(source='input.txt', operation='filter', destination='output.txt')
>>> pp(d, width=40) # Original order
{'source': 'input.txt',
'operation': 'filter',
'destination': 'output.txt'}
>>> pprint(d, width=40) # Keys sorted alphabetically
{'destination': 'output.txt',
'operation': 'filter',
'source': 'input.txt'}
(Contributed by Rémi Lapeyre in bpo-30670(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30670
File: python.info, Node: py_compile<2>, Next: shlex, Prev: pprint<3>, Up: Improved Modules<6>
1.6.5.29 py_compile
...................
*note py_compile.compile(): a24. now supports silent mode. (Contributed
by Joannah Nanjekye in bpo-22640(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22640
File: python.info, Node: shlex, Next: shutil<4>, Prev: py_compile<2>, Up: Improved Modules<6>
1.6.5.30 shlex
..............
The new *note shlex.join(): a26. function acts as the inverse of *note
shlex.split(): 538. (Contributed by Bo Bayles in bpo-32102(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32102
File: python.info, Node: shutil<4>, Next: socket<4>, Prev: shlex, Up: Improved Modules<6>
1.6.5.31 shutil
...............
*note shutil.copytree(): a28. now accepts a new ‘dirs_exist_ok’ keyword
argument. (Contributed by Josh Bronson in bpo-20849(1).)
*note shutil.make_archive(): 4af. now defaults to the modern pax
(POSIX.1-2001) format for new archives to improve portability and
standards conformance, inherited from the corresponding change to the
*note tarfile: de. module. (Contributed by C.A.M. Gerlach in
bpo-30661(2).)
*note shutil.rmtree(): 2fb. on Windows now removes directory junctions
without recursively removing their contents first. (Contributed by
Steve Dower in bpo-37834(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20849
(2) https://bugs.python.org/issue?@action=redirect&bpo=30661
(3) https://bugs.python.org/issue?@action=redirect&bpo=37834
File: python.info, Node: socket<4>, Next: ssl<4>, Prev: shutil<4>, Up: Improved Modules<6>
1.6.5.32 socket
...............
Added *note create_server(): a2a. and *note has_dualstack_ipv6(): a2b.
convenience functions to automate the necessary tasks usually involved
when creating a server socket, including accepting both IPv4 and IPv6
connections on the same socket. (Contributed by Giampaolo Rodolà in
bpo-17561(1).)
The *note socket.if_nameindex(): a2c, *note socket.if_nametoindex():
a2d, and *note socket.if_indextoname(): a2e. functions have been
implemented on Windows. (Contributed by Zackery Spytz in bpo-37007(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17561
(2) https://bugs.python.org/issue?@action=redirect&bpo=37007
File: python.info, Node: ssl<4>, Next: statistics<4>, Prev: socket<4>, Up: Improved Modules<6>
1.6.5.33 ssl
............
Added *note post_handshake_auth: a30. to enable and *note
verify_client_post_handshake(): a31. to initiate TLS 1.3 post-handshake
authentication. (Contributed by Christian Heimes in bpo-34670(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34670
File: python.info, Node: statistics<4>, Next: sys<7>, Prev: ssl<4>, Up: Improved Modules<6>
1.6.5.34 statistics
...................
Added *note statistics.fmean(): a33. as a faster, floating-point variant
of *note statistics.mean(): 6cd. (Contributed by Raymond Hettinger and
Steven D’Aprano in bpo-35904(1).)
Added *note statistics.geometric_mean(): a34. (Contributed by Raymond
Hettinger in bpo-27181(2).)
Added *note statistics.multimode(): a35. that returns a list of the most
common values. (Contributed by Raymond Hettinger in bpo-35892(3).)
Added *note statistics.quantiles(): a36. that divides data or a
distribution in to equiprobable intervals (e.g. quartiles, deciles, or
percentiles). (Contributed by Raymond Hettinger in bpo-36546(4).)
Added *note statistics.NormalDist: a37, a tool for creating and
manipulating normal distributions of a random variable. (Contributed by
Raymond Hettinger in bpo-36018(5).)
>>> temperature_feb = NormalDist.from_samples([4, 12, -3, 2, 7, 14])
>>> temperature_feb.mean
6.0
>>> temperature_feb.stdev
6.356099432828281
>>> temperature_feb.cdf(3) # Chance of being under 3 degrees
0.3184678262814532
>>> # Relative chance of being 7 degrees versus 10 degrees
>>> temperature_feb.pdf(7) / temperature_feb.pdf(10)
1.2039930378537762
>>> el_niño = NormalDist(4, 2.5)
>>> temperature_feb += el_niño # Add in a climate effect
>>> temperature_feb
NormalDist(mu=10.0, sigma=6.830080526611674)
>>> temperature_feb * (9/5) + 32 # Convert to Fahrenheit
NormalDist(mu=50.0, sigma=12.294144947901014)
>>> temperature_feb.samples(3) # Generate random samples
[7.672102882379219, 12.000027119750287, 4.647488369766392]
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35904
(2) https://bugs.python.org/issue?@action=redirect&bpo=27181
(3) https://bugs.python.org/issue?@action=redirect&bpo=35892
(4) https://bugs.python.org/issue?@action=redirect&bpo=36546
(5) https://bugs.python.org/issue?@action=redirect&bpo=36018
File: python.info, Node: sys<7>, Next: tarfile<5>, Prev: statistics<4>, Up: Improved Modules<6>
1.6.5.35 sys
............
Add new *note sys.unraisablehook(): 1f9. function which can be
overridden to control how “unraisable exceptions” are handled. It is
called when an exception has occurred but there is no way for Python to
handle it. For example, when a destructor raises an exception or during
garbage collection (*note gc.collect(): a39.). (Contributed by Victor
Stinner in bpo-36829(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36829
File: python.info, Node: tarfile<5>, Next: threading<4>, Prev: sys<7>, Up: Improved Modules<6>
1.6.5.36 tarfile
................
The *note tarfile: de. module now defaults to the modern pax
(POSIX.1-2001) format for new archives, instead of the previous
GNU-specific one. This improves cross-platform portability with a
consistent encoding (UTF-8) in a standardized and extensible format, and
offers several other benefits. (Contributed by C.A.M. Gerlach in
bpo-36268(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36268
File: python.info, Node: threading<4>, Next: tokenize<2>, Prev: tarfile<5>, Up: Improved Modules<6>
1.6.5.37 threading
..................
Add a new *note threading.excepthook(): 847. function which handles
uncaught *note threading.Thread.run(): a3c. exception. It can be
overridden to control how uncaught *note threading.Thread.run(): a3c.
exceptions are handled. (Contributed by Victor Stinner in
bpo-1230540(1).)
Add a new *note threading.get_native_id(): a3d. function and a *note
native_id: a3e. attribute to the *note threading.Thread: 94c. class.
These return the native integral Thread ID of the current thread
assigned by the kernel. This feature is only available on certain
platforms, see *note get_native_id: a3d. for more information.
(Contributed by Jake Tesler in bpo-36084(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1230540
(2) https://bugs.python.org/issue?@action=redirect&bpo=36084
File: python.info, Node: tokenize<2>, Next: tkinter<4>, Prev: threading<4>, Up: Improved Modules<6>
1.6.5.38 tokenize
.................
The *note tokenize: fb. module now implicitly emits a ‘NEWLINE’ token
when provided with input that does not have a trailing new line. This
behavior now matches what the C tokenizer does internally. (Contributed
by Ammar Askar in bpo-33899(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=33899
File: python.info, Node: tkinter<4>, Next: time<4>, Prev: tokenize<2>, Up: Improved Modules<6>
1.6.5.39 tkinter
................
Added methods ‘selection_from()’, ‘selection_present()’,
‘selection_range()’ and ‘selection_to()’ in the ‘tkinter.Spinbox’ class.
(Contributed by Juliette Monsel in bpo-34829(1).)
Added method ‘moveto()’ in the ‘tkinter.Canvas’ class. (Contributed by
Juliette Monsel in bpo-23831(2).)
The ‘tkinter.PhotoImage’ class now has ‘transparency_get()’ and
‘transparency_set()’ methods. (Contributed by Zackery Spytz in
bpo-25451(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34829
(2) https://bugs.python.org/issue?@action=redirect&bpo=23831
(3) https://bugs.python.org/issue?@action=redirect&bpo=25451
File: python.info, Node: time<4>, Next: typing<8>, Prev: tkinter<4>, Up: Improved Modules<6>
1.6.5.40 time
.............
Added new clock *note CLOCK_UPTIME_RAW: a42. for macOS 10.12.
(Contributed by Joannah Nanjekye in bpo-35702(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35702
File: python.info, Node: typing<8>, Next: unicodedata<5>, Prev: time<4>, Up: Improved Modules<6>
1.6.5.41 typing
...............
The *note typing: 104. module incorporates several new features:
* A dictionary type with per-key types. See PEP 589(1) and *note
typing.TypedDict: 162. TypedDict uses only string keys. By
default, every key is required to be present. Specify
“total=False” to allow keys to be optional:
class Location(TypedDict, total=False):
lat_long: tuple
grid_square: str
xy_coordinate: tuple
* Literal types. See PEP 586(2) and *note typing.Literal: 851.
Literal types indicate that a parameter or return value is
constrained to one or more specific literal values:
def get_status(port: int) -> Literal['connected', 'disconnected']:
...
* “Final” variables, functions, methods and classes. See PEP 591(3),
*note typing.Final: 269. and *note typing.final(): 6a9. The final
qualifier instructs a static type checker to restrict subclassing,
overriding, or reassignment:
pi: Final[float] = 3.1415926536
* Protocol definitions. See PEP 544(4), *note typing.Protocol: 266.
and *note typing.runtime_checkable(): 43e. Simple ABCs like *note
typing.SupportsInt: a44. are now ‘Protocol’ subclasses.
* New protocol class *note typing.SupportsIndex: a45.
* New functions *note typing.get_origin(): a46. and *note
typing.get_args(): 87c.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0589/
(2) https://peps.python.org/pep-0586/
(3) https://peps.python.org/pep-0591/
(4) https://peps.python.org/pep-0544/
File: python.info, Node: unicodedata<5>, Next: unittest<6>, Prev: typing<8>, Up: Improved Modules<6>
1.6.5.42 unicodedata
....................
The *note unicodedata: 105. module has been upgraded to use the Unicode
12.1.0(1) release.
New function *note is_normalized(): a48. can be used to verify a string
is in a specific normal form, often much faster than by actually
normalizing the string. (Contributed by Max Belanger, David Euresti,
and Greg Price in bpo-32285(2) and bpo-37966(3)).
---------- Footnotes ----------
(1) https://blog.unicode.org/2019/05/unicode-12-1-en.html
(2) https://bugs.python.org/issue?@action=redirect&bpo=32285
(3) https://bugs.python.org/issue?@action=redirect&bpo=37966
File: python.info, Node: unittest<6>, Next: venv<4>, Prev: unicodedata<5>, Up: Improved Modules<6>
1.6.5.43 unittest
.................
Added *note AsyncMock: a4a. to support an asynchronous version of *note
Mock: a4b. Appropriate new assert functions for testing have been added
as well. (Contributed by Lisa Roach in bpo-26467(1)).
Added *note addModuleCleanup(): a4c. and *note addClassCleanup(): a4d.
to unittest to support cleanups for ‘setUpModule()’ and *note
setUpClass(): a4e. (Contributed by Lisa Roach in bpo-24412(2).)
Several mock assert functions now also print a list of actual calls upon
failure. (Contributed by Petter Strandmark in bpo-35047(3).)
*note unittest: 106. module gained support for coroutines to be used as
test cases with *note unittest.IsolatedAsyncioTestCase: 306.
(Contributed by Andrew Svetlov in bpo-32972(4).)
Example:
import unittest
class TestRequest(unittest.IsolatedAsyncioTestCase):
async def asyncSetUp(self):
self.connection = await AsyncConnection()
async def test_get(self):
response = await self.connection.get("https://example.com")
self.assertEqual(response.status_code, 200)
async def asyncTearDown(self):
await self.connection.close()
if __name__ == "__main__":
unittest.main()
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26467
(2) https://bugs.python.org/issue?@action=redirect&bpo=24412
(3) https://bugs.python.org/issue?@action=redirect&bpo=35047
(4) https://bugs.python.org/issue?@action=redirect&bpo=32972
File: python.info, Node: venv<4>, Next: weakref, Prev: unittest<6>, Up: Improved Modules<6>
1.6.5.44 venv
.............
*note venv: 111. now includes an ‘Activate.ps1’ script on all platforms
for activating virtual environments under PowerShell Core 6.1.
(Contributed by Brett Cannon in bpo-32718(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32718
File: python.info, Node: weakref, Next: xml<4>, Prev: venv<4>, Up: Improved Modules<6>
1.6.5.45 weakref
................
The proxy objects returned by *note weakref.proxy(): a51. now support
the matrix multiplication operators ‘@’ and ‘@=’ in addition to the
other numeric operators. (Contributed by Mark Dickinson in
bpo-36669(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36669
File: python.info, Node: xml<4>, Next: xmlrpc, Prev: weakref, Up: Improved Modules<6>
1.6.5.46 xml
............
As mitigation against DTD and external entity retrieval, the *note
xml.dom.minidom: 122. and *note xml.sax: 129. modules no longer process
external entities by default. (Contributed by Christian Heimes in
bpo-17239(1).)
The ‘.find*()’ methods in the *note xml.etree.ElementTree: 125. module
support wildcard searches like ‘{*}tag’ which ignores the namespace and
‘{namespace}*’ which returns all tags in the given namespace.
(Contributed by Stefan Behnel in bpo-28238(2).)
The *note xml.etree.ElementTree: 125. module provides a new function
‘–xml.etree.ElementTree.canonicalize()’ that implements C14N 2.0.
(Contributed by Stefan Behnel in bpo-13611(3).)
The target object of *note xml.etree.ElementTree.XMLParser: a53. can
receive namespace declaration events through the new callback methods
‘start_ns()’ and ‘end_ns()’. Additionally, the *note
xml.etree.ElementTree.TreeBuilder: a54. target can be configured to
process events about comments and processing instructions to include
them in the generated tree. (Contributed by Stefan Behnel in
bpo-36676(4) and bpo-36673(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17239
(2) https://bugs.python.org/issue?@action=redirect&bpo=28238
(3) https://bugs.python.org/issue?@action=redirect&bpo=13611
(4) https://bugs.python.org/issue?@action=redirect&bpo=36676
(5) https://bugs.python.org/issue?@action=redirect&bpo=36673
File: python.info, Node: xmlrpc, Prev: xml<4>, Up: Improved Modules<6>
1.6.5.47 xmlrpc
...............
*note xmlrpc.client.ServerProxy: a56. now supports an optional 'headers'
keyword argument for a sequence of HTTP headers to be sent with each
request. Among other things, this makes it possible to upgrade from
default basic authentication to faster session authentication.
(Contributed by Cédric Krier in bpo-35153(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35153
File: python.info, Node: Optimizations<6>, Next: Build and C API Changes, Prev: Improved Modules<6>, Up: What’s New In Python 3 8
1.6.6 Optimizations
-------------------
* The *note subprocess: d6. module can now use the *note
os.posix_spawn(): 222. function in some cases for better
performance. Currently, it is only used on macOS and Linux (using
glibc 2.24 or newer) if all these conditions are met:
* 'close_fds' is false;
* 'preexec_fn', 'pass_fds', 'cwd' and 'start_new_session'
parameters are not set;
* the 'executable' path contains a directory.
(Contributed by Joannah Nanjekye and Victor Stinner in
bpo-35537(1).)
* *note shutil.copyfile(): a58, *note shutil.copy(): a59, *note
shutil.copy2(): a5a, *note shutil.copytree(): a28. and *note
shutil.move(): a5b. use platform-specific “fast-copy” syscalls on
Linux and macOS in order to copy the file more efficiently.
“fast-copy” means that the copying operation occurs within the
kernel, avoiding the use of userspace buffers in Python as in
“‘outfd.write(infd.read())’”. On Windows *note shutil.copyfile():
a58. uses a bigger default buffer size (1 MiB instead of 16 KiB)
and a *note memoryview(): 464.-based variant of *note
shutil.copyfileobj(): a5c. is used. The speedup for copying a 512
MiB file within the same partition is about +26% on Linux, +50% on
macOS and +40% on Windows. Also, much less CPU cycles are
consumed. See *note Platform-dependent efficient copy operations:
a5d. section. (Contributed by Giampaolo Rodolà in bpo-33671(2).)
* *note shutil.copytree(): a28. uses *note os.scandir(): a5e.
function and all copy functions depending from it use cached *note
os.stat(): 49c. values. The speedup for copying a directory with
8000 files is around +9% on Linux, +20% on Windows and +30% on a
Windows SMB share. Also the number of *note os.stat(): 49c.
syscalls is reduced by 38% making *note shutil.copytree(): a28.
especially faster on network filesystems. (Contributed by
Giampaolo Rodolà in bpo-33695(3).)
* The default protocol in the *note pickle: a6. module is now
Protocol 4, first introduced in Python 3.4. It offers better
performance and smaller size compared to Protocol 3 available since
Python 3.0.
* Removed one *note Py_ssize_t: a5f. member from ‘PyGC_Head’. All GC
tracked objects (e.g. tuple, list, dict) size is reduced 4 or 8
bytes. (Contributed by Inada Naoki in bpo-33597(4).)
* *note uuid.UUID: a60. now uses ‘__slots__’ to reduce its memory
footprint. (Contributed by Wouter Bolsterlee and Tal Einat in
bpo-30977(5))
* Improved performance of *note operator.itemgetter(): a61. by 33%.
Optimized argument handling and added a fast path for the common
case of a single non-negative integer index into a tuple (which is
the typical use case in the standard library). (Contributed by
Raymond Hettinger in bpo-35664(6).)
* Sped-up field lookups in *note collections.namedtuple(): 1ca. They
are now more than two times faster, making them the fastest form of
instance variable lookup in Python. (Contributed by Raymond
Hettinger, Pablo Galindo, and Joe Jevnik, Serhiy Storchaka in
bpo-32492(7).)
* The *note list: 60d. constructor does not overallocate the internal
item buffer if the input iterable has a known length (the input
implements ‘__len__’). This makes the created list 12% smaller on
average. (Contributed by Raymond Hettinger and Pablo Galindo in
bpo-33234(8).)
* Doubled the speed of class variable writes. When a non-dunder
attribute was updated, there was an unnecessary call to update
slots. (Contributed by Stefan Behnel, Pablo Galindo Salgado,
Raymond Hettinger, Neil Schemenauer, and Serhiy Storchaka in
bpo-36012(9).)
* Reduced an overhead of converting arguments passed to many builtin
functions and methods. This sped up calling some simple builtin
functions and methods up to 20–50%. (Contributed by Serhiy
Storchaka in bpo-23867(10), bpo-35582(11) and bpo-36127(12).)
* ‘LOAD_GLOBAL’ instruction now uses new “per opcode cache”
mechanism. It is about 40% faster now. (Contributed by Yury
Selivanov and Inada Naoki in bpo-26219(13).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35537
(2) https://bugs.python.org/issue?@action=redirect&bpo=33671
(3) https://bugs.python.org/issue?@action=redirect&bpo=33695
(4) https://bugs.python.org/issue?@action=redirect&bpo=33597
(5) https://bugs.python.org/issue?@action=redirect&bpo=30977
(6) https://bugs.python.org/issue?@action=redirect&bpo=35664
(7) https://bugs.python.org/issue?@action=redirect&bpo=32492
(8) https://bugs.python.org/issue?@action=redirect&bpo=33234
(9) https://bugs.python.org/issue?@action=redirect&bpo=36012
(10) https://bugs.python.org/issue?@action=redirect&bpo=23867
(11) https://bugs.python.org/issue?@action=redirect&bpo=35582
(12) https://bugs.python.org/issue?@action=redirect&bpo=36127
(13) https://bugs.python.org/issue?@action=redirect&bpo=26219
File: python.info, Node: Build and C API Changes, Next: Deprecated<8>, Prev: Optimizations<6>, Up: What’s New In Python 3 8
1.6.7 Build and C API Changes
-----------------------------
* Default *note sys.abiflags: a63. became an empty string: the ‘m’
flag for pymalloc became useless (builds with and without pymalloc
are ABI compatible) and so has been removed. (Contributed by
Victor Stinner in bpo-36707(1).)
Example of changes:
* Only ‘python3.8’ program is installed, ‘python3.8m’ program is
gone.
* Only ‘python3.8-config’ script is installed,
‘python3.8m-config’ script is gone.
* The ‘m’ flag has been removed from the suffix of dynamic
library filenames: extension modules in the standard library
as well as those produced and installed by third-party
packages, like those downloaded from PyPI. On Linux, for
example, the Python 3.7 suffix
‘.cpython-37m-x86_64-linux-gnu.so’ became
‘.cpython-38-x86_64-linux-gnu.so’ in Python 3.8.
* The header files have been reorganized to better separate the
different kinds of APIs:
* ‘Include/*.h’ should be the portable public stable C API.
* ‘Include/cpython/*.h’ should be the unstable C API specific to
CPython; public API, with some private API prefixed by ‘_Py’
or ‘_PY’.
* ‘Include/internal/*.h’ is the private internal C API very
specific to CPython. This API comes with no backward
compatibility warranty and should not be used outside CPython.
It is only exposed for very specific needs like debuggers and
profiles which has to access to CPython internals without
calling functions. This API is now installed by ‘make
install’.
(Contributed by Victor Stinner in bpo-35134(2) and bpo-35081(3),
work initiated by Eric Snow in Python 3.7.)
* Some macros have been converted to static inline functions:
parameter types and return type are well defined, they don’t have
issues specific to macros, variables have a local scopes.
Examples:
* *note Py_INCREF(): 56b, *note Py_DECREF(): 56c.
* *note Py_XINCREF(): a64, *note Py_XDECREF(): 78a.
* ‘PyObject_INIT()’, ‘PyObject_INIT_VAR()’
* Private functions: ‘_PyObject_GC_TRACK()’,
‘_PyObject_GC_UNTRACK()’, ‘_Py_Dealloc()’
(Contributed by Victor Stinner in bpo-35059(4).)
* The ‘PyByteArray_Init()’ and ‘PyByteArray_Fini()’ functions have
been removed. They did nothing since Python 2.7.4 and Python
3.2.0, were excluded from the limited API (stable ABI), and were
not documented. (Contributed by Victor Stinner in bpo-35713(5).)
* The result of *note PyExceptionClass_Name(): a65. is now of type
‘const char *’ rather of ‘char *’. (Contributed by Serhiy
Storchaka in bpo-33818(6).)
* The duality of ‘Modules/Setup.dist’ and ‘Modules/Setup’ has been
removed. Previously, when updating the CPython source tree, one
had to manually copy ‘Modules/Setup.dist’ (inside the source tree)
to ‘Modules/Setup’ (inside the build tree) in order to reflect any
changes upstream. This was of a small benefit to packagers at the
expense of a frequent annoyance to developers following CPython
development, as forgetting to copy the file could produce build
failures.
Now the build system always reads from ‘Modules/Setup’ inside the
source tree. People who want to customize that file are encouraged
to maintain their changes in a git fork of CPython or as patch
files, as they would do for any other change to the source tree.
(Contributed by Antoine Pitrou in bpo-32430(7).)
* Functions that convert Python number to C integer like *note
PyLong_AsLong(): 35b. and argument parsing functions like *note
PyArg_ParseTuple(): 56e. with integer converting format units like
‘'i'’ will now use the *note __index__(): 718. special method
instead of *note __int__(): 717, if available. The deprecation
warning will be emitted for objects with the ‘__int__()’ method but
without the ‘__index__()’ method (like *note Decimal: 29f. and
*note Fraction: 1e9.). *note PyNumber_Check(): a66. will now
return ‘1’ for objects implementing ‘__index__()’. *note
PyNumber_Long(): a67, *note PyNumber_Float(): a68. and *note
PyFloat_AsDouble(): a69. also now use the ‘__index__()’ method if
available. (Contributed by Serhiy Storchaka in bpo-36048(8) and
bpo-20092(9).)
* Heap-allocated type objects will now increase their reference count
in *note PyObject_Init(): a6a. (and its parallel macro
‘PyObject_INIT’) instead of in *note PyType_GenericAlloc(): a6b.
Types that modify instance allocation or deallocation may need to
be adjusted. (Contributed by Eddie Elizondo in bpo-35810(10).)
* The new function ‘PyCode_NewWithPosOnlyArgs()’ allows to create
code objects like ‘PyCode_New()’, but with an extra
'posonlyargcount' parameter for indicating the number of
positional-only arguments. (Contributed by Pablo Galindo in
bpo-37221(11).)
* ‘Py_SetPath()’ now sets *note sys.executable: 3b4. to the program
full path (*note Py_GetProgramFullPath(): 3b3.) rather than to the
program name (*note Py_GetProgramName(): 3b5.). (Contributed by
Victor Stinner in bpo-38234(12).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36707
(2) https://bugs.python.org/issue?@action=redirect&bpo=35134
(3) https://bugs.python.org/issue?@action=redirect&bpo=35081
(4) https://bugs.python.org/issue?@action=redirect&bpo=35059
(5) https://bugs.python.org/issue?@action=redirect&bpo=35713
(6) https://bugs.python.org/issue?@action=redirect&bpo=33818
(7) https://bugs.python.org/issue?@action=redirect&bpo=32430
(8) https://bugs.python.org/issue?@action=redirect&bpo=36048
(9) https://bugs.python.org/issue?@action=redirect&bpo=20092
(10) https://bugs.python.org/issue?@action=redirect&bpo=35810
(11) https://bugs.python.org/issue?@action=redirect&bpo=37221
(12) https://bugs.python.org/issue?@action=redirect&bpo=38234
File: python.info, Node: Deprecated<8>, Next: API and Feature Removals, Prev: Build and C API Changes, Up: What’s New In Python 3 8
1.6.8 Deprecated
----------------
* The distutils ‘bdist_wininst’ command is now deprecated, use
‘bdist_wheel’ (wheel packages) instead. (Contributed by Victor
Stinner in bpo-37481(1).)
* Deprecated methods ‘getchildren()’ and ‘getiterator()’ in the *note
ElementTree: 125. module now emit a *note DeprecationWarning: 1a5.
instead of *note PendingDeprecationWarning: 8c7. They will be
removed in Python 3.9. (Contributed by Serhiy Storchaka in
bpo-29209(2).)
* Passing an object that is not an instance of *note
concurrent.futures.ThreadPoolExecutor: 73d. to *note
loop.set_default_executor(): 73e. is deprecated and will be
prohibited in Python 3.9. (Contributed by Elvis Pranskevichus in
bpo-34075(3).)
* The *note __getitem__(): 285. methods of *note
xml.dom.pulldom.DOMEventStream: 730, *note
wsgiref.util.FileWrapper: 731. and *note fileinput.FileInput: 732.
have been deprecated.
Implementations of these methods have been ignoring their 'index'
parameter, and returning the next item instead. (Contributed by
Berker Peksag in bpo-9372(4).)
* The *note typing.NamedTuple: 2b2. class has deprecated the
‘_field_types’ attribute in favor of the ‘__annotations__’
attribute which has the same information. (Contributed by Raymond
Hettinger in bpo-36320(5).)
* *note ast: 8. classes ‘Num’, ‘Str’, ‘Bytes’, ‘NameConstant’ and
‘Ellipsis’ are considered deprecated and will be removed in future
Python versions. *note Constant: 2bb. should be used instead.
(Contributed by Serhiy Storchaka in bpo-32892(6).)
* *note ast.NodeVisitor: a6d. methods ‘visit_Num()’, ‘visit_Str()’,
‘visit_Bytes()’, ‘visit_NameConstant()’ and ‘visit_Ellipsis()’ are
deprecated now and will not be called in future Python versions.
Add the *note visit_Constant(): a6e. method to handle all constant
nodes. (Contributed by Serhiy Storchaka in bpo-36917(7).)
* The ‘asyncio.coroutine()’ *note decorator: 72c. is deprecated and
will be removed in version 3.10. Instead of ‘@asyncio.coroutine’,
use *note async def: 5cd. instead. (Contributed by Andrew Svetlov
in bpo-36921(8).)
* In *note asyncio: a, the explicit passing of a 'loop' argument has
been deprecated and will be removed in version 3.10 for the
following: *note asyncio.sleep(): a6f, *note asyncio.gather(): 5f9,
*note asyncio.shield(): a70, *note asyncio.wait_for(): 5fb, *note
asyncio.wait(): 47b, *note asyncio.as_completed(): 1aa, *note
asyncio.Task: 1be, *note asyncio.Lock: a71, *note asyncio.Event:
a72, *note asyncio.Condition: a73, *note asyncio.Semaphore: a74,
*note asyncio.BoundedSemaphore: a75, *note asyncio.Queue: a76,
*note asyncio.create_subprocess_exec(): a77, and *note
asyncio.create_subprocess_shell(): a78.
* The explicit passing of coroutine objects to *note asyncio.wait():
47b. has been deprecated and will be removed in version 3.11.
(Contributed by Yury Selivanov in bpo-34790(9).)
* The following functions and methods are deprecated in the *note
gettext: 63. module: ‘lgettext()’, ‘ldgettext()’, ‘lngettext()’ and
‘ldngettext()’. They return encoded bytes, and it’s possible that
you will get unexpected Unicode-related exceptions if there are
encoding problems with the translated strings. It’s much better to
use alternatives which return Unicode strings in Python 3. These
functions have been broken for a long time.
Function ‘bind_textdomain_codeset()’, methods ‘output_charset()’
and ‘set_output_charset()’, and the 'codeset' parameter of
functions *note translation(): a79. and *note install(): a7a. are
also deprecated, since they are only used for the ‘l*gettext()’
functions. (Contributed by Serhiy Storchaka in bpo-33710(10).)
* The ‘isAlive()’ method of *note threading.Thread: 94c. has been
deprecated. (Contributed by Donghee Na in bpo-35283(11).)
* Many builtin and extension functions that take integer arguments
will now emit a deprecation warning for *note Decimal: 29f.s, *note
Fraction: 1e9.s and any other objects that can be converted to
integers only with a loss (e.g. that have the *note __int__():
717. method but do not have the *note __index__(): 718. method).
In future version they will be errors. (Contributed by Serhiy
Storchaka in bpo-36048(12).)
* Deprecated passing the following arguments as keyword arguments:
- 'func' in *note functools.partialmethod(): a7b, *note
weakref.finalize(): a7c, *note profile.Profile.runcall(): a7d,
‘cProfile.Profile.runcall()’, *note bdb.Bdb.runcall(): a7e,
*note trace.Trace.runfunc(): a7f. and *note curses.wrapper():
a80.
- 'function' in *note unittest.TestCase.addCleanup(): a81.
- 'fn' in the *note submit(): a82. method of *note
concurrent.futures.ThreadPoolExecutor: 73d. and *note
concurrent.futures.ProcessPoolExecutor: 8ed.
- 'callback' in *note contextlib.ExitStack.callback(): a83,
‘contextlib.AsyncExitStack.callback()’ and *note
contextlib.AsyncExitStack.push_async_callback(): a84.
- 'c' and 'typeid' in the ‘create()’ method of
‘multiprocessing.managers.Server’ and
‘multiprocessing.managers.SharedMemoryServer’.
- 'obj' in *note weakref.finalize(): a7c.
In future releases of Python, they will be *note positional-only:
a85. (Contributed by Serhiy Storchaka in bpo-36492(13).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37481
(2) https://bugs.python.org/issue?@action=redirect&bpo=29209
(3) https://bugs.python.org/issue?@action=redirect&bpo=34075
(4) https://bugs.python.org/issue?@action=redirect&bpo=9372
(5) https://bugs.python.org/issue?@action=redirect&bpo=36320
(6) https://bugs.python.org/issue?@action=redirect&bpo=32892
(7) https://bugs.python.org/issue?@action=redirect&bpo=36917
(8) https://bugs.python.org/issue?@action=redirect&bpo=36921
(9) https://bugs.python.org/issue?@action=redirect&bpo=34790
(10) https://bugs.python.org/issue?@action=redirect&bpo=33710
(11) https://bugs.python.org/issue?@action=redirect&bpo=35283
(12) https://bugs.python.org/issue?@action=redirect&bpo=36048
(13) https://bugs.python.org/issue?@action=redirect&bpo=36492
File: python.info, Node: API and Feature Removals, Next: Porting to Python 3 8, Prev: Deprecated<8>, Up: What’s New In Python 3 8
1.6.9 API and Feature Removals
------------------------------
The following features and APIs have been removed from Python 3.8:
* Starting with Python 3.3, importing ABCs from *note collections:
1d. was deprecated, and importing should be done from *note
collections.abc: 1e. Being able to import from collections was
marked for removal in 3.8, but has been delayed to 3.9. (See
gh-81134(1).)
* The ‘macpath’ module, deprecated in Python 3.7, has been removed.
(Contributed by Victor Stinner in bpo-35471(2).)
* The function ‘platform.popen()’ has been removed, after having been
deprecated since Python 3.3: use *note os.popen(): a87. instead.
(Contributed by Victor Stinner in bpo-35345(3).)
* The function ‘time.clock()’ has been removed, after having been
deprecated since Python 3.3: use *note time.perf_counter(): a88. or
*note time.process_time(): a89. instead, depending on your
requirements, to have well-defined behavior. (Contributed by
Matthias Bussonnier in bpo-36895(4).)
* The ‘pyvenv’ script has been removed in favor of ‘python3.8 -m
venv’ to help eliminate confusion as to what Python interpreter the
‘pyvenv’ script is tied to. (Contributed by Brett Cannon in
bpo-25427(5).)
* ‘parse_qs’, ‘parse_qsl’, and ‘escape’ are removed from the ‘cgi’
module. They are deprecated in Python 3.2 or older. They should
be imported from the ‘urllib.parse’ and ‘html’ modules instead.
* ‘filemode’ function is removed from the *note tarfile: de. module.
It is not documented and deprecated since Python 3.3.
* The *note XMLParser: a53. constructor no longer accepts the 'html'
argument. It never had an effect and was deprecated in Python 3.4.
All other parameters are now *note keyword-only: 2a7. (Contributed
by Serhiy Storchaka in bpo-29209(6).)
* Removed the ‘doctype()’ method of *note XMLParser: a53.
(Contributed by Serhiy Storchaka in bpo-29209(7).)
* “unicode_internal” codec is removed. (Contributed by Inada Naoki
in bpo-36297(8).)
* The ‘Cache’ and ‘Statement’ objects of the *note sqlite3: cf.
module are not exposed to the user. (Contributed by Aviv Palivoda
in bpo-30262(9).)
* The ‘bufsize’ keyword argument of *note fileinput.input(): 7fb. and
*note fileinput.FileInput(): 732. which was ignored and deprecated
since Python 3.6 has been removed. bpo-36952(10) (Contributed by
Matthias Bussonnier.)
* The functions ‘sys.set_coroutine_wrapper()’ and
‘sys.get_coroutine_wrapper()’ deprecated in Python 3.7 have been
removed; bpo-36933(11) (Contributed by Matthias Bussonnier.)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/81134
(2) https://bugs.python.org/issue?@action=redirect&bpo=35471
(3) https://bugs.python.org/issue?@action=redirect&bpo=35345
(4) https://bugs.python.org/issue?@action=redirect&bpo=36895
(5) https://bugs.python.org/issue?@action=redirect&bpo=25427
(6) https://bugs.python.org/issue?@action=redirect&bpo=29209
(7) https://bugs.python.org/issue?@action=redirect&bpo=29209
(8) https://bugs.python.org/issue?@action=redirect&bpo=36297
(9) https://bugs.python.org/issue?@action=redirect&bpo=30262
(10) https://bugs.python.org/issue?@action=redirect&bpo=36952
(11) https://bugs.python.org/issue?@action=redirect&bpo=36933
File: python.info, Node: Porting to Python 3 8, Next: Notable changes in Python 3 8 1, Prev: API and Feature Removals, Up: What’s New In Python 3 8
1.6.10 Porting to Python 3.8
----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in Python behavior::
* Changes in the Python API: Changes in the Python API<5>.
* Changes in the C API: Changes in the C API<4>.
* CPython bytecode changes: CPython bytecode changes<5>.
* Demos and Tools: Demos and Tools<2>.
File: python.info, Node: Changes in Python behavior, Next: Changes in the Python API<5>, Up: Porting to Python 3 8
1.6.10.1 Changes in Python behavior
...................................
* Yield expressions (both ‘yield’ and ‘yield from’ clauses) are now
disallowed in comprehensions and generator expressions (aside from
the iterable expression in the leftmost ‘for’ clause).
(Contributed by Serhiy Storchaka in bpo-10544(1).)
* The compiler now produces a *note SyntaxWarning: 461. when identity
checks (‘is’ and ‘is not’) are used with certain types of literals
(e.g. strings, numbers). These can often work by accident in
CPython, but are not guaranteed by the language spec. The warning
advises users to use equality tests (‘==’ and ‘!=’) instead.
(Contributed by Serhiy Storchaka in bpo-34850(2).)
* The CPython interpreter can swallow exceptions in some
circumstances. In Python 3.8 this happens in fewer cases. In
particular, exceptions raised when getting the attribute from the
type dictionary are no longer ignored. (Contributed by Serhiy
Storchaka in bpo-35459(3).)
* Removed ‘__str__’ implementations from builtin types *note bool:
463, *note int: 259, *note float: 2f1, *note complex: 2f2. and few
classes from the standard library. They now inherit ‘__str__()’
from *note object: a8c. As result, defining the ‘__repr__()’
method in the subclass of these classes will affect their string
representation. (Contributed by Serhiy Storchaka in bpo-36793(4).)
* On AIX, *note sys.platform: a8d. doesn’t contain the major version
anymore. It is always ‘'aix'’, instead of ‘'aix3'’ .. ‘'aix7'’.
Since older Python versions include the version number, so it is
recommended to always use ‘sys.platform.startswith('aix')’.
(Contributed by M. Felt in bpo-36588(5).)
* ‘PyEval_AcquireLock()’ and ‘PyEval_AcquireThread()’ now terminate
the current thread if called while the interpreter is finalizing,
making them consistent with *note PyEval_RestoreThread(): 3a5,
*note Py_END_ALLOW_THREADS(): a8e, and *note PyGILState_Ensure():
3a7. If this behavior is not desired, guard the call by checking
‘_Py_IsFinalizing()’ or *note sys.is_finalizing(): a8f.
(Contributed by Joannah Nanjekye in bpo-36475(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10544
(2) https://bugs.python.org/issue?@action=redirect&bpo=34850
(3) https://bugs.python.org/issue?@action=redirect&bpo=35459
(4) https://bugs.python.org/issue?@action=redirect&bpo=36793
(5) https://bugs.python.org/issue?@action=redirect&bpo=36588
(6) https://bugs.python.org/issue?@action=redirect&bpo=36475
File: python.info, Node: Changes in the Python API<5>, Next: Changes in the C API<4>, Prev: Changes in Python behavior, Up: Porting to Python 3 8
1.6.10.2 Changes in the Python API
..................................
* The *note os.getcwdb(): a91. function now uses the UTF-8 encoding
on Windows, rather than the ANSI code page: see PEP 529(1) for the
rationale. The function is no longer deprecated on Windows.
(Contributed by Victor Stinner in bpo-37412(2).)
* *note subprocess.Popen: 199. can now use *note os.posix_spawn():
222. in some cases for better performance. On Windows Subsystem
for Linux and QEMU User Emulation, the ‘Popen’ constructor using
*note os.posix_spawn(): 222. no longer raises an exception on
errors like “missing program”. Instead the child process fails
with a non-zero ‘returncode’. (Contributed by Joannah Nanjekye and
Victor Stinner in bpo-35537(3).)
* The 'preexec_fn' argument of * *note subprocess.Popen: 199. is no
longer compatible with subinterpreters. The use of the parameter
in a subinterpreter now raises *note RuntimeError: 195.
(Contributed by Eric Snow in bpo-34651(4), modified by Christian
Heimes in bpo-37951(5).)
* The ‘imap.IMAP4.logout()’ method no longer silently ignores
arbitrary exceptions. (Contributed by Victor Stinner in
bpo-36348(6).)
* The function ‘platform.popen()’ has been removed, after having been
deprecated since Python 3.3: use *note os.popen(): a87. instead.
(Contributed by Victor Stinner in bpo-35345(7).)
* The *note statistics.mode(): a92. function no longer raises an
exception when given multimodal data. Instead, it returns the
first mode encountered in the input data. (Contributed by Raymond
Hettinger in bpo-35892(8).)
* The *note selection(): a93. method of the *note
tkinter.ttk.Treeview: a94. class no longer takes arguments. Using
it with arguments for changing the selection was deprecated in
Python 3.6. Use specialized methods like *note selection_set():
a95. for changing the selection. (Contributed by Serhiy Storchaka
in bpo-31508(9).)
* The ‘writexml()’, ‘toxml()’ and ‘toprettyxml()’ methods of *note
xml.dom.minidom: 122, and the ‘write()’ method of ‘xml.etree’, now
preserve the attribute order specified by the user. (Contributed
by Diego Rojas and Raymond Hettinger in bpo-34160(10).)
* A *note dbm.dumb: 32. database opened with flags ‘'r'’ is now
read-only. *note dbm.dumb.open(): a96. with flags ‘'r'’ and ‘'w'’
no longer creates a database if it does not exist. (Contributed by
Serhiy Storchaka in bpo-32749(11).)
* The ‘doctype()’ method defined in a subclass of *note XMLParser:
a53. will no longer be called and will emit a *note RuntimeWarning:
1bd. instead of a *note DeprecationWarning: 1a5. Define the *note
doctype(): a97. method on a target for handling an XML doctype
declaration. (Contributed by Serhiy Storchaka in bpo-29209(12).)
* A *note RuntimeError: 195. is now raised when the custom metaclass
doesn’t provide the ‘__classcell__’ entry in the namespace passed
to ‘type.__new__’. A *note DeprecationWarning: 1a5. was emitted in
Python 3.6–3.7. (Contributed by Serhiy Storchaka in
bpo-23722(13).)
* The ‘cProfile.Profile’ class can now be used as a context manager.
(Contributed by Scott Sanderson in bpo-29235(14).)
* *note shutil.copyfile(): a58, *note shutil.copy(): a59, *note
shutil.copy2(): a5a, *note shutil.copytree(): a28. and *note
shutil.move(): a5b. use platform-specific “fast-copy” syscalls (see
*note Platform-dependent efficient copy operations: a5d. section).
* *note shutil.copyfile(): a58. default buffer size on Windows was
changed from 16 KiB to 1 MiB.
* The ‘PyGC_Head’ struct has changed completely. All code that
touched the struct member should be rewritten. (See
bpo-33597(15).)
* The *note PyInterpreterState: 8bc. struct has been moved into the
“internal” header files (specifically
Include/internal/pycore_pystate.h). An opaque ‘PyInterpreterState’
is still available as part of the public API (and stable ABI). The
docs indicate that none of the struct’s fields are public, so we
hope no one has been using them. However, if you do rely on one or
more of those private fields and have no alternative then please
open a BPO issue. We’ll work on helping you adjust (possibly
including adding accessor functions to the public API). (See
bpo-35886(16).)
* The *note mmap.flush(): a98. method now returns ‘None’ on success
and raises an exception on error under all platforms. Previously,
its behavior was platform-dependent: a nonzero value was returned
on success; zero was returned on error under Windows. A zero value
was returned on success; an exception was raised on error under
Unix. (Contributed by Berker Peksag in bpo-2122(17).)
* *note xml.dom.minidom: 122. and *note xml.sax: 129. modules no
longer process external entities by default. (Contributed by
Christian Heimes in bpo-17239(18).)
* Deleting a key from a read-only *note dbm: 31. database (*note
dbm.dumb: 32, *note dbm.gnu: 33. or *note dbm.ndbm: 34.) raises
‘error’ (*note dbm.dumb.error: a99, *note dbm.gnu.error: a9a. or
*note dbm.ndbm.error: a9b.) instead of *note KeyError: 33f.
(Contributed by Xiang Zhang in bpo-33106(19).)
* Simplified AST for literals. All constants will be represented as
*note ast.Constant: 2bb. instances. Instantiating old classes
‘Num’, ‘Str’, ‘Bytes’, ‘NameConstant’ and ‘Ellipsis’ will return an
instance of ‘Constant’. (Contributed by Serhiy Storchaka in
bpo-32892(20).)
* *note expanduser(): a13. on Windows now prefers the ‘USERPROFILE’
environment variable and does not use ‘HOME’, which is not normally
set for regular user accounts. (Contributed by Anthony Sottile in
bpo-36264(21).)
* The exception *note asyncio.CancelledError: 1b8. now inherits from
*note BaseException: 5b7. rather than *note Exception: 9d9. and no
longer inherits from *note concurrent.futures.CancelledError: 9da.
(Contributed by Yury Selivanov in bpo-32528(22).)
* The function *note asyncio.wait_for(): 5fb. now correctly waits for
cancellation when using an instance of *note asyncio.Task: 1be.
Previously, upon reaching 'timeout', it was cancelled and
immediately returned. (Contributed by Elvis Pranskevichus in
bpo-32751(23).)
* The function *note asyncio.BaseTransport.get_extra_info(): a9c. now
returns a safe to use socket object when ‘socket’ is passed to the
'name' parameter. (Contributed by Yury Selivanov in
bpo-37027(24).)
* *note asyncio.BufferedProtocol: a9d. has graduated to the stable
API.
* DLL dependencies for extension modules and DLLs loaded with *note
ctypes: 2a. on Windows are now resolved more securely. Only the
system paths, the directory containing the DLL or PYD file, and
directories added with *note add_dll_directory(): 9e9. are searched
for load-time dependencies. Specifically, ‘PATH’ and the current
working directory are no longer used, and modifications to these
will no longer have any effect on normal DLL resolution. If your
application relies on these mechanisms, you should check for *note
add_dll_directory(): 9e9. and if it exists, use it to add your DLLs
directory while loading your library. Note that Windows 7 users
will need to ensure that Windows Update KB2533623 has been
installed (this is also verified by the installer). (Contributed
by Steve Dower in bpo-36085(25).)
* The header files and functions related to pgen have been removed
after its replacement by a pure Python implementation.
(Contributed by Pablo Galindo in bpo-36623(26).)
* *note types.CodeType: 2e3. has a new parameter in the second
position of the constructor ('posonlyargcount') to support
positional-only arguments defined in PEP 570(27). The first
argument ('argcount') now represents the total number of positional
arguments (including positional-only arguments). The new
‘replace()’ method of *note types.CodeType: 2e3. can be used to
make the code future-proof.
* The parameter ‘digestmod’ for *note hmac.new(): a9f. no longer uses
the MD5 digest by default.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0529/
(2) https://bugs.python.org/issue?@action=redirect&bpo=37412
(3) https://bugs.python.org/issue?@action=redirect&bpo=35537
(4) https://bugs.python.org/issue?@action=redirect&bpo=34651
(5) https://bugs.python.org/issue?@action=redirect&bpo=37951
(6) https://bugs.python.org/issue?@action=redirect&bpo=36348
(7) https://bugs.python.org/issue?@action=redirect&bpo=35345
(8) https://bugs.python.org/issue?@action=redirect&bpo=35892
(9) https://bugs.python.org/issue?@action=redirect&bpo=31508
(10) https://bugs.python.org/issue?@action=redirect&bpo=34160
(11) https://bugs.python.org/issue?@action=redirect&bpo=32749
(12) https://bugs.python.org/issue?@action=redirect&bpo=29209
(13) https://bugs.python.org/issue?@action=redirect&bpo=23722
(14) https://bugs.python.org/issue?@action=redirect&bpo=29235
(15) https://bugs.python.org/issue?@action=redirect&bpo=33597
(16) https://bugs.python.org/issue?@action=redirect&bpo=35886
(17) https://bugs.python.org/issue?@action=redirect&bpo=2122
(18) https://bugs.python.org/issue?@action=redirect&bpo=17239
(19) https://bugs.python.org/issue?@action=redirect&bpo=33106
(20) https://bugs.python.org/issue?@action=redirect&bpo=32892
(21) https://bugs.python.org/issue?@action=redirect&bpo=36264
(22) https://bugs.python.org/issue?@action=redirect&bpo=32528
(23) https://bugs.python.org/issue?@action=redirect&bpo=32751
(24) https://bugs.python.org/issue?@action=redirect&bpo=37027
(25) https://bugs.python.org/issue?@action=redirect&bpo=36085
(26) https://bugs.python.org/issue?@action=redirect&bpo=36623
(27) https://peps.python.org/pep-0570/
File: python.info, Node: Changes in the C API<4>, Next: CPython bytecode changes<5>, Prev: Changes in the Python API<5>, Up: Porting to Python 3 8
1.6.10.3 Changes in the C API
.............................
* The *note PyCompilerFlags: aa1. structure got a new
'cf_feature_version' field. It should be initialized to
‘PY_MINOR_VERSION’. The field is ignored by default, and is used
if and only if ‘PyCF_ONLY_AST’ flag is set in 'cf_flags'.
(Contributed by Guido van Rossum in bpo-35766(1).)
* The ‘PyEval_ReInitThreads()’ function has been removed from the C
API. It should not be called explicitly: use *note
PyOS_AfterFork_Child(): 3f7. instead. (Contributed by Victor
Stinner in bpo-36728(2).)
* On Unix, C extensions are no longer linked to libpython except on
Android and Cygwin. When Python is embedded, ‘libpython’ must not
be loaded with ‘RTLD_LOCAL’, but ‘RTLD_GLOBAL’ instead.
Previously, using ‘RTLD_LOCAL’, it was already not possible to load
C extensions which were not linked to ‘libpython’, like C
extensions of the standard library built by the ‘*shared*’ section
of ‘Modules/Setup’. (Contributed by Victor Stinner in
bpo-21536(3).)
* Use of ‘#’ variants of formats in parsing or building value (e.g.
*note PyArg_ParseTuple(): 56e, *note Py_BuildValue(): 8a6, *note
PyObject_CallFunction(): 39a, etc.) without ‘PY_SSIZE_T_CLEAN’
defined raises ‘DeprecationWarning’ now. It will be removed in
3.10 or 4.0. Read *note Parsing arguments and building values:
8a7. for detail. (Contributed by Inada Naoki in bpo-36381(4).)
* Instances of heap-allocated types (such as those created with *note
PyType_FromSpec(): 57a.) hold a reference to their type object.
Increasing the reference count of these type objects has been moved
from *note PyType_GenericAlloc(): a6b. to the more low-level
functions, *note PyObject_Init(): a6a. and ‘PyObject_INIT()’. This
makes types created through *note PyType_FromSpec(): 57a. behave
like other classes in managed code.
*note Statically allocated types: 77a. are not affected.
For the vast majority of cases, there should be no side effect.
However, types that manually increase the reference count after
allocating an instance (perhaps to work around the bug) may now
become immortal. To avoid this, these classes need to call
Py_DECREF on the type object during instance deallocation.
To correctly port these types into 3.8, please apply the following
changes:
* Remove *note Py_INCREF: 56b. on the type object after
allocating an instance - if any. This may happen after
calling *note PyObject_New: 985, *note PyObject_NewVar: 986,
*note PyObject_GC_New(): aa2, *note PyObject_GC_NewVar(): aa3,
or any other custom allocator that uses *note PyObject_Init():
a6a. or ‘PyObject_INIT()’.
Example:
static foo_struct *
foo_new(PyObject *type) {
foo_struct *foo = PyObject_GC_New(foo_struct, (PyTypeObject *) type);
if (foo == NULL)
return NULL;
#if PY_VERSION_HEX < 0x03080000
// Workaround for Python issue 35810; no longer necessary in Python 3.8
PY_INCREF(type)
#endif
return foo;
}
* Ensure that all custom ‘tp_dealloc’ functions of
heap-allocated types decrease the type’s reference count.
Example:
static void
foo_dealloc(foo_struct *instance) {
PyObject *type = Py_TYPE(instance);
PyObject_GC_Del(instance);
#if PY_VERSION_HEX >= 0x03080000
// This was not needed before Python 3.8 (Python issue 35810)
Py_DECREF(type);
#endif
}
(Contributed by Eddie Elizondo in bpo-35810(5).)
* The *note Py_DEPRECATED(): aa4. macro has been implemented for
MSVC. The macro now must be placed before the symbol name.
Example:
Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void);
(Contributed by Zackery Spytz in bpo-33407(6).)
* The interpreter does not pretend to support binary compatibility of
extension types across feature releases, anymore. A *note
PyTypeObject: aa5. exported by a third-party extension module is
supposed to have all the slots expected in the current Python
version, including *note tp_finalize: aa6. (*note
Py_TPFLAGS_HAVE_FINALIZE: 3ee. is not checked anymore before
reading *note tp_finalize: aa6.).
(Contributed by Antoine Pitrou in bpo-32388(7).)
* The functions ‘PyNode_AddChild()’ and ‘PyParser_AddToken()’ now
accept two additional ‘int’ arguments 'end_lineno' and
'end_col_offset'.
* The ‘libpython38.a’ file to allow MinGW tools to link directly
against ‘python38.dll’ is no longer included in the regular Windows
distribution. If you require this file, it may be generated with
the ‘gendef’ and ‘dlltool’ tools, which are part of the MinGW
binutils package:
gendef - python38.dll > tmp.def
dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a
The location of an installed ‘pythonXY.dll’ will depend on the
installation options and the version and language of Windows. See
*note Using Python on Windows: aa7. for more information. The
resulting library should be placed in the same directory as
‘pythonXY.lib’, which is generally the ‘libs’ directory under your
Python installation.
(Contributed by Steve Dower in bpo-37351(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35766
(2) https://bugs.python.org/issue?@action=redirect&bpo=36728
(3) https://bugs.python.org/issue?@action=redirect&bpo=21536
(4) https://bugs.python.org/issue?@action=redirect&bpo=36381
(5) https://bugs.python.org/issue?@action=redirect&bpo=35810
(6) https://bugs.python.org/issue?@action=redirect&bpo=33407
(7) https://bugs.python.org/issue?@action=redirect&bpo=32388
(8) https://bugs.python.org/issue?@action=redirect&bpo=37351
File: python.info, Node: CPython bytecode changes<5>, Next: Demos and Tools<2>, Prev: Changes in the C API<4>, Up: Porting to Python 3 8
1.6.10.4 CPython bytecode changes
.................................
* The interpreter loop has been simplified by moving the logic of
unrolling the stack of blocks into the compiler. The compiler
emits now explicit instructions for adjusting the stack of values
and calling the cleaning-up code for *note break: aa9, *note
continue: 9c9. and *note return: 9ce.
Removed opcodes ‘BREAK_LOOP’, ‘CONTINUE_LOOP’, ‘SETUP_LOOP’ and
‘SETUP_EXCEPT’. Added new opcodes ‘ROT_FOUR’, ‘BEGIN_FINALLY’,
‘CALL_FINALLY’ and ‘POP_FINALLY’. Changed the behavior of
‘END_FINALLY’ and ‘WITH_CLEANUP_START’.
(Contributed by Mark Shannon, Antoine Pitrou and Serhiy Storchaka
in bpo-17611(1).)
* Added new opcode *note END_ASYNC_FOR: aaa. for handling exceptions
raised when awaiting a next item in an *note async for: aab. loop.
(Contributed by Serhiy Storchaka in bpo-33041(2).)
* The *note MAP_ADD: aac. now expects the value as the first element
in the stack and the key as the second element. This change was
made so the key is always evaluated before the value in dictionary
comprehensions, as proposed by PEP 572(3). (Contributed by Jörn
Heissler in bpo-35224(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17611
(2) https://bugs.python.org/issue?@action=redirect&bpo=33041
(3) https://peps.python.org/pep-0572/
(4) https://bugs.python.org/issue?@action=redirect&bpo=35224
File: python.info, Node: Demos and Tools<2>, Prev: CPython bytecode changes<5>, Up: Porting to Python 3 8
1.6.10.5 Demos and Tools
........................
Added a benchmark script for timing various ways to access variables:
‘Tools/scripts/var_access_benchmark.py’. (Contributed by Raymond
Hettinger in bpo-35884(1).)
Here’s a summary of performance improvements since Python 3.3:
Python version 3.3 3.4 3.5 3.6 3.7 3.8
-------------- --- --- --- --- --- ---
Variable and attribute read access:
read_local 4.0 7.1 7.1 5.4 5.1 3.9
read_nonlocal 5.3 7.1 8.1 5.8 5.4 4.4
read_global 13.3 15.5 19.0 14.3 13.6 7.6
read_builtin 20.0 21.1 21.6 18.5 19.0 7.5
read_classvar_from_class 20.5 25.6 26.5 20.7 19.5 18.4
read_classvar_from_instance 18.5 22.8 23.5 18.8 17.1 16.4
read_instancevar 26.8 32.4 33.1 28.0 26.3 25.4
read_instancevar_slots 23.7 27.8 31.3 20.8 20.8 20.2
read_namedtuple 68.5 73.8 57.5 45.0 46.8 18.4
read_boundmethod 29.8 37.6 37.9 29.6 26.9 27.7
Variable and attribute write access:
write_local 4.6 8.7 9.3 5.5 5.3 4.3
write_nonlocal 7.3 10.5 11.1 5.6 5.5 4.7
write_global 15.9 19.7 21.2 18.0 18.0 15.8
write_classvar 81.9 92.9 96.0 104.6 102.1 39.2
write_instancevar 36.4 44.6 45.8 40.0 38.9 35.5
write_instancevar_slots 28.7 35.6 36.1 27.3 26.6 25.7
Data structure read access:
read_list 19.2 24.2 24.5 20.8 20.8 19.0
read_deque 19.9 24.7 25.5 20.2 20.6 19.8
read_dict 19.7 24.3 25.7 22.3 23.0 21.0
read_strdict 17.9 22.6 24.3 19.5 21.2 18.9
Data structure write access:
write_list 21.2 27.1 28.5 22.5 21.6 20.0
write_deque 23.8 28.7 30.1 22.7 21.8 23.5
write_dict 25.9 31.4 33.3 29.3 29.2 24.7
write_strdict 22.9 28.4 29.9 27.5 25.2 23.1
Stack (or queue) operations:
list_append_pop 144.2 93.4 112.7 75.4 74.2 50.8
deque_append_pop 30.4 43.5 57.0 49.4 49.2 42.5
deque_append_popleft 30.8 43.7 57.3 49.7 49.7 42.8
Timing loop:
loop_overhead 0.3 0.5 0.6 0.4 0.3 0.3
The benchmarks were measured on an Intel® Core™ i7-4960HQ processor(2)
running the macOS 64-bit builds found at python.org(3). The benchmark
script displays timings in nanoseconds.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=35884
(2)
https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html
(3) https://www.python.org/downloads/macos/
File: python.info, Node: Notable changes in Python 3 8 1, Next: Notable changes in Python 3 8 2, Prev: Porting to Python 3 8, Up: What’s New In Python 3 8
1.6.11 Notable changes in Python 3.8.1
--------------------------------------
Due to significant security concerns, the 'reuse_address' parameter of
*note asyncio.loop.create_datagram_endpoint(): 72e. is no longer
supported. This is because of the behavior of the socket option
‘SO_REUSEADDR’ in UDP. For more details, see the documentation for
‘loop.create_datagram_endpoint()’. (Contributed by Kyle Stanley,
Antoine Pitrou, and Yury Selivanov in bpo-37228(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37228
File: python.info, Node: Notable changes in Python 3 8 2, Next: Notable changes in Python 3 8 3, Prev: Notable changes in Python 3 8 1, Up: What’s New In Python 3 8
1.6.12 Notable changes in Python 3.8.2
--------------------------------------
Fixed a regression with the ‘ignore’ callback of *note
shutil.copytree(): a28. The argument types are now str and List[str]
again. (Contributed by Manuel Barkhau and Giampaolo Rodola in
gh-83571(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/83571
File: python.info, Node: Notable changes in Python 3 8 3, Next: Notable changes in Python 3 8 8, Prev: Notable changes in Python 3 8 2, Up: What’s New In Python 3 8
1.6.13 Notable changes in Python 3.8.3
--------------------------------------
The constant values of future flags in the *note __future__: 0. module
are updated in order to prevent collision with compiler flags.
Previously ‘PyCF_ALLOW_TOP_LEVEL_AWAIT’ was clashing with
‘CO_FUTURE_DIVISION’. (Contributed by Batuhan Taskaya in gh-83743(1))
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/83743
File: python.info, Node: Notable changes in Python 3 8 8, Next: Notable changes in Python 3 8 9, Prev: Notable changes in Python 3 8 3, Up: What’s New In Python 3 8
1.6.14 Notable changes in Python 3.8.8
--------------------------------------
Earlier Python versions allowed using both ‘;’ and ‘&’ as query
parameter separators in *note urllib.parse.parse_qs(): 27c. and *note
urllib.parse.parse_qsl(): 27b. Due to security concerns, and to conform
with newer W3C recommendations, this has been changed to allow only a
single separator key, with ‘&’ as the default. This change also affects
‘cgi.parse()’ and ‘cgi.parse_multipart()’ as they use the affected
functions internally. For more details, please see their respective
documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and
Ken Jin in bpo-42967(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42967
File: python.info, Node: Notable changes in Python 3 8 9, Next: Notable changes in Python 3 8 10, Prev: Notable changes in Python 3 8 8, Up: What’s New In Python 3 8
1.6.15 Notable changes in Python 3.8.9
--------------------------------------
A security fix alters the *note ftplib.FTP: 8f9. behavior to not trust
the IPv4 address sent from the remote server when setting up a passive
data channel. We reuse the ftp server IP address instead. For unusual
code requiring the old behavior, set a ‘trust_server_pasv_ipv4_address’
attribute on your FTP instance to ‘True’. (See gh-87451(1))
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/87451
File: python.info, Node: Notable changes in Python 3 8 10, Next: Notable changes in Python 3 8 10<2>, Prev: Notable changes in Python 3 8 9, Up: What’s New In Python 3 8
1.6.16 Notable changes in Python 3.8.10
---------------------------------------
* Menu:
* macOS 11.0 (Big Sur) and Apple Silicon Mac support: macOS 11 0 Big Sur and Apple Silicon Mac support<2>.
File: python.info, Node: macOS 11 0 Big Sur and Apple Silicon Mac support<2>, Up: Notable changes in Python 3 8 10
1.6.16.1 macOS 11.0 (Big Sur) and Apple Silicon Mac support
...........................................................
As of 3.8.10, Python now supports building and running on macOS 11 (Big
Sur) and on Apple Silicon Macs (based on the ‘ARM64’ architecture). A
new universal build variant, ‘universal2’, is now available to natively
support both ‘ARM64’ and ‘Intel 64’ in one set of executables. Note
that support for “weaklinking”, building binaries targeted for newer
versions of macOS that will also run correctly on older versions by
testing at runtime for missing features, is not included in this
backport from Python 3.9; to support a range of macOS versions, continue
to target for and build on the oldest version in the range.
(Originally contributed by Ronald Oussoren and Lawrence D’Anna in
gh-85272(1), with fixes by FX Coudert and Eli Rykoff, and backported to
3.8 by Maxime Bélanger and Ned Deily)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/85272
File: python.info, Node: Notable changes in Python 3 8 10<2>, Next: Notable changes in Python 3 8 12, Prev: Notable changes in Python 3 8 10, Up: What’s New In Python 3 8
1.6.17 Notable changes in Python 3.8.10
---------------------------------------
* Menu:
* urllib.parse: urllib parse<4>.
File: python.info, Node: urllib parse<4>, Up: Notable changes in Python 3 8 10<2>
1.6.17.1 urllib.parse
.....................
The presence of newline or tab characters in parts of a URL allows for
some forms of attacks. Following the WHATWG specification that updates
RFC 3986(1), ASCII newline ‘\n’, ‘\r’ and tab ‘\t’ characters are
stripped from the URL by the parser in *note urllib.parse: 10a.
preventing such attacks. The removal characters are controlled by a new
module level variable ‘urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE’. (See
bpo-43882(2))
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc3986.html
(2) https://bugs.python.org/issue?@action=redirect&bpo=43882
File: python.info, Node: Notable changes in Python 3 8 12, Next: Notable security feature in 3 8 14, Prev: Notable changes in Python 3 8 10<2>, Up: What’s New In Python 3 8
1.6.18 Notable changes in Python 3.8.12
---------------------------------------
* Menu:
* Changes in the Python API: Changes in the Python API<6>.
File: python.info, Node: Changes in the Python API<6>, Up: Notable changes in Python 3 8 12
1.6.18.1 Changes in the Python API
..................................
Starting with Python 3.8.12 the *note ipaddress: 80. module no longer
accepts any leading zeros in IPv4 address strings. Leading zeros are
ambiguous and interpreted as octal notation by some libraries. For
example the legacy function *note socket.inet_aton(): 961. treats
leading zeros as octal notation. glibc implementation of modern *note
inet_pton(): 962. does not accept any leading zeros.
(Originally contributed by Christian Heimes in bpo-36384(1), and
backported to 3.8 by Achraf Merzouki.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=36384
File: python.info, Node: Notable security feature in 3 8 14, Next: Notable changes in 3 8 17, Prev: Notable changes in Python 3 8 12, Up: What’s New In Python 3 8
1.6.19 Notable security feature in 3.8.14
-----------------------------------------
Converting between *note int: 259. and *note str: 447. in bases other
than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10
(decimal) now raises a *note ValueError: 204. if the number of digits in
string form is above a limit to avoid potential denial of service
attacks due to the algorithmic complexity. This is a mitigation for CVE
2020-10735(1). This limit can be configured or disabled by environment
variable, command line flag, or *note sys: d9. APIs. See the *note
integer string conversion length limitation: 5f1. documentation. The
default limit is 4300 digits in string form.
---------- Footnotes ----------
(1) https://www.cve.org/CVERecord?id=CVE-2020-10735
File: python.info, Node: Notable changes in 3 8 17, Prev: Notable security feature in 3 8 14, Up: What’s New In Python 3 8
1.6.20 Notable changes in 3.8.17
--------------------------------
* Menu:
* tarfile: tarfile<6>.
File: python.info, Node: tarfile<6>, Up: Notable changes in 3 8 17
1.6.20.1 tarfile
................
* The extraction methods in *note tarfile: de, and *note
shutil.unpack_archive(): 467, have a new a 'filter' argument that
allows limiting tar features than may be surprising or dangerous,
such as creating files outside the destination directory. See
*note Extraction filters: 468. for details. In Python 3.12, use
without the 'filter' argument will show a *note DeprecationWarning:
1a5. In Python 3.14, the default will switch to ‘'data'’.
(Contributed by Petr Viktorin in PEP 706(1).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0706/
File: python.info, Node: What’s New In Python 3 7, Next: What’s New In Python 3 6, Prev: What’s New In Python 3 8, Up: What’s New in Python
1.7 What’s New In Python 3.7
============================
Editor: Elvis Pranskevichus <>
This article explains the new features in Python 3.7, compared to 3.6.
Python 3.7 was released on June 27, 2018. For full details, see the
*note changelog: 13c.
* Menu:
* Summary – Release Highlights: Summary – Release Highlights<2>.
* New Features: New Features<12>.
* Other Language Changes: Other Language Changes<7>.
* New Modules: New Modules<7>.
* Improved Modules: Improved Modules<7>.
* C API Changes: C API Changes<6>.
* Build Changes: Build Changes<6>.
* Optimizations: Optimizations<7>.
* Other CPython Implementation Changes: Other CPython Implementation Changes<2>.
* Deprecated Python Behavior::
* Deprecated Python modules, functions and methods: Deprecated Python modules functions and methods.
* Deprecated functions and types of the C API::
* Platform Support Removals::
* API and Feature Removals: API and Feature Removals<2>.
* Module Removals::
* Windows-only Changes::
* Porting to Python 3.7: Porting to Python 3 7.
* Notable changes in Python 3.7.1: Notable changes in Python 3 7 1.
* Notable changes in Python 3.7.2: Notable changes in Python 3 7 2.
* Notable changes in Python 3.7.6: Notable changes in Python 3 7 6.
* Notable changes in Python 3.7.10: Notable changes in Python 3 7 10.
* Notable changes in Python 3.7.11: Notable changes in Python 3 7 11.
* Notable security feature in 3.7.14: Notable security feature in 3 7 14.
File: python.info, Node: Summary – Release Highlights<2>, Next: New Features<12>, Up: What’s New In Python 3 7
1.7.1 Summary – Release Highlights
----------------------------------
New syntax features:
* *note PEP 563: abf, postponed evaluation of type annotations.
Backwards incompatible syntax changes:
* *note async: 72d. and *note await: 1b9. are now reserved keywords.
New library modules:
* *note contextvars: 24.: *note PEP 567 – Context Variables: ac0.
* *note dataclasses: 2f.: *note PEP 557 – Data Classes: ac1.
* *note importlib.resources: ac2.
New built-in features:
* *note PEP 553: ac3, the new *note breakpoint(): 236. function.
Python data model improvements:
* *note PEP 562: ac4, customization of access to module attributes.
* *note PEP 560: ac5, core support for typing module and generic
types.
* the insertion-order preservation nature of *note dict: ac6. objects
has been declared(1) to be an official part of the Python language
spec.
Significant improvements in the standard library:
* The *note asyncio: a. module has received new features, significant
*note usability and performance improvements: ac7.
* The *note time: ee. module gained support for *note functions with
nanosecond resolution: ac8.
CPython implementation improvements:
* Avoiding the use of ASCII as a default text encoding:
* *note PEP 538: ac9, legacy C locale coercion
* *note PEP 540: aca, forced UTF-8 runtime mode
* *note PEP 552: acb, deterministic .pycs
* *note New Python Development Mode: acc.
* *note PEP 565: acd, improved *note DeprecationWarning: 1a5.
handling
C API improvements:
* *note PEP 539: ace, new C API for thread-local storage
Documentation improvements:
* *note PEP 545: acf, Python documentation translations
* New documentation translations: Japanese(2), French(3), and
Korean(4).
This release features notable performance improvements in many areas.
The *note Optimizations: ad0. section lists them in detail.
For a list of changes that may affect compatibility with previous Python
releases please refer to the *note Porting to Python 3.7: ad1. section.
---------- Footnotes ----------
(1)
https://mail.python.org/pipermail/python-dev/2017-December/151283.html
(2) https://docs.python.org/ja/
(3) https://docs.python.org/fr/
(4) https://docs.python.org/ko/
File: python.info, Node: New Features<12>, Next: Other Language Changes<7>, Prev: Summary – Release Highlights<2>, Up: What’s New In Python 3 7
1.7.2 New Features
------------------
* Menu:
* PEP 563; Postponed Evaluation of Annotations: PEP 563 Postponed Evaluation of Annotations.
* PEP 538; Legacy C Locale Coercion: PEP 538 Legacy C Locale Coercion.
* PEP 540; Forced UTF-8 Runtime Mode: PEP 540 Forced UTF-8 Runtime Mode.
* PEP 553; Built-in breakpoint(): PEP 553 Built-in breakpoint.
* PEP 539; New C API for Thread-Local Storage: PEP 539 New C API for Thread-Local Storage.
* PEP 562; Customization of Access to Module Attributes: PEP 562 Customization of Access to Module Attributes.
* PEP 564; New Time Functions With Nanosecond Resolution: PEP 564 New Time Functions With Nanosecond Resolution.
* PEP 565; Show DeprecationWarning in __main__: PEP 565 Show DeprecationWarning in __main__.
* PEP 560; Core Support for typing module and Generic Types: PEP 560 Core Support for typing module and Generic Types.
* PEP 552; Hash-based .pyc Files: PEP 552 Hash-based pyc Files.
* PEP 545; Python Documentation Translations: PEP 545 Python Documentation Translations.
* Python Development Mode (-X dev): Python Development Mode -X dev.
File: python.info, Node: PEP 563 Postponed Evaluation of Annotations, Next: PEP 538 Legacy C Locale Coercion, Up: New Features<12>
1.7.2.1 PEP 563: Postponed Evaluation of Annotations
....................................................
The advent of type hints in Python uncovered two glaring usability
issues with the functionality of annotations added in PEP 3107(1) and
refined further in PEP 526(2):
* annotations could only use names which were already available in
the current scope, in other words they didn’t support forward
references of any kind; and
* annotating source code had adverse effects on startup time of
Python programs.
Both of these issues are fixed by postponing the evaluation of
annotations. Instead of compiling code which executes expressions in
annotations at their definition time, the compiler stores the annotation
in a string form equivalent to the AST of the expression in question.
If needed, annotations can be resolved at runtime using *note
typing.get_type_hints(): 6ad. In the common case where this is not
required, the annotations are cheaper to store (since short strings are
interned by the interpreter) and make startup time faster.
Usability-wise, annotations now support forward references, making the
following syntax valid:
class C:
@classmethod
def from_string(cls, source: str) -> C:
...
def validate_b(self, obj: B) -> bool:
...
class B:
...
Since this change breaks compatibility, the new behavior needs to be
enabled on a per-module basis in Python 3.7 using a *note __future__: 0.
import:
from __future__ import annotations
It will become the default in Python 3.10.
See also
........
PEP 563(3) – Postponed evaluation of annotations
PEP written and implemented by Łukasz Langa.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3107/
(2) https://peps.python.org/pep-0526/
(3) https://peps.python.org/pep-0563/
File: python.info, Node: PEP 538 Legacy C Locale Coercion, Next: PEP 540 Forced UTF-8 Runtime Mode, Prev: PEP 563 Postponed Evaluation of Annotations, Up: New Features<12>
1.7.2.2 PEP 538: Legacy C Locale Coercion
.........................................
An ongoing challenge within the Python 3 series has been determining a
sensible default strategy for handling the “7-bit ASCII” text encoding
assumption currently implied by the use of the default C or POSIX locale
on non-Windows platforms.
PEP 538(1) updates the default interpreter command line interface to
automatically coerce that locale to an available UTF-8 based locale as
described in the documentation of the new *note PYTHONCOERCECLOCALE:
ad5. environment variable. Automatically setting ‘LC_CTYPE’ this way
means that both the core interpreter and locale-aware C extensions (such
as *note readline: ba.) will assume the use of UTF-8 as the default text
encoding, rather than ASCII.
The platform support definition in PEP 11(2) has also been updated to
limit full text handling support to suitably configured non-ASCII based
locales.
As part of this change, the default error handler for *note stdin: 539.
and *note stdout: ad6. is now ‘surrogateescape’ (rather than ‘strict’)
when using any of the defined coercion target locales (currently
‘C.UTF-8’, ‘C.utf8’, and ‘UTF-8’). The default error handler for *note
stderr: 939. continues to be ‘backslashreplace’, regardless of locale.
Locale coercion is silent by default, but to assist in debugging
potentially locale related integration problems, explicit warnings
(emitted directly on *note stderr: 939.) can be requested by setting
‘PYTHONCOERCECLOCALE=warn’. This setting will also cause the Python
runtime to emit a warning if the legacy C locale remains active when the
core interpreter is initialized.
While PEP 538(3)’s locale coercion has the benefit of also affecting
extension modules (such as GNU ‘readline’), as well as child processes
(including those running non-Python applications and older versions of
Python), it has the downside of requiring that a suitable target locale
be present on the running system. To better handle the case where no
suitable target locale is available (as occurs on RHEL/CentOS 7, for
example), Python 3.7 also implements *note PEP 540; Forced UTF-8 Runtime
Mode: aca.
See also
........
PEP 538(4) – Coercing the legacy C locale to a UTF-8 based locale
PEP written and implemented by Nick Coghlan.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0538/
(2) https://peps.python.org/pep-0011/
(3) https://peps.python.org/pep-0538/
(4) https://peps.python.org/pep-0538/
File: python.info, Node: PEP 540 Forced UTF-8 Runtime Mode, Next: PEP 553 Built-in breakpoint, Prev: PEP 538 Legacy C Locale Coercion, Up: New Features<12>
1.7.2.3 PEP 540: Forced UTF-8 Runtime Mode
..........................................
The new *note -X: 176. ‘utf8’ command line option and *note PYTHONUTF8:
ad8. environment variable can be used to enable the *note Python UTF-8
Mode: 652.
When in UTF-8 mode, CPython ignores the locale settings, and uses the
UTF-8 encoding by default. The error handlers for *note sys.stdin: 539.
and *note sys.stdout: ad6. streams are set to ‘surrogateescape’.
The forced UTF-8 mode can be used to change the text handling behavior
in an embedded Python interpreter without changing the locale settings
of an embedding application.
While PEP 540(1)’s UTF-8 mode has the benefit of working regardless of
which locales are available on the running system, it has the downside
of having no effect on extension modules (such as GNU ‘readline’), child
processes running non-Python applications, and child processes running
older versions of Python. To reduce the risk of corrupting text data
when communicating with such components, Python 3.7 also implements
*note PEP 540; Forced UTF-8 Runtime Mode: aca.).
The UTF-8 mode is enabled by default when the locale is ‘C’ or ‘POSIX’,
and the PEP 538(2) locale coercion feature fails to change it to a UTF-8
based alternative (whether that failure is due to
‘PYTHONCOERCECLOCALE=0’ being set, ‘LC_ALL’ being set, or the lack of a
suitable target locale).
See also
........
PEP 540(3) – Add a new UTF-8 mode
PEP written and implemented by Victor Stinner
---------- Footnotes ----------
(1) https://peps.python.org/pep-0540/
(2) https://peps.python.org/pep-0538/
(3) https://peps.python.org/pep-0540/
File: python.info, Node: PEP 553 Built-in breakpoint, Next: PEP 539 New C API for Thread-Local Storage, Prev: PEP 540 Forced UTF-8 Runtime Mode, Up: New Features<12>
1.7.2.4 PEP 553: Built-in ‘breakpoint()’
........................................
Python 3.7 includes the new built-in *note breakpoint(): 236. function
as an easy and consistent way to enter the Python debugger.
Built-in ‘breakpoint()’ calls *note sys.breakpointhook(): ada. By
default, the latter imports *note pdb: a5. and then calls
‘pdb.set_trace()’, but by binding ‘sys.breakpointhook()’ to the function
of your choosing, ‘breakpoint()’ can enter any debugger. Additionally,
the environment variable *note PYTHONBREAKPOINT: adb. can be set to the
callable of your debugger of choice. Set ‘PYTHONBREAKPOINT=0’ to
completely disable built-in ‘breakpoint()’.
See also
........
PEP 553(1) – Built-in breakpoint()
PEP written and implemented by Barry Warsaw
---------- Footnotes ----------
(1) https://peps.python.org/pep-0553/
File: python.info, Node: PEP 539 New C API for Thread-Local Storage, Next: PEP 562 Customization of Access to Module Attributes, Prev: PEP 553 Built-in breakpoint, Up: New Features<12>
1.7.2.5 PEP 539: New C API for Thread-Local Storage
...................................................
While Python provides a C API for thread-local storage support; the
existing *note Thread Local Storage (TLS) API: add. has used int to
represent TLS keys across all platforms. This has not generally been a
problem for officially support platforms, but that is neither
POSIX-compliant, nor portable in any practical sense.
PEP 539(1) changes this by providing a new *note Thread Specific Storage
(TSS) API: ade. to CPython which supersedes use of the existing TLS API
within the CPython interpreter, while deprecating the existing API. The
TSS API uses a new type *note Py_tss_t: adf. instead of int to represent
TSS keys–an opaque type the definition of which may depend on the
underlying TLS implementation. Therefore, this will allow to build
CPython on platforms where the native TLS key is defined in a way that
cannot be safely cast to int.
Note that on platforms where the native TLS key is defined in a way that
cannot be safely cast to int, all functions of the existing TLS API will
be no-op and immediately return failure. This indicates clearly that
the old API is not supported on platforms where it cannot be used
reliably, and that no effort will be made to add such support.
See also
........
PEP 539(2) – A New C-API for Thread-Local Storage in CPython
PEP written by Erik M. Bray; implementation by Masayuki Yamamoto.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0539/
(2) https://peps.python.org/pep-0539/
File: python.info, Node: PEP 562 Customization of Access to Module Attributes, Next: PEP 564 New Time Functions With Nanosecond Resolution, Prev: PEP 539 New C API for Thread-Local Storage, Up: New Features<12>
1.7.2.6 PEP 562: Customization of Access to Module Attributes
.............................................................
Python 3.7 allows defining ‘__getattr__()’ on modules and will call it
whenever a module attribute is otherwise not found. Defining
‘__dir__()’ on modules is now also allowed.
A typical example of where this may be useful is module attribute
deprecation and lazy loading.
See also
........
PEP 562(1) – Module ‘__getattr__’ and ‘__dir__’
PEP written and implemented by Ivan Levkivskyi
---------- Footnotes ----------
(1) https://peps.python.org/pep-0562/
File: python.info, Node: PEP 564 New Time Functions With Nanosecond Resolution, Next: PEP 565 Show DeprecationWarning in __main__, Prev: PEP 562 Customization of Access to Module Attributes, Up: New Features<12>
1.7.2.7 PEP 564: New Time Functions With Nanosecond Resolution
..............................................................
The resolution of clocks in modern systems can exceed the limited
precision of a floating-point number returned by the *note time.time():
256. function and its variants. To avoid loss of precision, PEP 564(1)
adds six new “nanosecond” variants of the existing timer functions to
the *note time: ee. module:
* *note time.clock_gettime_ns(): ae2.
* *note time.clock_settime_ns(): ae3.
* *note time.monotonic_ns(): ae4.
* *note time.perf_counter_ns(): ae5.
* *note time.process_time_ns(): ae6.
* *note time.time_ns(): ae7.
The new functions return the number of nanoseconds as an integer value.
Measurements(2) show that on Linux and Windows the resolution of *note
time.time_ns(): ae7. is approximately 3 times better than that of *note
time.time(): 256.
See also
........
PEP 564(3) – Add new time functions with nanosecond resolution
PEP written and implemented by Victor Stinner
---------- Footnotes ----------
(1) https://peps.python.org/pep-0564/
(2)
https://peps.python.org/pep-0564/#annex-clocks-resolution-in-python
(3) https://peps.python.org/pep-0564/
File: python.info, Node: PEP 565 Show DeprecationWarning in __main__, Next: PEP 560 Core Support for typing module and Generic Types, Prev: PEP 564 New Time Functions With Nanosecond Resolution, Up: New Features<12>
1.7.2.8 PEP 565: Show DeprecationWarning in ‘__main__’
......................................................
The default handling of *note DeprecationWarning: 1a5. has been changed
such that these warnings are once more shown by default, but only when
the code triggering them is running directly in the *note __main__: 1.
module. As a result, developers of single file scripts and those using
Python interactively should once again start seeing deprecation warnings
for the APIs they use, but deprecation warnings triggered by imported
application, library and framework modules will continue to be hidden by
default.
As a result of this change, the standard library now allows developers
to choose between three different deprecation warning behaviours:
* *note FutureWarning: 411.: always displayed by default, recommended
for warnings intended to be seen by application end users (e.g.
for deprecated application configuration settings).
* *note DeprecationWarning: 1a5.: displayed by default only in *note
__main__: 1. and when running tests, recommended for warnings
intended to be seen by other Python developers where a version
upgrade may result in changed behaviour or an error.
* *note PendingDeprecationWarning: 8c7.: displayed by default only
when running tests, intended for cases where a future version
upgrade will change the warning category to *note
DeprecationWarning: 1a5. or *note FutureWarning: 411.
Previously both *note DeprecationWarning: 1a5. and *note
PendingDeprecationWarning: 8c7. were only visible when running tests,
which meant that developers primarily writing single file scripts or
using Python interactively could be surprised by breaking changes in the
APIs they used.
See also
........
PEP 565(1) – Show DeprecationWarning in ‘__main__’
PEP written and implemented by Nick Coghlan
---------- Footnotes ----------
(1) https://peps.python.org/pep-0565/
File: python.info, Node: PEP 560 Core Support for typing module and Generic Types, Next: PEP 552 Hash-based pyc Files, Prev: PEP 565 Show DeprecationWarning in __main__, Up: New Features<12>
1.7.2.9 PEP 560: Core Support for ‘typing’ module and Generic Types
...................................................................
Initially PEP 484(1) was designed in such way that it would not
introduce 'any' changes to the core CPython interpreter. Now type hints
and the *note typing: 104. module are extensively used by the community,
so this restriction is removed. The PEP introduces two special methods
‘__class_getitem__()’ and ‘__mro_entries__’, these methods are now used
by most classes and special constructs in *note typing: 104. As a
result, the speed of various operations with types increased up to 7
times, the generic types can be used without metaclass conflicts, and
several long standing bugs in *note typing: 104. module are fixed.
See also
........
PEP 560(2) – Core support for typing module and generic types
PEP written and implemented by Ivan Levkivskyi
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0560/
File: python.info, Node: PEP 552 Hash-based pyc Files, Next: PEP 545 Python Documentation Translations, Prev: PEP 560 Core Support for typing module and Generic Types, Up: New Features<12>
1.7.2.10 PEP 552: Hash-based .pyc Files
.......................................
Python has traditionally checked the up-to-dateness of bytecode cache
files (i.e., ‘.pyc’ files) by comparing the source metadata
(last-modified timestamp and size) with source metadata saved in the
cache file header when it was generated. While effective, this
invalidation method has its drawbacks. When filesystem timestamps are
too coarse, Python can miss source updates, leading to user confusion.
Additionally, having a timestamp in the cache file is problematic for
build reproducibility(1) and content-based build systems.
PEP 552(2) extends the pyc format to allow the hash of the source file
to be used for invalidation instead of the source timestamp. Such
‘.pyc’ files are called “hash-based”. By default, Python still uses
timestamp-based invalidation and does not generate hash-based ‘.pyc’
files at runtime. Hash-based ‘.pyc’ files may be generated with *note
py_compile: b3. or *note compileall: 20.
Hash-based ‘.pyc’ files come in two variants: checked and unchecked.
Python validates checked hash-based ‘.pyc’ files against the
corresponding source files at runtime but doesn’t do so for unchecked
hash-based pycs. Unchecked hash-based ‘.pyc’ files are a useful
performance optimization for environments where a system external to
Python (e.g., the build system) is responsible for keeping ‘.pyc’ files
up-to-date.
See *note Cached bytecode invalidation: 5e8. for more information.
See also
........
PEP 552(3) – Deterministic pycs
PEP written and implemented by Benjamin Peterson
---------- Footnotes ----------
(1) https://reproducible-builds.org/
(2) https://peps.python.org/pep-0552/
(3) https://peps.python.org/pep-0552/
File: python.info, Node: PEP 545 Python Documentation Translations, Next: Python Development Mode -X dev, Prev: PEP 552 Hash-based pyc Files, Up: New Features<12>
1.7.2.11 PEP 545: Python Documentation Translations
...................................................
PEP 545(1) describes the process of creating and maintaining Python
documentation translations.
Three new translations have been added:
- Japanese: ‘https://docs.python.org/ja/’
- French: ‘https://docs.python.org/fr/’
- Korean: ‘https://docs.python.org/ko/’
See also
........
PEP 545(2) – Python Documentation Translations
PEP written and implemented by Julien Palard, Inada Naoki, and
Victor Stinner.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0545/
(2) https://peps.python.org/pep-0545/
File: python.info, Node: Python Development Mode -X dev, Prev: PEP 545 Python Documentation Translations, Up: New Features<12>
1.7.2.12 Python Development Mode (-X dev)
.........................................
The new *note -X: 176. ‘dev’ command line option or the new *note
PYTHONDEVMODE: aed. environment variable can be used to enable *note
Python Development Mode: 1fa. When in development mode, Python performs
additional runtime checks that are too expensive to be enabled by
default. See *note Python Development Mode: 1fa. documentation for the
full description.
File: python.info, Node: Other Language Changes<7>, Next: New Modules<7>, Prev: New Features<12>, Up: What’s New In Python 3 7
1.7.3 Other Language Changes
----------------------------
* An *note await: 1b9. expression and comprehensions containing an
*note async for: aab. clause were illegal in the expressions in
*note formatted string literals: 9a9. due to a problem with the
implementation. In Python 3.7 this restriction was lifted.
* More than 255 arguments can now be passed to a function, and a
function can now have more than 255 parameters. (Contributed by
Serhiy Storchaka in bpo-12844(1) and bpo-18896(2).)
* *note bytes.fromhex(): aef. and *note bytearray.fromhex(): af0. now
ignore all ASCII whitespace, not only spaces. (Contributed by
Robert Xiao in bpo-28927(3).)
* *note str: 447, *note bytes: 1c2, and *note bytearray: 53a. gained
support for the new *note isascii(): af1. method, which can be used
to test if a string or bytes contain only the ASCII characters.
(Contributed by INADA Naoki in bpo-32677(4).)
* *note ImportError: 415. now displays module name and module
‘__file__’ path when ‘from ... import ...’ fails. (Contributed by
Matthias Bussonnier in bpo-29546(5).)
* Circular imports involving absolute imports with binding a
submodule to a name are now supported. (Contributed by Serhiy
Storchaka in bpo-30024(6).)
* ‘object.__format__(x, '')’ is now equivalent to ‘str(x)’ rather
than ‘format(str(self), '')’. (Contributed by Serhiy Storchaka in
bpo-28974(7).)
* In order to better support dynamic creation of stack traces, *note
types.TracebackType: af2. can now be instantiated from Python code,
and the *note tb_next: af3. attribute on *note tracebacks: af4. is
now writable. (Contributed by Nathaniel J. Smith in bpo-30579(8).)
* When using the *note -m: 5dd. switch, ‘sys.path[0]’ is now eagerly
expanded to the full starting directory path, rather than being
left as the empty directory (which allows imports from the
'current' working directory at the time when an import occurs)
(Contributed by Nick Coghlan in bpo-33053(9).)
* The new *note -X: 176. ‘importtime’ option or the *note
PYTHONPROFILEIMPORTTIME: af5. environment variable can be used to
show the timing of each module import. (Contributed by Inada Naoki
in bpo-31415(10).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12844
(2) https://bugs.python.org/issue?@action=redirect&bpo=18896
(3) https://bugs.python.org/issue?@action=redirect&bpo=28927
(4) https://bugs.python.org/issue?@action=redirect&bpo=32677
(5) https://bugs.python.org/issue?@action=redirect&bpo=29546
(6) https://bugs.python.org/issue?@action=redirect&bpo=30024
(7) https://bugs.python.org/issue?@action=redirect&bpo=28974
(8) https://bugs.python.org/issue?@action=redirect&bpo=30579
(9) https://bugs.python.org/issue?@action=redirect&bpo=33053
(10) https://bugs.python.org/issue?@action=redirect&bpo=31415
File: python.info, Node: New Modules<7>, Next: Improved Modules<7>, Prev: Other Language Changes<7>, Up: What’s New In Python 3 7
1.7.4 New Modules
-----------------
* Menu:
* contextvars::
* dataclasses: dataclasses<3>.
* importlib.resources: importlib resources<2>.
File: python.info, Node: contextvars, Next: dataclasses<3>, Up: New Modules<7>
1.7.4.1 contextvars
...................
The new *note contextvars: 24. module and a set of *note new C APIs:
af8. introduce support for 'context variables'. Context variables are
conceptually similar to thread-local variables. Unlike TLS, context
variables support asynchronous code correctly.
The *note asyncio: a. and *note decimal: 36. modules have been updated
to use and support context variables out of the box. Particularly the
active decimal context is now stored in a context variable, which allows
decimal operations to work with the correct context in asynchronous
code.
See also
........
PEP 567(1) – Context Variables
PEP written and implemented by Yury Selivanov
---------- Footnotes ----------
(1) https://peps.python.org/pep-0567/
File: python.info, Node: dataclasses<3>, Next: importlib resources<2>, Prev: contextvars, Up: New Modules<7>
1.7.4.2 dataclasses
...................
The new *note dataclass(): 1cb. decorator provides a way to declare
'data classes'. A data class describes its attributes using class
variable annotations. Its constructor and other magic methods, such as
*note __repr__(): 618, *note __eq__(): afa, and *note __hash__(): afb.
are generated automatically.
Example:
@dataclass
class Point:
x: float
y: float
z: float = 0.0
p = Point(1.5, 2.5)
print(p) # produces "Point(x=1.5, y=2.5, z=0.0)"
See also
........
PEP 557(1) – Data Classes
PEP written and implemented by Eric V. Smith
---------- Footnotes ----------
(1) https://peps.python.org/pep-0557/
File: python.info, Node: importlib resources<2>, Prev: dataclasses<3>, Up: New Modules<7>
1.7.4.3 importlib.resources
...........................
The new *note importlib.resources: 7b. module provides several new APIs
and one new ABC for access to, opening, and reading 'resources' inside
packages. Resources are roughly similar to files inside packages, but
they needn’t be actual files on the physical file system. Module
loaders can provide a ‘get_resource_reader()’ function which returns a
*note importlib.abc.ResourceReader: afd. instance to support this new
API. Built-in file path loaders and zip file loaders both support this.
Contributed by Barry Warsaw and Brett Cannon in bpo-32248(1).
See also
........
importlib_resources(2) – a PyPI backport for earlier Python versions.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32248
(2) https://importlib-resources.readthedocs.io/en/latest/
File: python.info, Node: Improved Modules<7>, Next: C API Changes<6>, Prev: New Modules<7>, Up: What’s New In Python 3 7
1.7.5 Improved Modules
----------------------
* Menu:
* argparse: argparse<3>.
* asyncio: asyncio<7>.
* binascii::
* calendar: calendar<2>.
* collections: collections<2>.
* compileall: compileall<3>.
* concurrent.futures: concurrent futures<3>.
* contextlib: contextlib<3>.
* cProfile: cProfile<2>.
* crypt::
* datetime: datetime<4>.
* dbm: dbm<2>.
* decimal::
* dis: dis<3>.
* distutils: distutils<4>.
* enum: enum<5>.
* functools: functools<3>.
* gc: gc<4>.
* hmac: hmac<2>.
* http.client: http client.
* http.server: http server.
* idlelib and IDLE::
* importlib: importlib<4>.
* io: io<4>.
* ipaddress: ipaddress<3>.
* itertools: itertools<5>.
* locale: locale<4>.
* logging: logging<3>.
* math: math<6>.
* mimetypes: mimetypes<2>.
* msilib::
* multiprocessing: multiprocessing<4>.
* os: os<7>.
* pathlib: pathlib<8>.
* pdb: pdb<4>.
* py_compile: py_compile<3>.
* pydoc: pydoc<2>.
* queue: queue<2>.
* re: re<4>.
* signal: signal<2>.
* socket: socket<5>.
* socketserver::
* sqlite3: sqlite3<6>.
* ssl: ssl<5>.
* string: string<2>.
* subprocess: subprocess<2>.
* sys: sys<8>.
* time: time<5>.
* tkinter: tkinter<5>.
* tracemalloc: tracemalloc<2>.
* types: types<4>.
* unicodedata: unicodedata<6>.
* unittest: unittest<7>.
* unittest.mock: unittest mock.
* urllib.parse: urllib parse<5>.
* uu::
* uuid: uuid<2>.
* warnings: warnings<3>.
* xml: xml<5>.
* xml.etree: xml etree.
* xmlrpc.server: xmlrpc server.
* zipapp::
* zipfile: zipfile<2>.
File: python.info, Node: argparse<3>, Next: asyncio<7>, Up: Improved Modules<7>
1.7.5.1 argparse
................
The new *note ArgumentParser.parse_intermixed_args(): b00. method allows
intermixing options and positional arguments. (Contributed by paul.j3
in bpo-14191(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14191
File: python.info, Node: asyncio<7>, Next: binascii, Prev: argparse<3>, Up: Improved Modules<7>
1.7.5.2 asyncio
...............
The *note asyncio: a. module has received many new features, usability
and *note performance improvements: b02. Notable changes include:
* The new *note provisional: b03. *note asyncio.run(): 478. function
can be used to run a coroutine from synchronous code by
automatically creating and destroying the event loop. (Contributed
by Yury Selivanov in bpo-32314(1).)
* asyncio gained support for *note contextvars: 24. *note
loop.call_soon(): b04, *note loop.call_soon_threadsafe(): b05,
*note loop.call_later(): b06, *note loop.call_at(): b07, and *note
Future.add_done_callback(): b08. have a new optional keyword-only
'context' parameter. *note Tasks: 1be. now track their context
automatically. See PEP 567(2) for more details. (Contributed by
Yury Selivanov in bpo-32436(3).)
* The new *note asyncio.create_task(): 1bf. function has been added
as a shortcut to ‘asyncio.get_event_loop().create_task()’.
(Contributed by Andrew Svetlov in bpo-32311(4).)
* The new *note loop.start_tls(): b09. method can be used to upgrade
an existing connection to TLS. (Contributed by Yury Selivanov in
bpo-23749(5).)
* The new *note loop.sock_recv_into(): b0a. method allows reading
data from a socket directly into a provided buffer making it
possible to reduce data copies. (Contributed by Antoine Pitrou in
bpo-31819(6).)
* The new *note asyncio.current_task(): 479. function returns the
currently running *note Task: 1be. instance, and the new *note
asyncio.all_tasks(): 956. function returns a set of all existing
‘Task’ instances in a given loop. The ‘Task.current_task()’ and
‘Task.all_tasks()’ methods have been deprecated. (Contributed by
Andrew Svetlov in bpo-32250(7).)
* The new 'provisional' *note BufferedProtocol: a9d. class allows
implementing streaming protocols with manual control over the
receive buffer. (Contributed by Yury Selivanov in bpo-32251(8).)
* The new *note asyncio.get_running_loop(): b0b. function returns the
currently running loop, and raises a *note RuntimeError: 195. if no
loop is running. This is in contrast with *note
asyncio.get_event_loop(): 2c4, which will 'create' a new event loop
if none is running. (Contributed by Yury Selivanov in
bpo-32269(9).)
* The new *note StreamWriter.wait_closed(): b0c. coroutine method
allows waiting until the stream writer is closed. The new *note
StreamWriter.is_closing(): b0d. method can be used to determine if
the writer is closing. (Contributed by Andrew Svetlov in
bpo-32391(10).)
* The new *note loop.sock_sendfile(): b0e. coroutine method allows
sending files using *note os.sendfile: b0f. when possible.
(Contributed by Andrew Svetlov in bpo-32410(11).)
* The new *note Future.get_loop(): b10. and ‘Task.get_loop()’ methods
return the instance of the loop on which a task or a future were
created. *note Server.get_loop(): b11. allows doing the same for
*note asyncio.Server: b12. objects. (Contributed by Yury Selivanov
in bpo-32415(12) and Srinivas Reddy Thatiparthy in bpo-32418(13).)
* It is now possible to control how instances of *note
asyncio.Server: b12. begin serving. Previously, the server would
start serving immediately when created. The new 'start_serving'
keyword argument to *note loop.create_server(): b13. and *note
loop.create_unix_server(): 1ae, as well as *note
Server.start_serving(): b14, and *note Server.serve_forever(): b15.
can be used to decouple server instantiation and serving. The new
*note Server.is_serving(): b16. method returns ‘True’ if the server
is serving. *note Server: b12. objects are now asynchronous
context managers:
srv = await loop.create_server(...)
async with srv:
# some code
# At this point, srv is closed and no longer accepts new connections.
(Contributed by Yury Selivanov in bpo-32662(14).)
* Callback objects returned by *note loop.call_later(): b06. gained
the new *note when(): b17. method which returns an absolute
scheduled callback timestamp. (Contributed by Andrew Svetlov in
bpo-32741(15).)
* The *note loop.create_datagram_endpoint(): 72e. method gained
support for Unix sockets. (Contributed by Quentin Dawans in
bpo-31245(16).)
* The *note asyncio.open_connection(): b18, *note
asyncio.start_server(): b19. functions, *note
loop.create_connection(): 5ff, *note loop.create_server(): b13,
*note loop.create_accepted_socket(): b1a. methods and their
corresponding UNIX socket variants now accept the
'ssl_handshake_timeout' keyword argument. (Contributed by Neil
Aspinall in bpo-29970(17).)
* The new *note Handle.cancelled(): b1b. method returns ‘True’ if the
callback was cancelled. (Contributed by Marat Sharafutdinov in
bpo-31943(18).)
* The asyncio source has been converted to use the *note async:
72d./*note await: 1b9. syntax. (Contributed by Andrew Svetlov in
bpo-32193(19).)
* The new *note ReadTransport.is_reading(): b1c. method can be used
to determine the reading state of the transport. Additionally,
calls to *note ReadTransport.resume_reading(): b1d. and *note
ReadTransport.pause_reading(): b1e. are now idempotent.
(Contributed by Yury Selivanov in bpo-32356(20).)
* Loop methods which accept socket paths now support passing *note
path-like objects: 2a2. (Contributed by Yury Selivanov in
bpo-32066(21).)
* In *note asyncio: a. TCP sockets on Linux are now created with
‘TCP_NODELAY’ flag set by default. (Contributed by Yury Selivanov
and Victor Stinner in bpo-27456(22).)
* Exceptions occurring in cancelled tasks are no longer logged.
(Contributed by Yury Selivanov in bpo-30508(23).)
* New ‘WindowsSelectorEventLoopPolicy’ and
‘WindowsProactorEventLoopPolicy’ classes. (Contributed by Yury
Selivanov in bpo-33792(24).)
Several ‘asyncio’ APIs have been *note deprecated: b1f.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32314
(2) https://peps.python.org/pep-0567/
(3) https://bugs.python.org/issue?@action=redirect&bpo=32436
(4) https://bugs.python.org/issue?@action=redirect&bpo=32311
(5) https://bugs.python.org/issue?@action=redirect&bpo=23749
(6) https://bugs.python.org/issue?@action=redirect&bpo=31819
(7) https://bugs.python.org/issue?@action=redirect&bpo=32250
(8) https://bugs.python.org/issue?@action=redirect&bpo=32251
(9) https://bugs.python.org/issue?@action=redirect&bpo=32269
(10) https://bugs.python.org/issue?@action=redirect&bpo=32391
(11) https://bugs.python.org/issue?@action=redirect&bpo=32410
(12) https://bugs.python.org/issue?@action=redirect&bpo=32415
(13) https://bugs.python.org/issue?@action=redirect&bpo=32418
(14) https://bugs.python.org/issue?@action=redirect&bpo=32662
(15) https://bugs.python.org/issue?@action=redirect&bpo=32741
(16) https://bugs.python.org/issue?@action=redirect&bpo=31245
(17) https://bugs.python.org/issue?@action=redirect&bpo=29970
(18) https://bugs.python.org/issue?@action=redirect&bpo=31943
(19) https://bugs.python.org/issue?@action=redirect&bpo=32193
(20) https://bugs.python.org/issue?@action=redirect&bpo=32356
(21) https://bugs.python.org/issue?@action=redirect&bpo=32066
(22) https://bugs.python.org/issue?@action=redirect&bpo=27456
(23) https://bugs.python.org/issue?@action=redirect&bpo=30508
(24) https://bugs.python.org/issue?@action=redirect&bpo=33792
File: python.info, Node: binascii, Next: calendar<2>, Prev: asyncio<7>, Up: Improved Modules<7>
1.7.5.3 binascii
................
The *note b2a_uu(): b21. function now accepts an optional 'backtick'
keyword argument. When it’s true, zeros are represented by ‘'`'’
instead of spaces. (Contributed by Xiang Zhang in bpo-30103(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30103
File: python.info, Node: calendar<2>, Next: collections<2>, Prev: binascii, Up: Improved Modules<7>
1.7.5.4 calendar
................
The *note HTMLCalendar: b23. class has new class attributes which ease
the customization of CSS classes in the produced HTML calendar.
(Contributed by Oz Tiram in bpo-30095(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30095
File: python.info, Node: collections<2>, Next: compileall<3>, Prev: calendar<2>, Up: Improved Modules<7>
1.7.5.5 collections
...................
‘collections.namedtuple()’ now supports default values. (Contributed by
Raymond Hettinger in bpo-32320(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32320
File: python.info, Node: compileall<3>, Next: concurrent futures<3>, Prev: collections<2>, Up: Improved Modules<7>
1.7.5.6 compileall
..................
*note compileall.compile_dir(): b26. learned the new 'invalidation_mode'
parameter, which can be used to enable *note hash-based .pyc
invalidation: acb. The invalidation mode can also be specified on the
command line using the new ‘--invalidation-mode’ argument. (Contributed
by Benjamin Peterson in bpo-31650(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31650
File: python.info, Node: concurrent futures<3>, Next: contextlib<3>, Prev: compileall<3>, Up: Improved Modules<7>
1.7.5.7 concurrent.futures
..........................
*note ProcessPoolExecutor: 8ed. and *note ThreadPoolExecutor: 73d. now
support the new 'initializer' and 'initargs' constructor arguments.
(Contributed by Antoine Pitrou in bpo-21423(1).)
The *note ProcessPoolExecutor: 8ed. can now take the multiprocessing
context via the new 'mp_context' argument. (Contributed by Thomas
Moreau in bpo-31540(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21423
(2) https://bugs.python.org/issue?@action=redirect&bpo=31540
File: python.info, Node: contextlib<3>, Next: cProfile<2>, Prev: concurrent futures<3>, Up: Improved Modules<7>
1.7.5.8 contextlib
..................
The new *note nullcontext(): 7e9. is a simpler and faster no-op context
manager than *note ExitStack: b29. (Contributed by Jesse-Bakker in
bpo-10049(1).)
The new *note asynccontextmanager(): b2a, *note
AbstractAsyncContextManager: b2b, and *note AsyncExitStack: b2c. have
been added to complement their synchronous counterparts. (Contributed
by Jelle Zijlstra in bpo-29679(2) and bpo-30241(3), and by Alexander
Mohr and Ilya Kulakov in bpo-29302(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10049
(2) https://bugs.python.org/issue?@action=redirect&bpo=29679
(3) https://bugs.python.org/issue?@action=redirect&bpo=30241
(4) https://bugs.python.org/issue?@action=redirect&bpo=29302
File: python.info, Node: cProfile<2>, Next: crypt, Prev: contextlib<3>, Up: Improved Modules<7>
1.7.5.9 cProfile
................
The *note cProfile: 27. command line now accepts ‘-m module_name’ as an
alternative to script path. (Contributed by Sanyam Khurana in
bpo-21862(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21862
File: python.info, Node: crypt, Next: datetime<4>, Prev: cProfile<2>, Up: Improved Modules<7>
1.7.5.10 crypt
..............
The ‘crypt’ module now supports the Blowfish hashing method.
(Contributed by Serhiy Storchaka in bpo-31664(1).)
The ‘mksalt()’ function now allows specifying the number of rounds for
hashing. (Contributed by Serhiy Storchaka in bpo-31702(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31664
(2) https://bugs.python.org/issue?@action=redirect&bpo=31702
File: python.info, Node: datetime<4>, Next: dbm<2>, Prev: crypt, Up: Improved Modules<7>
1.7.5.11 datetime
.................
The new *note datetime.fromisoformat(): 613. method constructs a *note
datetime: 1cc. object from a string in one of the formats output by
*note datetime.isoformat(): b30. (Contributed by Paul Ganssle in
bpo-15873(1).)
The *note tzinfo: 5da. class now supports sub-minute offsets.
(Contributed by Alexander Belopolsky in bpo-5288(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15873
(2) https://bugs.python.org/issue?@action=redirect&bpo=5288
File: python.info, Node: dbm<2>, Next: decimal, Prev: datetime<4>, Up: Improved Modules<7>
1.7.5.12 dbm
............
*note dbm.dumb: 32. now supports reading read-only files and no longer
writes the index file when it is not changed.
File: python.info, Node: decimal, Next: dis<3>, Prev: dbm<2>, Up: Improved Modules<7>
1.7.5.13 decimal
................
The *note decimal: 36. module now uses *note context variables: ac0. to
store the decimal context. (Contributed by Yury Selivanov in
bpo-32630(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32630
File: python.info, Node: dis<3>, Next: distutils<4>, Prev: decimal, Up: Improved Modules<7>
1.7.5.14 dis
............
The *note dis(): b34. function is now able to disassemble nested code
objects (the code of comprehensions, generator expressions and nested
functions, and the code used for building nested classes). The maximum
depth of disassembly recursion is controlled by the new 'depth'
parameter. (Contributed by Serhiy Storchaka in bpo-11822(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11822
File: python.info, Node: distutils<4>, Next: enum<5>, Prev: dis<3>, Up: Improved Modules<7>
1.7.5.15 distutils
..................
‘README.rst’ is now included in the list of distutils standard READMEs
and therefore included in source distributions. (Contributed by Ryan
Gonzalez in bpo-11913(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11913
File: python.info, Node: enum<5>, Next: functools<3>, Prev: distutils<4>, Up: Improved Modules<7>
1.7.5.16 enum
.............
The *note Enum: 62c. learned the new ‘_ignore_’ class property, which
allows listing the names of properties which should not become enum
members. (Contributed by Ethan Furman in bpo-31801(1).)
In Python 3.8, attempting to check for non-Enum objects in ‘Enum’
classes will raise a *note TypeError: 534. (e.g. ‘1 in Color’);
similarly, attempting to check for non-Flag objects in a ‘Flag’ member
will raise *note TypeError: 534. (e.g. ‘1 in Perm.RW’); currently, both
operations return *note False: b37. instead and are deprecated.
(Contributed by Ethan Furman in bpo-33217(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31801
(2) https://bugs.python.org/issue?@action=redirect&bpo=33217
File: python.info, Node: functools<3>, Next: gc<4>, Prev: enum<5>, Up: Improved Modules<7>
1.7.5.17 functools
..................
*note functools.singledispatch(): 635. now supports registering
implementations using type annotations. (Contributed by Łukasz Langa in
bpo-32227(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32227
File: python.info, Node: gc<4>, Next: hmac<2>, Prev: functools<3>, Up: Improved Modules<7>
1.7.5.18 gc
...........
The new *note gc.freeze(): b3a. function allows freezing all objects
tracked by the garbage collector and excluding them from future
collections. This can be used before a POSIX ‘fork()’ call to make the
GC copy-on-write friendly or to speed up collection. The new *note
gc.unfreeze(): b3b. functions reverses this operation. Additionally,
*note gc.get_freeze_count(): b3c. can be used to obtain the number of
frozen objects. (Contributed by Li Zekun in bpo-31558(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31558
File: python.info, Node: hmac<2>, Next: http client, Prev: gc<4>, Up: Improved Modules<7>
1.7.5.19 hmac
.............
The *note hmac: 6a. module now has an optimized one-shot *note digest():
b3e. function, which is up to three times faster than ‘HMAC()’.
(Contributed by Christian Heimes in bpo-32433(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32433
File: python.info, Node: http client, Next: http server, Prev: hmac<2>, Up: Improved Modules<7>
1.7.5.20 http.client
....................
*note HTTPConnection: b40. and *note HTTPSConnection: b41. now support
the new 'blocksize' argument for improved upload throughput.
(Contributed by Nir Soffer in bpo-31945(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31945
File: python.info, Node: http server, Next: idlelib and IDLE, Prev: http client, Up: Improved Modules<7>
1.7.5.21 http.server
....................
*note SimpleHTTPRequestHandler: b43. now supports the HTTP
‘If-Modified-Since’ header. The server returns the 304 response status
if the target file was not modified after the time specified in the
header. (Contributed by Pierre Quentel in bpo-29654(1).)
*note SimpleHTTPRequestHandler: b43. accepts the new 'directory'
argument, in addition to the new ‘--directory’ command line argument.
With this parameter, the server serves the specified directory, by
default it uses the current working directory. (Contributed by Stéphane
Wirtel and Julien Palard in bpo-28707(2).)
The new *note ThreadingHTTPServer: b44. class uses threads to handle
requests using ‘ThreadingMixin’. It is used when ‘http.server’ is run
with ‘-m’. (Contributed by Julien Palard in bpo-31639(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=29654
(2) https://bugs.python.org/issue?@action=redirect&bpo=28707
(3) https://bugs.python.org/issue?@action=redirect&bpo=31639
File: python.info, Node: idlelib and IDLE, Next: importlib<4>, Prev: http server, Up: Improved Modules<7>
1.7.5.22 idlelib and IDLE
.........................
Multiple fixes for autocompletion. (Contributed by Louie Lu in
bpo-15786(1).)
Module Browser (on the File menu, formerly called Class Browser), now
displays nested functions and classes in addition to top-level functions
and classes. (Contributed by Guilherme Polo, Cheryl Sabella, and Terry
Jan Reedy in bpo-1612262(2).)
The Settings dialog (Options, Configure IDLE) has been partly rewritten
to improve both appearance and function. (Contributed by Cheryl Sabella
and Terry Jan Reedy in multiple issues.)
The font sample now includes a selection of non-Latin characters so that
users can better see the effect of selecting a particular font.
(Contributed by Terry Jan Reedy in bpo-13802(3).) The sample can be
edited to include other characters. (Contributed by Serhiy Storchaka in
bpo-31860(4).)
The IDLE features formerly implemented as extensions have been
reimplemented as normal features. Their settings have been moved from
the Extensions tab to other dialog tabs. (Contributed by Charles
Wohlganger and Terry Jan Reedy in bpo-27099(5).)
Editor code context option revised. Box displays all context lines up
to maxlines. Clicking on a context line jumps the editor to that line.
Context colors for custom themes is added to Highlights tab of Settings
dialog. (Contributed by Cheryl Sabella and Terry Jan Reedy in
bpo-33642(6), bpo-33768(7), and bpo-33679(8).)
On Windows, a new API call tells Windows that tk scales for DPI. On
Windows 8.1+ or 10, with DPI compatibility properties of the Python
binary unchanged, and a monitor resolution greater than 96 DPI, this
should make text and lines sharper. It should otherwise have no effect.
(Contributed by Terry Jan Reedy in bpo-33656(9).)
New in 3.7.1:
Output over N lines (50 by default) is squeezed down to a button. N can
be changed in the PyShell section of the General page of the Settings
dialog. Fewer, but possibly extra long, lines can be squeezed by right
clicking on the output. Squeezed output can be expanded in place by
double-clicking the button or into the clipboard or a separate window by
right-clicking the button. (Contributed by Tal Einat in
bpo-1529353(10).)
The changes above have been backported to 3.6 maintenance releases.
NEW in 3.7.4:
Add “Run Customized” to the Run menu to run a module with customized
settings. Any command line arguments entered are added to sys.argv.
They re-appear in the box for the next customized run. One can also
suppress the normal Shell main module restart. (Contributed by Cheryl
Sabella, Terry Jan Reedy, and others in bpo-5680(11) and bpo-37627(12).)
New in 3.7.5:
Add optional line numbers for IDLE editor windows. Windows open without
line numbers unless set otherwise in the General tab of the
configuration dialog. Line numbers for an existing window are shown and
hidden in the Options menu. (Contributed by Tal Einat and Saimadhav
Heblikar in bpo-17535(13).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15786
(2) https://bugs.python.org/issue?@action=redirect&bpo=1612262
(3) https://bugs.python.org/issue?@action=redirect&bpo=13802
(4) https://bugs.python.org/issue?@action=redirect&bpo=31860
(5) https://bugs.python.org/issue?@action=redirect&bpo=27099
(6) https://bugs.python.org/issue?@action=redirect&bpo=33642
(7) https://bugs.python.org/issue?@action=redirect&bpo=33768
(8) https://bugs.python.org/issue?@action=redirect&bpo=33679
(9) https://bugs.python.org/issue?@action=redirect&bpo=33656
(10) https://bugs.python.org/issue?@action=redirect&bpo=1529353
(11) https://bugs.python.org/issue?@action=redirect&bpo=5680
(12) https://bugs.python.org/issue?@action=redirect&bpo=37627
(13) https://bugs.python.org/issue?@action=redirect&bpo=17535
File: python.info, Node: importlib<4>, Next: io<4>, Prev: idlelib and IDLE, Up: Improved Modules<7>
1.7.5.23 importlib
..................
The *note importlib.abc.ResourceReader: afd. ABC was introduced to
support the loading of resources from packages. See also *note
importlib.resources: ac2. (Contributed by Barry Warsaw, Brett Cannon in
bpo-32248(1).)
*note importlib.reload(): 514. now raises *note ModuleNotFoundError:
b47. if the module lacks a spec. (Contributed by Garvit Khatri in
bpo-29851(2).)
‘importlib.find_spec()’ now raises *note ModuleNotFoundError: b47.
instead of *note AttributeError: 348. if the specified parent module is
not a package (i.e. lacks a ‘__path__’ attribute). (Contributed by
Milan Oberkirch in bpo-30436(3).)
The new ‘importlib.source_hash()’ can be used to compute the hash of the
passed source. A *note hash-based .pyc file: acb. embeds the value
returned by this function.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32248
(2) https://bugs.python.org/issue?@action=redirect&bpo=29851
(3) https://bugs.python.org/issue?@action=redirect&bpo=30436
File: python.info, Node: io<4>, Next: ipaddress<3>, Prev: importlib<4>, Up: Improved Modules<7>
1.7.5.24 io
...........
The new *note TextIOWrapper.reconfigure(): b49. method can be used to
reconfigure the text stream with the new settings. (Contributed by
Antoine Pitrou in bpo-30526(1) and INADA Naoki in bpo-15216(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30526
(2) https://bugs.python.org/issue?@action=redirect&bpo=15216
File: python.info, Node: ipaddress<3>, Next: itertools<5>, Prev: io<4>, Up: Improved Modules<7>
1.7.5.25 ipaddress
..................
The new ‘subnet_of()’ and ‘supernet_of()’ methods of *note
ipaddress.IPv6Network: 201. and *note ipaddress.IPv4Network: 200. can be
used for network containment tests. (Contributed by Michel Albert and
Cheryl Sabella in bpo-20825(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20825
File: python.info, Node: itertools<5>, Next: locale<4>, Prev: ipaddress<3>, Up: Improved Modules<7>
1.7.5.26 itertools
..................
*note itertools.islice(): b4c. now accepts *note integer-like objects:
718. as start, stop, and slice arguments. (Contributed by Will Roberts
in bpo-30537(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30537
File: python.info, Node: locale<4>, Next: logging<3>, Prev: itertools<5>, Up: Improved Modules<7>
1.7.5.27 locale
...............
The new 'monetary' argument to *note locale.format_string(): 51a. can be
used to make the conversion use monetary thousands separators and
grouping strings. (Contributed by Garvit in bpo-10379(1).)
The *note locale.getpreferredencoding(): 536. function now always
returns ‘'UTF-8'’ on Android or when in the *note forced UTF-8 mode:
aca.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10379
File: python.info, Node: logging<3>, Next: math<6>, Prev: locale<4>, Up: Improved Modules<7>
1.7.5.28 logging
................
*note Logger: b4f. instances can now be pickled. (Contributed by Vinay
Sajip in bpo-30520(1).)
The new *note StreamHandler.setStream(): b50. method can be used to
replace the logger stream after handler creation. (Contributed by Vinay
Sajip in bpo-30522(2).)
It is now possible to specify keyword arguments to handler constructors
in configuration passed to *note logging.config.fileConfig(): b51.
(Contributed by Preston Landers in bpo-31080(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30520
(2) https://bugs.python.org/issue?@action=redirect&bpo=30522
(3) https://bugs.python.org/issue?@action=redirect&bpo=31080
File: python.info, Node: math<6>, Next: mimetypes<2>, Prev: logging<3>, Up: Improved Modules<7>
1.7.5.29 math
.............
The new *note math.remainder(): b53. function implements the IEEE
754-style remainder operation. (Contributed by Mark Dickinson in
bpo-29962(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=29962
File: python.info, Node: mimetypes<2>, Next: msilib, Prev: math<6>, Up: Improved Modules<7>
1.7.5.30 mimetypes
..................
The MIME type of .bmp has been changed from ‘'image/x-ms-bmp'’ to
‘'image/bmp'’. (Contributed by Nitish Chandra in bpo-22589(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22589
File: python.info, Node: msilib, Next: multiprocessing<4>, Prev: mimetypes<2>, Up: Improved Modules<7>
1.7.5.31 msilib
...............
The new ‘Database.Close()’ method can be used to close the MSI database.
(Contributed by Berker Peksag in bpo-20486(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20486
File: python.info, Node: multiprocessing<4>, Next: os<7>, Prev: msilib, Up: Improved Modules<7>
1.7.5.32 multiprocessing
........................
The new *note Process.close(): b57. method explicitly closes the process
object and releases all resources associated with it. *note ValueError:
204. is raised if the underlying process is still running. (Contributed
by Antoine Pitrou in bpo-30596(1).)
The new *note Process.kill(): b58. method can be used to terminate the
process using the ‘SIGKILL’ signal on Unix. (Contributed by Vitor
Pereira in bpo-30794(2).)
Non-daemonic threads created by *note Process: b59. are now joined on
process exit. (Contributed by Antoine Pitrou in bpo-18966(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30596
(2) https://bugs.python.org/issue?@action=redirect&bpo=30794
(3) https://bugs.python.org/issue?@action=redirect&bpo=18966
File: python.info, Node: os<7>, Next: pathlib<8>, Prev: multiprocessing<4>, Up: Improved Modules<7>
1.7.5.33 os
...........
*note os.fwalk(): b5b. now accepts the 'path' argument as *note bytes:
1c2. (Contributed by Serhiy Storchaka in bpo-28682(1).)
*note os.scandir(): a5e. gained support for *note file descriptors: b5c.
(Contributed by Serhiy Storchaka in bpo-25996(2).)
The new *note register_at_fork(): 2fa. function allows registering
Python callbacks to be executed at process fork. (Contributed by
Antoine Pitrou in bpo-16500(3).)
Added *note os.preadv(): b5d. (combine the functionality of *note
os.readv(): b5e. and *note os.pread(): b5f.) and *note os.pwritev():
b60. functions (combine the functionality of *note os.writev(): b61. and
*note os.pwrite(): b62.). (Contributed by Pablo Galindo in
bpo-31368(4).)
The mode argument of *note os.makedirs(): 220. no longer affects the
file permission bits of newly created intermediate-level directories.
(Contributed by Serhiy Storchaka in bpo-19930(5).)
*note os.dup2(): b63. now returns the new file descriptor. Previously,
‘None’ was always returned. (Contributed by Benjamin Peterson in
bpo-32441(6).)
The structure returned by *note os.stat(): 49c. now contains the *note
st_fstype: b64. attribute on Solaris and its derivatives. (Contributed
by Jesús Cea Avión in bpo-32659(7).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28682
(2) https://bugs.python.org/issue?@action=redirect&bpo=25996
(3) https://bugs.python.org/issue?@action=redirect&bpo=16500
(4) https://bugs.python.org/issue?@action=redirect&bpo=31368
(5) https://bugs.python.org/issue?@action=redirect&bpo=19930
(6) https://bugs.python.org/issue?@action=redirect&bpo=32441
(7) https://bugs.python.org/issue?@action=redirect&bpo=32659
File: python.info, Node: pathlib<8>, Next: pdb<4>, Prev: os<7>, Up: Improved Modules<7>
1.7.5.34 pathlib
................
The new *note Path.is_mount(): a16. method is now available on POSIX
systems and can be used to determine whether a path is a mount point.
(Contributed by Cooper Ry Lees in bpo-30897(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30897
File: python.info, Node: pdb<4>, Next: py_compile<3>, Prev: pathlib<8>, Up: Improved Modules<7>
1.7.5.35 pdb
............
*note pdb.set_trace(): 237. now takes an optional 'header' keyword-only
argument. If given, it is printed to the console just before debugging
begins. (Contributed by Barry Warsaw in bpo-31389(1).)
*note pdb: a5. command line now accepts ‘-m module_name’ as an
alternative to script file. (Contributed by Mario Corchero in
bpo-32206(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31389
(2) https://bugs.python.org/issue?@action=redirect&bpo=32206
File: python.info, Node: py_compile<3>, Next: pydoc<2>, Prev: pdb<4>, Up: Improved Modules<7>
1.7.5.36 py_compile
...................
*note py_compile.compile(): a24. – and by extension, *note compileall:
20. – now respects the ‘SOURCE_DATE_EPOCH’ environment variable by
unconditionally creating ‘.pyc’ files for hash-based validation. This
allows for guaranteeing reproducible builds(1) of ‘.pyc’ files when they
are created eagerly. (Contributed by Bernhard M. Wiedemann in
bpo-29708(2).)
---------- Footnotes ----------
(1) https://reproducible-builds.org/
(2) https://bugs.python.org/issue?@action=redirect&bpo=29708
File: python.info, Node: pydoc<2>, Next: queue<2>, Prev: py_compile<3>, Up: Improved Modules<7>
1.7.5.37 pydoc
..............
The pydoc server can now bind to an arbitrary hostname specified by the
new ‘-n’ command-line argument. (Contributed by Feanil Patel in
bpo-31128(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31128
File: python.info, Node: queue<2>, Next: re<4>, Prev: pydoc<2>, Up: Improved Modules<7>
1.7.5.38 queue
..............
The new *note SimpleQueue: b6a. class is an unbounded FIFO queue.
(Contributed by Antoine Pitrou in bpo-14976(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14976
File: python.info, Node: re<4>, Next: signal<2>, Prev: queue<2>, Up: Improved Modules<7>
1.7.5.39 re
...........
The flags *note re.ASCII: 628, *note re.LOCALE: b6c. and *note
re.UNICODE: b6d. can be set within the scope of a group. (Contributed
by Serhiy Storchaka in bpo-31690(1).)
*note re.split(): 2a4. now supports splitting on a pattern like ‘r'\b'’,
‘'^$'’ or ‘(?=-)’ that matches an empty string. (Contributed by Serhiy
Storchaka in bpo-25054(2).)
Regular expressions compiled with the *note re.LOCALE: b6c. flag no
longer depend on the locale at compile time. Locale settings are
applied only when the compiled regular expression is used. (Contributed
by Serhiy Storchaka in bpo-30215(3).)
*note FutureWarning: 411. is now emitted if a regular expression
contains character set constructs that will change semantically in the
future, such as nested sets and set operations. (Contributed by Serhiy
Storchaka in bpo-30349(4).)
Compiled regular expression and match objects can now be copied using
*note copy.copy(): 52f. and *note copy.deepcopy(): b6e. (Contributed by
Serhiy Storchaka in bpo-10076(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31690
(2) https://bugs.python.org/issue?@action=redirect&bpo=25054
(3) https://bugs.python.org/issue?@action=redirect&bpo=30215
(4) https://bugs.python.org/issue?@action=redirect&bpo=30349
(5) https://bugs.python.org/issue?@action=redirect&bpo=10076
File: python.info, Node: signal<2>, Next: socket<5>, Prev: re<4>, Up: Improved Modules<7>
1.7.5.40 signal
...............
The new 'warn_on_full_buffer' argument to the *note
signal.set_wakeup_fd(): b70. function makes it possible to specify
whether Python prints a warning on stderr when the wakeup buffer
overflows. (Contributed by Nathaniel J. Smith in bpo-30050(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30050
File: python.info, Node: socket<5>, Next: socketserver, Prev: signal<2>, Up: Improved Modules<7>
1.7.5.41 socket
...............
The new *note socket.getblocking(): b72. method returns ‘True’ if the
socket is in blocking mode and ‘False’ otherwise. (Contributed by Yury
Selivanov in bpo-32373(1).)
The new *note socket.close(): b73. function closes the passed socket
file descriptor. This function should be used instead of *note
os.close(): b74. for better compatibility across platforms.
(Contributed by Christian Heimes in bpo-32454(2).)
The *note socket: cc. module now exposes the ‘socket.TCP_CONGESTION’
(Linux 2.6.13), ‘socket.TCP_USER_TIMEOUT’ (Linux 2.6.37), and
‘socket.TCP_NOTSENT_LOWAT’ (Linux 3.12) constants. (Contributed by Omar
Sandoval in bpo-26273(3) and Nathaniel J. Smith in bpo-29728(4).)
Support for *note socket.AF_VSOCK: b75. sockets has been added to allow
communication between virtual machines and their hosts. (Contributed by
Cathy Avery in bpo-27584(5).)
Sockets now auto-detect family, type and protocol from file descriptor
by default. (Contributed by Christian Heimes in bpo-28134(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32373
(2) https://bugs.python.org/issue?@action=redirect&bpo=32454
(3) https://bugs.python.org/issue?@action=redirect&bpo=26273
(4) https://bugs.python.org/issue?@action=redirect&bpo=29728
(5) https://bugs.python.org/issue?@action=redirect&bpo=27584
(6) https://bugs.python.org/issue?@action=redirect&bpo=28134
File: python.info, Node: socketserver, Next: sqlite3<6>, Prev: socket<5>, Up: Improved Modules<7>
1.7.5.42 socketserver
.....................
‘socketserver.ThreadingMixIn.server_close()’ now waits until all
non-daemon threads complete. ‘socketserver.ForkingMixIn.server_close()’
now waits until all child processes complete.
Add a new ‘socketserver.ForkingMixIn.block_on_close’ class attribute to
*note socketserver.ForkingMixIn: b77. and *note
socketserver.ThreadingMixIn: b78. classes. Set the class attribute to
‘False’ to get the pre-3.7 behaviour.
File: python.info, Node: sqlite3<6>, Next: ssl<5>, Prev: socketserver, Up: Improved Modules<7>
1.7.5.43 sqlite3
................
*note sqlite3.Connection: 247. now exposes the *note backup(): b7a.
method when the underlying SQLite library is at version 3.6.11 or
higher. (Contributed by Lele Gaifax in bpo-27645(1).)
The 'database' argument of *note sqlite3.connect(): 2aa. now accepts any
*note path-like object: 2a2, instead of just a string. (Contributed by
Anders Lorentsen in bpo-31843(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27645
(2) https://bugs.python.org/issue?@action=redirect&bpo=31843
File: python.info, Node: ssl<5>, Next: string<2>, Prev: sqlite3<6>, Up: Improved Modules<7>
1.7.5.44 ssl
............
The *note ssl: d0. module now uses OpenSSL’s builtin API instead of
‘match_hostname()’ to check a host name or an IP address. Values are
validated during TLS handshake. Any certificate validation error
including failing the host name check now raises *note
SSLCertVerificationError: b7c. and aborts the handshake with a proper
TLS Alert message. The new exception contains additional information.
Host name validation can be customized with *note
SSLContext.hostname_checks_common_name: b7d. (Contributed by Christian
Heimes in bpo-31399(1).)
Note: The improved host name check requires a 'libssl'
implementation compatible with OpenSSL 1.0.2 or 1.1. Consequently,
OpenSSL 0.9.8 and 1.0.1 are no longer supported (see *note Platform
Support Removals: b7e. for more details). The ssl module is mostly
compatible with LibreSSL 2.7.2 and newer.
The ‘ssl’ module no longer sends IP addresses in SNI TLS extension.
(Contributed by Christian Heimes in bpo-32185(2).)
‘match_hostname()’ no longer supports partial wildcards like
‘www*.example.org’. (Contributed by Mandeep Singh in bpo-23033(3) and
Christian Heimes in bpo-31399(4).)
The default cipher suite selection of the ‘ssl’ module now uses a
blacklist approach rather than a hard-coded whitelist. Python no longer
re-enables ciphers that have been blocked by OpenSSL security updates.
Default cipher suite selection can be configured at compile time.
(Contributed by Christian Heimes in bpo-31429(5).)
Validation of server certificates containing internationalized domain
names (IDNs) is now supported. As part of this change, the *note
SSLSocket.server_hostname: b7f. attribute now stores the expected
hostname in A-label form (‘"xn--pythn-mua.org"’), rather than the
U-label form (‘"pythön.org"’). (Contributed by Nathaniel J. Smith and
Christian Heimes in bpo-28414(6).)
The ‘ssl’ module has preliminary and experimental support for TLS 1.3
and OpenSSL 1.1.1. At the time of Python 3.7.0 release, OpenSSL 1.1.1
is still under development and TLS 1.3 hasn’t been finalized yet. The
TLS 1.3 handshake and protocol behaves slightly differently than TLS 1.2
and earlier, see *note TLS 1.3: b80. (Contributed by Christian Heimes
in bpo-32947(7), bpo-20995(8), bpo-29136(9), bpo-30622(10) and
bpo-33618(11))
*note SSLSocket: 8e9. and *note SSLObject: b81. no longer have a public
constructor. Direct instantiation was never a documented and supported
feature. Instances must be created with *note SSLContext: 296. methods
*note wrap_socket(): 520. and *note wrap_bio(): b82. (Contributed by
Christian Heimes in bpo-32951(12))
OpenSSL 1.1 APIs for setting the minimum and maximum TLS protocol
version are available as *note SSLContext.minimum_version: 86d. and
*note SSLContext.maximum_version: 86e. Supported protocols are
indicated by several new flags, such as *note HAS_TLSv1_1: b83.
(Contributed by Christian Heimes in bpo-32609(13).)
Added *note ssl.SSLContext.post_handshake_auth: a30. to enable and *note
ssl.SSLSocket.verify_client_post_handshake(): a31. to initiate TLS 1.3
post-handshake authentication. (Contributed by Christian Heimes in
gh-78851(14).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31399
(2) https://bugs.python.org/issue?@action=redirect&bpo=32185
(3) https://bugs.python.org/issue?@action=redirect&bpo=23033
(4) https://bugs.python.org/issue?@action=redirect&bpo=31399
(5) https://bugs.python.org/issue?@action=redirect&bpo=31429
(6) https://bugs.python.org/issue?@action=redirect&bpo=28414
(7) https://bugs.python.org/issue?@action=redirect&bpo=32947
(8) https://bugs.python.org/issue?@action=redirect&bpo=20995
(9) https://bugs.python.org/issue?@action=redirect&bpo=29136
(10) https://bugs.python.org/issue?@action=redirect&bpo=30622
(11) https://bugs.python.org/issue?@action=redirect&bpo=33618
(12) https://bugs.python.org/issue?@action=redirect&bpo=32951
(13) https://bugs.python.org/issue?@action=redirect&bpo=32609
(14) https://github.com/python/cpython/issues/78851
File: python.info, Node: string<2>, Next: subprocess<2>, Prev: ssl<5>, Up: Improved Modules<7>
1.7.5.45 string
...............
*note string.Template: 683. now lets you to optionally modify the
regular expression pattern for braced placeholders and non-braced
placeholders separately. (Contributed by Barry Warsaw in
bpo-1198569(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1198569
File: python.info, Node: subprocess<2>, Next: sys<8>, Prev: string<2>, Up: Improved Modules<7>
1.7.5.46 subprocess
...................
The *note subprocess.run(): b86. function accepts the new
'capture_output' keyword argument. When true, stdout and stderr will be
captured. This is equivalent to passing *note subprocess.PIPE: b87. as
'stdout' and 'stderr' arguments. (Contributed by Bo Bayles in
bpo-32102(1).)
The ‘subprocess.run’ function and the *note subprocess.Popen: 199.
constructor now accept the 'text' keyword argument as an alias to
'universal_newlines'. (Contributed by Andrew Clegg in bpo-31756(2).)
On Windows the default for 'close_fds' was changed from ‘False’ to
‘True’ when redirecting the standard handles. It’s now possible to set
'close_fds' to true when redirecting the standard handles. See *note
subprocess.Popen: 199. This means that 'close_fds' now defaults to
‘True’ on all supported platforms. (Contributed by Segev Finer in
bpo-19764(3).)
The subprocess module is now more graceful when handling *note
KeyboardInterrupt: 9d1. during *note subprocess.call(): b88, *note
subprocess.run(): b86, or in a *note Popen: 199. context manager. It
now waits a short amount of time for the child to exit, before
continuing the handling of the ‘KeyboardInterrupt’ exception.
(Contributed by Gregory P. Smith in bpo-25942(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32102
(2) https://bugs.python.org/issue?@action=redirect&bpo=31756
(3) https://bugs.python.org/issue?@action=redirect&bpo=19764
(4) https://bugs.python.org/issue?@action=redirect&bpo=25942
File: python.info, Node: sys<8>, Next: time<5>, Prev: subprocess<2>, Up: Improved Modules<7>
1.7.5.47 sys
............
The new *note sys.breakpointhook(): ada. hook function is called by the
built-in *note breakpoint(): 236. (Contributed by Barry Warsaw in
bpo-31353(1).)
On Android, the new *note sys.getandroidapilevel(): b8a. returns the
build-time Android API version. (Contributed by Victor Stinner in
bpo-28740(2).)
The new *note sys.get_coroutine_origin_tracking_depth(): b8b. function
returns the current coroutine origin tracking depth, as set by the new
*note sys.set_coroutine_origin_tracking_depth(): b8c. *note asyncio: a.
has been converted to use this new API instead of the deprecated
‘sys.set_coroutine_wrapper()’. (Contributed by Nathaniel J. Smith in
bpo-32591(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31353
(2) https://bugs.python.org/issue?@action=redirect&bpo=28740
(3) https://bugs.python.org/issue?@action=redirect&bpo=32591
File: python.info, Node: time<5>, Next: tkinter<5>, Prev: sys<8>, Up: Improved Modules<7>
1.7.5.48 time
.............
PEP 564(1) adds six new functions with nanosecond resolution to the
*note time: ee. module:
* *note time.clock_gettime_ns(): ae2.
* *note time.clock_settime_ns(): ae3.
* *note time.monotonic_ns(): ae4.
* *note time.perf_counter_ns(): ae5.
* *note time.process_time_ns(): ae6.
* *note time.time_ns(): ae7.
New clock identifiers have been added:
* *note time.CLOCK_BOOTTIME: b8e. (Linux): Identical to *note
time.CLOCK_MONOTONIC: 695, except it also includes any time that
the system is suspended.
* *note time.CLOCK_PROF: b8f. (FreeBSD, NetBSD and OpenBSD):
High-resolution per-process CPU timer.
* *note time.CLOCK_UPTIME: b90. (FreeBSD, OpenBSD): Time whose
absolute value is the time the system has been running and not
suspended, providing accurate uptime measurement.
The new *note time.thread_time(): 936. and *note time.thread_time_ns():
b91. functions can be used to get per-thread CPU time measurements.
(Contributed by Antoine Pitrou in bpo-32025(2).)
The new *note time.pthread_getcpuclockid(): b92. function returns the
clock ID of the thread-specific CPU-time clock.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0564/
(2) https://bugs.python.org/issue?@action=redirect&bpo=32025
File: python.info, Node: tkinter<5>, Next: tracemalloc<2>, Prev: time<5>, Up: Improved Modules<7>
1.7.5.49 tkinter
................
The new *note tkinter.ttk.Spinbox: b94. class is now available.
(Contributed by Alan Moore in bpo-32585(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32585
File: python.info, Node: tracemalloc<2>, Next: types<4>, Prev: tkinter<5>, Up: Improved Modules<7>
1.7.5.50 tracemalloc
....................
*note tracemalloc.Traceback: b96. behaves more like regular tracebacks,
sorting the frames from oldest to most recent. *note
Traceback.format(): b97. now accepts negative 'limit', truncating the
result to the ‘abs(limit)’ oldest frames. To get the old behaviour, use
the new 'most_recent_first' argument to ‘Traceback.format()’.
(Contributed by Jesse Bakker in bpo-32121(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32121
File: python.info, Node: types<4>, Next: unicodedata<6>, Prev: tracemalloc<2>, Up: Improved Modules<7>
1.7.5.51 types
..............
The new *note WrapperDescriptorType: b99, *note MethodWrapperType: 647,
*note MethodDescriptorType: b9a, and *note ClassMethodDescriptorType:
b9b. classes are now available. (Contributed by Manuel Krebber and
Guido van Rossum in bpo-29377(1), and Serhiy Storchaka in bpo-32265(2).)
The new *note types.resolve_bases(): b9c. function resolves MRO entries
dynamically as specified by PEP 560(3). (Contributed by Ivan Levkivskyi
in bpo-32717(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=29377
(2) https://bugs.python.org/issue?@action=redirect&bpo=32265
(3) https://peps.python.org/pep-0560/
(4) https://bugs.python.org/issue?@action=redirect&bpo=32717
File: python.info, Node: unicodedata<6>, Next: unittest<7>, Prev: types<4>, Up: Improved Modules<7>
1.7.5.52 unicodedata
....................
The internal *note unicodedata: 105. database has been upgraded to use
Unicode 11(1). (Contributed by Benjamin Peterson.)
---------- Footnotes ----------
(1) https://www.unicode.org/versions/Unicode11.0.0/
File: python.info, Node: unittest<7>, Next: unittest mock, Prev: unicodedata<6>, Up: Improved Modules<7>
1.7.5.53 unittest
.................
The new ‘-k’ command-line option allows filtering tests by a name
substring or a Unix shell-like pattern. For example, ‘python -m
unittest -k foo’ runs ‘foo_tests.SomeTest.test_something’,
‘bar_tests.SomeTest.test_foo’, but not
‘bar_tests.FooTest.test_something’. (Contributed by Jonas Haag in
bpo-32071(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32071
File: python.info, Node: unittest mock, Next: urllib parse<5>, Prev: unittest<7>, Up: Improved Modules<7>
1.7.5.54 unittest.mock
......................
The *note sentinel: ba0. attributes now preserve their identity when
they are *note copied: 25. or *note pickled: a6. (Contributed by Serhiy
Storchaka in bpo-20804(1).)
The new *note seal(): ba1. function allows sealing *note Mock: a4b.
instances, which will disallow further creation of attribute mocks. The
seal is applied recursively to all attributes that are themselves mocks.
(Contributed by Mario Corchero in bpo-30541(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20804
(2) https://bugs.python.org/issue?@action=redirect&bpo=30541
File: python.info, Node: urllib parse<5>, Next: uu, Prev: unittest mock, Up: Improved Modules<7>
1.7.5.55 urllib.parse
.....................
*note urllib.parse.quote(): ba3. has been updated from RFC 2396(1) to
RFC 3986(2), adding ‘~’ to the set of characters that are never quoted
by default. (Contributed by Christian Theune and Ratnadeep Debnath in
bpo-16285(3).)
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2396.html
(2) https://datatracker.ietf.org/doc/html/rfc3986.html
(3) https://bugs.python.org/issue?@action=redirect&bpo=16285
File: python.info, Node: uu, Next: uuid<2>, Prev: urllib parse<5>, Up: Improved Modules<7>
1.7.5.56 uu
...........
The ‘uu.encode()’ function now accepts an optional 'backtick' keyword
argument. When it’s true, zeros are represented by ‘'`'’ instead of
spaces. (Contributed by Xiang Zhang in bpo-30103(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30103
File: python.info, Node: uuid<2>, Next: warnings<3>, Prev: uu, Up: Improved Modules<7>
1.7.5.57 uuid
.............
The new *note UUID.is_safe: ba6. attribute relays information from the
platform about whether generated UUIDs are generated with a
multiprocessing-safe method. (Contributed by Barry Warsaw in
bpo-22807(1).)
*note uuid.getnode(): ba7. now prefers universally administered MAC
addresses over locally administered MAC addresses. This makes a better
guarantee for global uniqueness of UUIDs returned from *note
uuid.uuid1(): ba8. If only locally administered MAC addresses are
available, the first such one found is returned. (Contributed by Barry
Warsaw in bpo-32107(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22807
(2) https://bugs.python.org/issue?@action=redirect&bpo=32107
File: python.info, Node: warnings<3>, Next: xml<5>, Prev: uuid<2>, Up: Improved Modules<7>
1.7.5.58 warnings
.................
The initialization of the default warnings filters has changed as
follows:
* warnings enabled via command line options (including those for
*note -b: 5df. and the new CPython-specific *note -X: 176. ‘dev’
option) are always passed to the warnings machinery via the *note
sys.warnoptions: 3ac. attribute.
* warnings filters enabled via the command line or the environment
now have the following order of precedence:
* the ‘BytesWarning’ filter for *note -b: 5df. (or ‘-bb’)
* any filters specified with the *note -W: 8c6. option
* any filters specified with the *note PYTHONWARNINGS: baa.
environment variable
* any other CPython specific filters (e.g. the ‘default’ filter
added for the new ‘-X dev’ mode)
* any implicit filters defined directly by the warnings
machinery
* in *note CPython debug builds: 1fb, all warnings are now displayed
by default (the implicit filter list is empty)
(Contributed by Nick Coghlan and Victor Stinner in bpo-20361(1),
bpo-32043(2), and bpo-32230(3).)
Deprecation warnings are once again shown by default in single-file
scripts and at the interactive prompt. See *note PEP 565; Show
DeprecationWarning in __main__: acd. for details. (Contributed by Nick
Coghlan in bpo-31975(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20361
(2) https://bugs.python.org/issue?@action=redirect&bpo=32043
(3) https://bugs.python.org/issue?@action=redirect&bpo=32230
(4) https://bugs.python.org/issue?@action=redirect&bpo=31975
File: python.info, Node: xml<5>, Next: xml etree, Prev: warnings<3>, Up: Improved Modules<7>
1.7.5.59 xml
............
As mitigation against DTD and external entity retrieval, the *note
xml.dom.minidom: 122. and *note xml.sax: 129. modules no longer process
external entities by default. (Contributed by Christian Heimes in
gh-61441(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/61441
File: python.info, Node: xml etree, Next: xmlrpc server, Prev: xml<5>, Up: Improved Modules<7>
1.7.5.60 xml.etree
..................
*note ElementPath: bad. predicates in the ‘find()’ methods can now
compare text of the current node with ‘[. = "text"]’, not only text in
children. Predicates also allow adding spaces for better readability.
(Contributed by Stefan Behnel in bpo-31648(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31648
File: python.info, Node: xmlrpc server, Next: zipapp, Prev: xml etree, Up: Improved Modules<7>
1.7.5.61 xmlrpc.server
......................
‘SimpleXMLRPCDispatcher.register_function’ can now be used as a
decorator. (Contributed by Xiang Zhang in bpo-7769(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=7769
File: python.info, Node: zipapp, Next: zipfile<2>, Prev: xmlrpc server, Up: Improved Modules<7>
1.7.5.62 zipapp
...............
Function *note create_archive(): bb0. now accepts an optional 'filter'
argument to allow the user to select which files should be included in
the archive. (Contributed by Irmen de Jong in bpo-31072(1).)
Function *note create_archive(): bb0. now accepts an optional
'compressed' argument to generate a compressed archive. A command line
option ‘--compress’ has also been added to support compression.
(Contributed by Zhiming Wang in bpo-31638(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31072
(2) https://bugs.python.org/issue?@action=redirect&bpo=31638
File: python.info, Node: zipfile<2>, Prev: zipapp, Up: Improved Modules<7>
1.7.5.63 zipfile
................
*note ZipFile: 6c0. now accepts the new 'compresslevel' parameter to
control the compression level. (Contributed by Bo Bayles in
bpo-21417(1).)
Subdirectories in archives created by ‘ZipFile’ are now stored in
alphabetical order. (Contributed by Bernhard M. Wiedemann in
bpo-30693(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21417
(2) https://bugs.python.org/issue?@action=redirect&bpo=30693
File: python.info, Node: C API Changes<6>, Next: Build Changes<6>, Prev: Improved Modules<7>, Up: What’s New In Python 3 7
1.7.6 C API Changes
-------------------
A new API for thread-local storage has been implemented. See *note PEP
539; New C API for Thread-Local Storage: ace. for an overview and *note
Thread Specific Storage (TSS) API: ade. for a complete reference.
(Contributed by Masayuki Yamamoto in bpo-25658(1).)
The new *note context variables: ac0. functionality exposes a number of
*note new C APIs: af8.
The new *note PyImport_GetModule(): bb3. function returns the previously
imported module with the given name. (Contributed by Eric Snow in
bpo-28411(2).)
The new *note Py_RETURN_RICHCOMPARE: bb4. macro eases writing rich
comparison functions. (Contributed by Petr Victorin in bpo-23699(3).)
The new *note Py_UNREACHABLE: bb5. macro can be used to mark unreachable
code paths. (Contributed by Barry Warsaw in bpo-31338(4).)
The *note tracemalloc: ff. now exposes a C API through the new *note
PyTraceMalloc_Track(): bb6. and *note PyTraceMalloc_Untrack(): bb7.
functions. (Contributed by Victor Stinner in bpo-30054(5).)
The new ‘import__find__load__start()’ and ‘import__find__load__done()’
static markers can be used to trace module imports. (Contributed by
Christian Heimes in bpo-31574(6).)
The fields ‘name’ and ‘doc’ of structures *note PyMemberDef: 54c, *note
PyGetSetDef: bb8, *note PyStructSequence_Field: bb9, *note
PyStructSequence_Desc: bba, and ‘wrapperbase’ are now of type ‘const
char *’ rather of ‘char *’. (Contributed by Serhiy Storchaka in
bpo-28761(7).)
The result of *note PyUnicode_AsUTF8AndSize(): 897. and *note
PyUnicode_AsUTF8(): bbb. is now of type ‘const char *’ rather of ‘char
*’. (Contributed by Serhiy Storchaka in bpo-28769(8).)
The result of *note PyMapping_Keys(): bbc, *note PyMapping_Values():
bbd. and *note PyMapping_Items(): bbe. is now always a list, rather than
a list or a tuple. (Contributed by Oren Milman in bpo-28280(9).)
Added functions *note PySlice_Unpack(): 3f9. and *note
PySlice_AdjustIndices(): 3fa. (Contributed by Serhiy Storchaka in
bpo-27867(10).)
*note PyOS_AfterFork(): 3f6. is deprecated in favour of the new
functions *note PyOS_BeforeFork(): bbf, *note PyOS_AfterFork_Parent():
bc0. and *note PyOS_AfterFork_Child(): 3f7. (Contributed by Antoine
Pitrou in bpo-16500(11).)
The ‘PyExc_RecursionErrorInst’ singleton that was part of the public API
has been removed as its members being never cleared may cause a segfault
during finalization of the interpreter. Contributed by Xavier de Gaye
in bpo-22898(12) and bpo-30697(13).
Added C API support for timezones with timezone constructors *note
PyTimeZone_FromOffset(): bc1. and *note PyTimeZone_FromOffsetAndName():
bc2, and access to the UTC singleton with *note PyDateTime_TimeZone_UTC:
bc3. Contributed by Paul Ganssle in bpo-10381(14).
The type of results of ‘PyThread_start_new_thread()’ and
‘PyThread_get_thread_ident()’, and the 'id' parameter of *note
PyThreadState_SetAsyncExc(): bc4. changed from long to unsigned long.
(Contributed by Serhiy Storchaka in bpo-6532(15).)
*note PyUnicode_AsWideCharString(): 8bb. now raises a *note ValueError:
204. if the second argument is ‘NULL’ and the wchar_t* string contains
null characters. (Contributed by Serhiy Storchaka in bpo-30708(16).)
Changes to the startup sequence and the management of dynamic memory
allocators mean that the long documented requirement to call *note
Py_Initialize(): 8ab. before calling most C API functions is now relied
on more heavily, and failing to abide by it may lead to segfaults in
embedding applications. See the *note Porting to Python 3.7: ad1.
section in this document and the *note Before Python Initialization:
bc5. section in the C API documentation for more details.
The new *note PyInterpreterState_GetID(): bc6. returns the unique ID for
a given interpreter. (Contributed by Eric Snow in bpo-29102(17).)
*note Py_DecodeLocale(): bc7, *note Py_EncodeLocale(): bc8. now use the
UTF-8 encoding when the *note UTF-8 mode: aca. is enabled. (Contributed
by Victor Stinner in bpo-29240(18).)
*note PyUnicode_DecodeLocaleAndSize(): bc9. and *note
PyUnicode_EncodeLocale(): bca. now use the current locale encoding for
‘surrogateescape’ error handler. (Contributed by Victor Stinner in
bpo-29240(19).)
The 'start' and 'end' parameters of *note PyUnicode_FindChar(): 8b8. are
now adjusted to behave like string slices. (Contributed by Xiang Zhang
in bpo-28822(20).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25658
(2) https://bugs.python.org/issue?@action=redirect&bpo=28411
(3) https://bugs.python.org/issue?@action=redirect&bpo=23699
(4) https://bugs.python.org/issue?@action=redirect&bpo=31338
(5) https://bugs.python.org/issue?@action=redirect&bpo=30054
(6) https://bugs.python.org/issue?@action=redirect&bpo=31574
(7) https://bugs.python.org/issue?@action=redirect&bpo=28761
(8) https://bugs.python.org/issue?@action=redirect&bpo=28769
(9) https://bugs.python.org/issue?@action=redirect&bpo=28280
(10) https://bugs.python.org/issue?@action=redirect&bpo=27867
(11) https://bugs.python.org/issue?@action=redirect&bpo=16500
(12) https://bugs.python.org/issue?@action=redirect&bpo=22898
(13) https://bugs.python.org/issue?@action=redirect&bpo=30697
(14) https://bugs.python.org/issue?@action=redirect&bpo=10381
(15) https://bugs.python.org/issue?@action=redirect&bpo=6532
(16) https://bugs.python.org/issue?@action=redirect&bpo=30708
(17) https://bugs.python.org/issue?@action=redirect&bpo=29102
(18) https://bugs.python.org/issue?@action=redirect&bpo=29240
(19) https://bugs.python.org/issue?@action=redirect&bpo=29240
(20) https://bugs.python.org/issue?@action=redirect&bpo=28822
File: python.info, Node: Build Changes<6>, Next: Optimizations<7>, Prev: C API Changes<6>, Up: What’s New In Python 3 7
1.7.7 Build Changes
-------------------
Support for building ‘--without-threads’ has been removed. The *note
threading: ed. module is now always available. (Contributed by Antoine
Pitrou in bpo-31370(1).).
A full copy of libffi is no longer bundled for use when building the
*note _ctypes: 2a. module on non-OSX UNIX platforms. An installed copy
of libffi is now required when building ‘_ctypes’ on such platforms.
(Contributed by Zachary Ware in bpo-27979(2).)
The Windows build process no longer depends on Subversion to pull in
external sources, a Python script is used to download zipfiles from
GitHub instead. If Python 3.6 is not found on the system (via ‘py
-3.6’), NuGet is used to download a copy of 32-bit Python for this
purpose. (Contributed by Zachary Ware in bpo-30450(3).)
The *note ssl: d0. module requires OpenSSL 1.0.2 or 1.1 compatible
libssl. OpenSSL 1.0.1 has reached end of lifetime on 2016-12-31 and is
no longer supported. LibreSSL is temporarily not supported as well.
LibreSSL releases up to version 2.6.4 are missing required OpenSSL 1.0.2
APIs.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31370
(2) https://bugs.python.org/issue?@action=redirect&bpo=27979
(3) https://bugs.python.org/issue?@action=redirect&bpo=30450
File: python.info, Node: Optimizations<7>, Next: Other CPython Implementation Changes<2>, Prev: Build Changes<6>, Up: What’s New In Python 3 7
1.7.8 Optimizations
-------------------
The overhead of calling many methods of various standard library classes
implemented in C has been significantly reduced by porting more code to
use the ‘METH_FASTCALL’ convention. (Contributed by Victor Stinner in
bpo-29300(1), bpo-29507(2), bpo-29452(3), and bpo-29286(4).)
Various optimizations have reduced Python startup time by 10% on Linux
and up to 30% on macOS. (Contributed by Victor Stinner, INADA Naoki in
bpo-29585(5), and Ivan Levkivskyi in bpo-31333(6).)
Method calls are now up to 20% faster due to the bytecode changes which
avoid creating bound method instances. (Contributed by Yury Selivanov
and INADA Naoki in bpo-26110(7).) The *note asyncio: a. module received
a number of notable optimizations for commonly used functions:
* The *note asyncio.get_event_loop(): 2c4. function has been
reimplemented in C to make it up to 15 times faster. (Contributed
by Yury Selivanov in bpo-32296(8).)
* *note asyncio.Future: bcd. callback management has been optimized.
(Contributed by Yury Selivanov in bpo-32348(9).)
* *note asyncio.gather(): 5f9. is now up to 15% faster. (Contributed
by Yury Selivanov in bpo-32355(10).)
* *note asyncio.sleep(): a6f. is now up to 2 times faster when the
'delay' argument is zero or negative. (Contributed by Andrew
Svetlov in bpo-32351(11).)
* The performance overhead of asyncio debug mode has been reduced.
(Contributed by Antoine Pitrou in bpo-31970(12).)
As a result of *note PEP 560 work: ac5, the import time of *note typing:
104. has been reduced by a factor of 7, and many typing operations are
now faster. (Contributed by Ivan Levkivskyi in bpo-32226(13).)
*note sorted(): bce. and *note list.sort(): bcf. have been optimized for
common cases to be up to 40-75% faster. (Contributed by Elliot
Gorokhovsky in bpo-28685(14).)
*note dict.copy(): bd0. is now up to 5.5 times faster. (Contributed by
Yury Selivanov in bpo-31179(15).)
*note hasattr(): 4ce. and *note getattr(): bd1. are now about 4 times
faster when 'name' is not found and 'obj' does not override *note
object.__getattr__(): 4cf. or *note object.__getattribute__(): bd2.
(Contributed by INADA Naoki in bpo-32544(16).)
Searching for certain Unicode characters (like Ukrainian capital “Є”) in
a string was up to 25 times slower than searching for other characters.
It is now only 3 times slower in the worst case. (Contributed by Serhiy
Storchaka in bpo-24821(17).)
The *note collections.namedtuple(): 1ca. factory has been reimplemented
to make the creation of named tuples 4 to 6 times faster. (Contributed
by Jelle Zijlstra with further improvements by INADA Naoki, Serhiy
Storchaka, and Raymond Hettinger in bpo-28638(18).)
‘date.fromordinal()’ and ‘date.fromtimestamp()’ are now up to 30% faster
in the common case. (Contributed by Paul Ganssle in bpo-32403(19).)
The *note os.fwalk(): b5b. function is now up to 2 times faster thanks
to the use of *note os.scandir(): a5e. (Contributed by Serhiy Storchaka
in bpo-25996(20).)
The speed of the *note shutil.rmtree(): 2fb. function has been improved
by 20–40% thanks to the use of the *note os.scandir(): a5e. function.
(Contributed by Serhiy Storchaka in bpo-28564(21).)
Optimized case-insensitive matching and searching of *note regular
expressions: b9. Searching some patterns can now be up to 20 times
faster. (Contributed by Serhiy Storchaka in bpo-30285(22).)
*note re.compile(): bd3. now converts ‘flags’ parameter to int object if
it is ‘RegexFlag’. It is now as fast as Python 3.5, and faster than
Python 3.6 by about 10% depending on the pattern. (Contributed by INADA
Naoki in bpo-31671(23).)
The *note modify(): bd4. methods of classes *note
selectors.EpollSelector: bd5, *note selectors.PollSelector: bd6. and
*note selectors.DevpollSelector: bd7. may be around 10% faster under
heavy loads. (Contributed by Giampaolo Rodola’ in bpo-30014(24))
Constant folding has been moved from the peephole optimizer to the new
AST optimizer, which is able perform optimizations more consistently.
(Contributed by Eugene Toder and INADA Naoki in bpo-29469(25) and
bpo-11549(26).)
Most functions and methods in *note abc: 4. have been rewritten in C.
This makes creation of abstract base classes, and calling *note
isinstance(): 43d. and *note issubclass(): 7bc. on them 1.5x faster.
This also reduces Python start-up time by up to 10%. (Contributed by
Ivan Levkivskyi and INADA Naoki in bpo-31333(27))
Significant speed improvements to alternate constructors for *note
datetime.date: 1cd. and *note datetime.datetime: 1cc. by using fast-path
constructors when not constructing subclasses. (Contributed by Paul
Ganssle in bpo-32403(28))
The speed of comparison of *note array.array: 1a0. instances has been
improved considerably in certain cases. It is now from 10x to 70x
faster when comparing arrays holding values of the same integer type.
(Contributed by Adrian Wielgosik in bpo-24700(29).)
The *note math.erf(): bd8. and *note math.erfc(): bd9. functions now use
the (faster) C library implementation on most platforms. (Contributed
by Serhiy Storchaka in bpo-26121(30).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=29300
(2) https://bugs.python.org/issue?@action=redirect&bpo=29507
(3) https://bugs.python.org/issue?@action=redirect&bpo=29452
(4) https://bugs.python.org/issue?@action=redirect&bpo=29286
(5) https://bugs.python.org/issue?@action=redirect&bpo=29585
(6) https://bugs.python.org/issue?@action=redirect&bpo=31333
(7) https://bugs.python.org/issue?@action=redirect&bpo=26110
(8) https://bugs.python.org/issue?@action=redirect&bpo=32296
(9) https://bugs.python.org/issue?@action=redirect&bpo=32348
(10) https://bugs.python.org/issue?@action=redirect&bpo=32355
(11) https://bugs.python.org/issue?@action=redirect&bpo=32351
(12) https://bugs.python.org/issue?@action=redirect&bpo=31970
(13) https://bugs.python.org/issue?@action=redirect&bpo=32226
(14) https://bugs.python.org/issue?@action=redirect&bpo=28685
(15) https://bugs.python.org/issue?@action=redirect&bpo=31179
(16) https://bugs.python.org/issue?@action=redirect&bpo=32544
(17) https://bugs.python.org/issue?@action=redirect&bpo=24821
(18) https://bugs.python.org/issue?@action=redirect&bpo=28638
(19) https://bugs.python.org/issue?@action=redirect&bpo=32403
(20) https://bugs.python.org/issue?@action=redirect&bpo=25996
(21) https://bugs.python.org/issue?@action=redirect&bpo=28564
(22) https://bugs.python.org/issue?@action=redirect&bpo=30285
(23) https://bugs.python.org/issue?@action=redirect&bpo=31671
(24) https://bugs.python.org/issue?@action=redirect&bpo=30014
(25) https://bugs.python.org/issue?@action=redirect&bpo=29469
(26) https://bugs.python.org/issue?@action=redirect&bpo=11549
(27) https://bugs.python.org/issue?@action=redirect&bpo=31333
(28) https://bugs.python.org/issue?@action=redirect&bpo=32403
(29) https://bugs.python.org/issue?@action=redirect&bpo=24700
(30) https://bugs.python.org/issue?@action=redirect&bpo=26121
File: python.info, Node: Other CPython Implementation Changes<2>, Next: Deprecated Python Behavior, Prev: Optimizations<7>, Up: What’s New In Python 3 7
1.7.9 Other CPython Implementation Changes
------------------------------------------
* Trace hooks may now opt out of receiving the ‘line’ and opt into
receiving the ‘opcode’ events from the interpreter by setting the
corresponding new *note f_trace_lines: bdb. and *note
f_trace_opcodes: bdc. attributes on the frame being traced.
(Contributed by Nick Coghlan in bpo-31344(1).)
* Fixed some consistency problems with namespace package module
attributes. Namespace module objects now have an ‘__file__’ that
is set to ‘None’ (previously unset), and their ‘__spec__.origin’ is
also set to ‘None’ (previously the string ‘"namespace"’). See
bpo-32305(2). Also, the namespace module object’s
‘__spec__.loader’ is set to the same value as ‘__loader__’
(previously, the former was set to ‘None’). See bpo-32303(3).
* The *note locals(): 141. dictionary now displays in the lexical
order that variables were defined. Previously, the order was
undefined. (Contributed by Raymond Hettinger in bpo-32690(4).)
* The ‘distutils’ ‘upload’ command no longer tries to change CR
end-of-line characters to CRLF. This fixes a corruption issue with
sdists that ended with a byte equivalent to CR. (Contributed by Bo
Bayles in bpo-32304(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31344
(2) https://bugs.python.org/issue?@action=redirect&bpo=32305
(3) https://bugs.python.org/issue?@action=redirect&bpo=32303
(4) https://bugs.python.org/issue?@action=redirect&bpo=32690
(5) https://bugs.python.org/issue?@action=redirect&bpo=32304
File: python.info, Node: Deprecated Python Behavior, Next: Deprecated Python modules functions and methods, Prev: Other CPython Implementation Changes<2>, Up: What’s New In Python 3 7
1.7.10 Deprecated Python Behavior
---------------------------------
Yield expressions (both ‘yield’ and ‘yield from’ clauses) are now
deprecated in comprehensions and generator expressions (aside from the
iterable expression in the leftmost ‘for’ clause). This ensures that
comprehensions always immediately return a container of the appropriate
type (rather than potentially returning a *note generator iterator: bde.
object), while generator expressions won’t attempt to interleave their
implicit output with the output from any explicit yield expressions. In
Python 3.7, such expressions emit *note DeprecationWarning: 1a5. when
compiled, in Python 3.8 this will be a *note SyntaxError: 18d.
(Contributed by Serhiy Storchaka in bpo-10544(1).)
Returning a subclass of *note complex: 2f2. from *note
object.__complex__(): 5e3. is deprecated and will be an error in future
Python versions. This makes ‘__complex__()’ consistent with *note
object.__int__(): 717. and *note object.__float__(): 9cc. (Contributed
by Serhiy Storchaka in bpo-28894(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10544
(2) https://bugs.python.org/issue?@action=redirect&bpo=28894
File: python.info, Node: Deprecated Python modules functions and methods, Next: Deprecated functions and types of the C API, Prev: Deprecated Python Behavior, Up: What’s New In Python 3 7
1.7.11 Deprecated Python modules, functions and methods
-------------------------------------------------------
* Menu:
* aifc::
* asyncio: asyncio<8>.
* collections: collections<3>.
* dbm: dbm<3>.
* enum: enum<6>.
* gettext: gettext<2>.
* importlib: importlib<5>.
* locale: locale<5>.
* macpath::
* threading: threading<5>.
* socket: socket<6>.
* ssl: ssl<6>.
* sunau::
* sys: sys<9>.
* wave::
File: python.info, Node: aifc, Next: asyncio<8>, Up: Deprecated Python modules functions and methods
1.7.11.1 aifc
.............
‘aifc.openfp()’ has been deprecated and will be removed in Python 3.9.
Use ‘aifc.open()’ instead. (Contributed by Brian Curtin in
bpo-31985(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31985
File: python.info, Node: asyncio<8>, Next: collections<3>, Prev: aifc, Up: Deprecated Python modules functions and methods
1.7.11.2 asyncio
................
Support for directly ‘await’-ing instances of *note asyncio.Lock: a71.
and other asyncio synchronization primitives has been deprecated. An
asynchronous context manager must be used in order to acquire and
release the synchronization resource. (Contributed by Andrew Svetlov in
bpo-32253(1).)
The ‘asyncio.Task.current_task()’ and ‘asyncio.Task.all_tasks()’ methods
have been deprecated. (Contributed by Andrew Svetlov in bpo-32250(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=32253
(2) https://bugs.python.org/issue?@action=redirect&bpo=32250
File: python.info, Node: collections<3>, Next: dbm<3>, Prev: asyncio<8>, Up: Deprecated Python modules functions and methods
1.7.11.3 collections
....................
In Python 3.8, the abstract base classes in *note collections.abc: 1e.
will no longer be exposed in the regular *note collections: 1d. module.
This will help create a clearer distinction between the concrete classes
and the abstract base classes. (Contributed by Serhiy Storchaka in
bpo-25988(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25988
File: python.info, Node: dbm<3>, Next: enum<6>, Prev: collections<3>, Up: Deprecated Python modules functions and methods
1.7.11.4 dbm
............
*note dbm.dumb: 32. now supports reading read-only files and no longer
writes the index file when it is not changed. A deprecation warning is
now emitted if the index file is missing and recreated in the ‘'r'’ and
‘'w'’ modes (this will be an error in future Python releases).
(Contributed by Serhiy Storchaka in bpo-28847(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28847
File: python.info, Node: enum<6>, Next: gettext<2>, Prev: dbm<3>, Up: Deprecated Python modules functions and methods
1.7.11.5 enum
.............
In Python 3.8, attempting to check for non-Enum objects in ‘Enum’
classes will raise a *note TypeError: 534. (e.g. ‘1 in Color’);
similarly, attempting to check for non-Flag objects in a ‘Flag’ member
will raise *note TypeError: 534. (e.g. ‘1 in Perm.RW’); currently, both
operations return *note False: b37. instead. (Contributed by Ethan
Furman in bpo-33217(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=33217
File: python.info, Node: gettext<2>, Next: importlib<5>, Prev: enum<6>, Up: Deprecated Python modules functions and methods
1.7.11.6 gettext
................
Using non-integer value for selecting a plural form in *note gettext:
63. is now deprecated. It never correctly worked. (Contributed by
Serhiy Storchaka in bpo-28692(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28692
File: python.info, Node: importlib<5>, Next: locale<5>, Prev: gettext<2>, Up: Deprecated Python modules functions and methods
1.7.11.7 importlib
..................
Methods ‘MetaPathFinder.find_module()’ (replaced by *note
MetaPathFinder.find_spec(): 865.) and ‘PathEntryFinder.find_loader()’
(replaced by *note PathEntryFinder.find_spec(): 869.) both deprecated in
Python 3.4 now emit *note DeprecationWarning: 1a5. (Contributed by
Matthias Bussonnier in bpo-29576(1).)
The *note importlib.abc.ResourceLoader: be7. ABC has been deprecated in
favour of *note importlib.abc.ResourceReader: afd.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=29576
File: python.info, Node: locale<5>, Next: macpath, Prev: importlib<5>, Up: Deprecated Python modules functions and methods
1.7.11.8 locale
...............
‘locale.format()’ has been deprecated, use *note locale.format_string():
51a. instead. (Contributed by Garvit in bpo-10379(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10379
File: python.info, Node: macpath, Next: threading<5>, Prev: locale<5>, Up: Deprecated Python modules functions and methods
1.7.11.9 macpath
................
The ‘macpath’ is now deprecated and will be removed in Python 3.8.
(Contributed by Chi Hsuan Yen in bpo-9850(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9850
File: python.info, Node: threading<5>, Next: socket<6>, Prev: macpath, Up: Deprecated Python modules functions and methods
1.7.11.10 threading
...................
‘dummy_threading’ and ‘_dummy_thread’ have been deprecated. It is no
longer possible to build Python with threading disabled. Use *note
threading: ed. instead. (Contributed by Antoine Pitrou in
bpo-31370(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31370
File: python.info, Node: socket<6>, Next: ssl<6>, Prev: threading<5>, Up: Deprecated Python modules functions and methods
1.7.11.11 socket
................
The silent argument value truncation in *note socket.htons(): 87d. and
*note socket.ntohs(): 87e. has been deprecated. In future versions of
Python, if the passed argument is larger than 16 bits, an exception will
be raised. (Contributed by Oren Milman in bpo-28332(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28332
File: python.info, Node: ssl<6>, Next: sunau, Prev: socket<6>, Up: Deprecated Python modules functions and methods
1.7.11.12 ssl
.............
‘ssl.wrap_socket()’ is deprecated. Use *note
ssl.SSLContext.wrap_socket(): 520. instead. (Contributed by Christian
Heimes in bpo-28124(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28124
File: python.info, Node: sunau, Next: sys<9>, Prev: ssl<6>, Up: Deprecated Python modules functions and methods
1.7.11.13 sunau
...............
‘sunau.openfp()’ has been deprecated and will be removed in Python 3.9.
Use ‘sunau.open()’ instead. (Contributed by Brian Curtin in
bpo-31985(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31985
File: python.info, Node: sys<9>, Next: wave, Prev: sunau, Up: Deprecated Python modules functions and methods
1.7.11.14 sys
.............
Deprecated ‘sys.set_coroutine_wrapper()’ and
‘sys.get_coroutine_wrapper()’.
The undocumented ‘sys.callstats()’ function has been deprecated and will
be removed in a future Python version. (Contributed by Victor Stinner
in bpo-28799(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28799
File: python.info, Node: wave, Prev: sys<9>, Up: Deprecated Python modules functions and methods
1.7.11.15 wave
..............
‘wave.openfp()’ has been deprecated and will be removed in Python 3.9.
Use *note wave.open(): 94b. instead. (Contributed by Brian Curtin in
bpo-31985(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31985
File: python.info, Node: Deprecated functions and types of the C API, Next: Platform Support Removals, Prev: Deprecated Python modules functions and methods, Up: What’s New In Python 3 7
1.7.12 Deprecated functions and types of the C API
--------------------------------------------------
Function *note PySlice_GetIndicesEx(): 3f8. is deprecated and replaced
with a macro if ‘Py_LIMITED_API’ is not set or set to a value in the
range between ‘0x03050400’ and ‘0x03060000’ (not inclusive), or is
‘0x03060100’ or higher. (Contributed by Serhiy Storchaka in
bpo-27867(1).)
*note PyOS_AfterFork(): 3f6. has been deprecated. Use *note
PyOS_BeforeFork(): bbf, *note PyOS_AfterFork_Parent(): bc0. or *note
PyOS_AfterFork_Child(): 3f7. instead. (Contributed by Antoine Pitrou in
bpo-16500(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27867
(2) https://bugs.python.org/issue?@action=redirect&bpo=16500
File: python.info, Node: Platform Support Removals, Next: API and Feature Removals<2>, Prev: Deprecated functions and types of the C API, Up: What’s New In Python 3 7
1.7.13 Platform Support Removals
--------------------------------
* FreeBSD 9 and older are no longer officially supported.
* For full Unicode support, including within extension modules, *nix
platforms are now expected to provide at least one of ‘C.UTF-8’
(full locale), ‘C.utf8’ (full locale) or ‘UTF-8’ (‘LC_CTYPE’-only
locale) as an alternative to the legacy ‘ASCII’-based ‘C’ locale.
* OpenSSL 0.9.8 and 1.0.1 are no longer supported, which means
building CPython 3.7 with SSL/TLS support on older platforms still
using these versions requires custom build options that link to a
more recent version of OpenSSL.
Notably, this issue affects the Debian 8 (aka “jessie”) and Ubuntu
14.04 (aka “Trusty”) LTS Linux distributions, as they still use
OpenSSL 1.0.1 by default.
Debian 9 (“stretch”) and Ubuntu 16.04 (“xenial”), as well as recent
releases of other LTS Linux releases (e.g. RHEL/CentOS 7.5, SLES
12-SP3), use OpenSSL 1.0.2 or later, and remain supported in the
default build configuration.
CPython’s own CI configuration file(1) provides an example of using
the SSL compatibility testing infrastructure(2) in CPython’s test
suite to build and link against OpenSSL 1.1.0 rather than an
outdated system provided OpenSSL.
---------- Footnotes ----------
(1) https://github.com/python/cpython/blob/v3.7.13/.travis.yml
(2)
https://github.com/python/cpython/tree/3.13/Tools/ssl/multissltests.py
File: python.info, Node: API and Feature Removals<2>, Next: Module Removals, Prev: Platform Support Removals, Up: What’s New In Python 3 7
1.7.14 API and Feature Removals
-------------------------------
The following features and APIs have been removed from Python 3.7:
* The ‘os.stat_float_times()’ function has been removed. It was
introduced in Python 2.3 for backward compatibility with Python
2.2, and was deprecated since Python 3.1.
* Unknown escapes consisting of ‘'\'’ and an ASCII letter in
replacement templates for *note re.sub(): 2a5. were deprecated in
Python 3.5, and will now cause an error.
* Removed support of the 'exclude' argument in *note
tarfile.TarFile.add(): bf3. It was deprecated in Python 2.7 and
3.2. Use the 'filter' argument instead.
* The ‘ntpath.splitunc()’ function was deprecated in Python 3.1, and
has now been removed. Use *note splitdrive(): bf4. instead.
* *note collections.namedtuple(): 1ca. no longer supports the
'verbose' parameter or ‘_source’ attribute which showed the
generated source code for the named tuple class. This was part of
an optimization designed to speed-up class creation. (Contributed
by Jelle Zijlstra with further improvements by INADA Naoki, Serhiy
Storchaka, and Raymond Hettinger in bpo-28638(1).)
* Functions *note bool(): 463, *note float(): 2f1, *note list(): 60d.
and *note tuple(): 36b. no longer take keyword arguments. The
first argument of *note int(): 259. can now be passed only as
positional argument.
* Removed previously deprecated in Python 2.4 classes ‘Plist’, ‘Dict’
and ‘_InternalDict’ in the *note plistlib: ab. module. Dict values
in the result of functions ‘readPlist()’ and ‘readPlistFromBytes()’
are now normal dicts. You no longer can use attribute access to
access items of these dictionaries.
* The ‘asyncio.windows_utils.socketpair()’ function has been removed.
Use the *note socket.socketpair(): bf5. function instead, it is
available on all platforms since Python 3.5.
‘asyncio.windows_utils.socketpair’ was just an alias to
‘socket.socketpair’ on Python 3.5 and newer.
* *note asyncio: a. no longer exports the *note selectors: c2. and
‘_overlapped’ modules as ‘asyncio.selectors’ and
‘asyncio._overlapped’. Replace ‘from asyncio import selectors’
with ‘import selectors’.
* Direct instantiation of *note ssl.SSLSocket: 8e9. and *note
ssl.SSLObject: b81. objects is now prohibited. The constructors
were never documented, tested, or designed as public constructors.
Users were supposed to use ‘ssl.wrap_socket()’ or *note
ssl.SSLContext: 296. (Contributed by Christian Heimes in
bpo-32951(2).)
* The unused ‘distutils’ ‘install_misc’ command has been removed.
(Contributed by Eric N. Vander Weele in bpo-29218(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28638
(2) https://bugs.python.org/issue?@action=redirect&bpo=32951
(3) https://bugs.python.org/issue?@action=redirect&bpo=29218
File: python.info, Node: Module Removals, Next: Windows-only Changes, Prev: API and Feature Removals<2>, Up: What’s New In Python 3 7
1.7.15 Module Removals
----------------------
The ‘fpectl’ module has been removed. It was never enabled by default,
never worked correctly on x86-64, and it changed the Python ABI in ways
that caused unexpected breakage of C extensions. (Contributed by
Nathaniel J. Smith in bpo-29137(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=29137
File: python.info, Node: Windows-only Changes, Next: Porting to Python 3 7, Prev: Module Removals, Up: What’s New In Python 3 7
1.7.16 Windows-only Changes
---------------------------
The python launcher, (py.exe), can accept 32 & 64 bit specifiers
'without' having to specify a minor version as well. So ‘py -3-32’ and
‘py -3-64’ become valid as well as ‘py -3.7-32’, also the -'m'-64 and
-'m.n'-64 forms are now accepted to force 64 bit python even if 32 bit
would have otherwise been used. If the specified version is not
available py.exe will error exit. (Contributed by Steve Barnes in
bpo-30291(1).)
The launcher can be run as ‘py -0’ to produce a list of the installed
pythons, 'with default marked with an asterisk'. Running ‘py -0p’ will
include the paths. If py is run with a version specifier that cannot be
matched it will also print the 'short form' list of available
specifiers. (Contributed by Steve Barnes in bpo-30362(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30291
(2) https://bugs.python.org/issue?@action=redirect&bpo=30362
File: python.info, Node: Porting to Python 3 7, Next: Notable changes in Python 3 7 1, Prev: Windows-only Changes, Up: What’s New In Python 3 7
1.7.17 Porting to Python 3.7
----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in Python Behavior::
* Changes in the Python API: Changes in the Python API<7>.
* Changes in the C API: Changes in the C API<5>.
* CPython bytecode changes: CPython bytecode changes<6>.
* Windows-only Changes: Windows-only Changes<2>.
* Other CPython implementation changes::
File: python.info, Node: Changes in Python Behavior, Next: Changes in the Python API<7>, Up: Porting to Python 3 7
1.7.17.1 Changes in Python Behavior
...................................
* *note async: 72d. and *note await: 1b9. names are now reserved
keywords. Code using these names as identifiers will now raise a
*note SyntaxError: 18d. (Contributed by Jelle Zijlstra in
bpo-30406(1).)
* PEP 479(2) is enabled for all code in Python 3.7, meaning that
*note StopIteration: bfa. exceptions raised directly or indirectly
in coroutines and generators are transformed into *note
RuntimeError: 195. exceptions. (Contributed by Yury Selivanov in
bpo-32670(3).)
* *note object.__aiter__(): bfb. methods can no longer be declared as
asynchronous. (Contributed by Yury Selivanov in bpo-31709(4).)
* Due to an oversight, earlier Python versions erroneously accepted
the following syntax:
f(1 for x in [1],)
class C(1 for x in [1]):
pass
Python 3.7 now correctly raises a *note SyntaxError: 18d, as a
generator expression always needs to be directly inside a set of
parentheses and cannot have a comma on either side, and the
duplication of the parentheses can be omitted only on calls.
(Contributed by Serhiy Storchaka in bpo-32012(5) and bpo-32023(6).)
* When using the *note -m: 5dd. switch, the initial working directory
is now added to *note sys.path: 3b0, rather than an empty string
(which dynamically denoted the current working directory at the
time of each import). Any programs that are checking for the empty
string, or otherwise relying on the previous behaviour, will need
to be updated accordingly (e.g. by also checking for ‘os.getcwd()’
or ‘os.path.dirname(__main__.__file__)’, depending on why the code
was checking for the empty string in the first place).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=30406
(2) https://peps.python.org/pep-0479/
(3) https://bugs.python.org/issue?@action=redirect&bpo=32670
(4) https://bugs.python.org/issue?@action=redirect&bpo=31709
(5) https://bugs.python.org/issue?@action=redirect&bpo=32012
(6) https://bugs.python.org/issue?@action=redirect&bpo=32023
File: python.info, Node: Changes in the Python API<7>, Next: Changes in the C API<5>, Prev: Changes in Python Behavior, Up: Porting to Python 3 7
1.7.17.2 Changes in the Python API
..................................
* ‘socketserver.ThreadingMixIn.server_close()’ now waits until all
non-daemon threads complete. Set the new *note
socketserver.ThreadingMixIn.block_on_close: bfd. class attribute to
‘False’ to get the pre-3.7 behaviour. (Contributed by Victor
Stinner in bpo-31233(1) and bpo-33540(2).)
* ‘socketserver.ForkingMixIn.server_close()’ now waits until all
child processes complete. Set the new
‘socketserver.ForkingMixIn.block_on_close’ class attribute to
‘False’ to get the pre-3.7 behaviour. (Contributed by Victor
Stinner in bpo-31151(3) and bpo-33540(4).)
* The *note locale.localeconv(): bfe. function now temporarily sets
the ‘LC_CTYPE’ locale to the value of ‘LC_NUMERIC’ in some cases.
(Contributed by Victor Stinner in bpo-31900(5).)
* *note pkgutil.walk_packages(): bff. now raises a *note ValueError:
204. if 'path' is a string. Previously an empty list was returned.
(Contributed by Sanyam Khurana in bpo-24744(6).)
* A format string argument for *note string.Formatter.format(): c00.
is now *note positional-only: a85. Passing it as a keyword
argument was deprecated in Python 3.5. (Contributed by Serhiy
Storchaka in bpo-29193(7).)
* Attributes *note key: c01, *note value: c02. and *note coded_value:
c03. of class *note http.cookies.Morsel: c04. are now read-only.
Assigning to them was deprecated in Python 3.5. Use the *note
set(): c05. method for setting them. (Contributed by Serhiy
Storchaka in bpo-29192(8).)
* The 'mode' argument of *note os.makedirs(): 220. no longer affects
the file permission bits of newly created intermediate-level
directories. To set their file permission bits you can set the
umask before invoking ‘makedirs()’. (Contributed by Serhiy
Storchaka in bpo-19930(9).)
* The *note struct.Struct.format: c06. type is now *note str: 447.
instead of *note bytes: 1c2. (Contributed by Victor Stinner in
bpo-21071(10).)
* ‘cgi.parse_multipart()’ now accepts the 'encoding' and 'errors'
arguments and returns the same results as ‘FieldStorage’: for
non-file fields, the value associated to a key is a list of
strings, not bytes. (Contributed by Pierre Quentel in
bpo-29979(11).)
* Due to internal changes in *note socket: cc, calling *note
socket.fromshare(): c07. on a socket created by *note socket.share:
c08. in older Python versions is not supported.
* ‘repr’ for *note BaseException: 5b7. has changed to not include the
trailing comma. Most exceptions are affected by this change.
(Contributed by Serhiy Storchaka in bpo-30399(12).)
* ‘repr’ for *note datetime.timedelta: 9cf. has changed to include
the keyword arguments in the output. (Contributed by Utkarsh
Upadhyay in bpo-30302(13).)
* Because *note shutil.rmtree(): 2fb. is now implemented using the
*note os.scandir(): a5e. function, the user specified handler
'onerror' is now called with the first argument ‘os.scandir’
instead of ‘os.listdir’ when listing the directory is failed.
* Support for nested sets and set operations in regular expressions
as in Unicode Technical Standard #18(14) might be added in the
future. This would change the syntax. To facilitate this future
change a *note FutureWarning: 411. will be raised in ambiguous
cases for the time being. That include sets starting with a
literal ‘'['’ or containing literal character sequences ‘'--'’,
‘'&&'’, ‘'~~'’, and ‘'||'’. To avoid a warning, escape them with a
backslash. (Contributed by Serhiy Storchaka in bpo-30349(15).)
* The result of splitting a string on a *note regular expression: b9.
that could match an empty string has been changed. For example
splitting on ‘r'\s*'’ will now split not only on whitespaces as it
did previously, but also on empty strings before all non-whitespace
characters and just before the end of the string. The previous
behavior can be restored by changing the pattern to ‘r'\s+'’. A
*note FutureWarning: 411. was emitted for such patterns since
Python 3.5.
For patterns that match both empty and non-empty strings, the
result of searching for all matches may also be changed in other
cases. For example in the string ‘'a\n\n'’, the pattern
‘r'(?m)^\s*?$'’ will not only match empty strings at positions 2
and 3, but also the string ‘'\n'’ at positions 2–3. To match only
blank lines, the pattern should be rewritten as
‘r'(?m)^[^\S\n]*$'’.
*note re.sub(): 2a5. now replaces empty matches adjacent to a
previous non-empty match. For example ‘re.sub('x*', '-', 'abxd')’
returns now ‘'-a-b--d-'’ instead of ‘'-a-b-d-'’ (the first minus
between ‘b’ and ‘d’ replaces ‘x’, and the second minus replaces an
empty string between ‘x’ and ‘d’).
(Contributed by Serhiy Storchaka in bpo-25054(16) and
bpo-32308(17).)
* Change *note re.escape(): c09. to only escape regex special
characters instead of escaping all characters other than ASCII
letters, numbers, and ‘'_'’. (Contributed by Serhiy Storchaka in
bpo-29995(18).)
* *note tracemalloc.Traceback: b96. frames are now sorted from oldest
to most recent to be more consistent with *note traceback: fe.
(Contributed by Jesse Bakker in bpo-32121(19).)
* On OSes that support *note socket.SOCK_NONBLOCK: c0a. or *note
socket.SOCK_CLOEXEC: c0b. bit flags, the *note socket.type: c0c. no
longer has them applied. Therefore, checks like ‘if sock.type ==
socket.SOCK_STREAM’ work as expected on all platforms.
(Contributed by Yury Selivanov in bpo-32331(20).)
* On Windows the default for the 'close_fds' argument of *note
subprocess.Popen: 199. was changed from *note False: b37. to *note
True: c0d. when redirecting the standard handles. If you
previously depended on handles being inherited when using *note
subprocess.Popen: 199. with standard io redirection, you will have
to pass ‘close_fds=False’ to preserve the previous behaviour, or
use *note STARTUPINFO.lpAttributeList: c0e.
* *note importlib.machinery.PathFinder.invalidate_caches(): c0f. –
which implicitly affects *note importlib.invalidate_caches(): c10.
– now deletes entries in *note sys.path_importer_cache: 5e0. which
are set to ‘None’. (Contributed by Brett Cannon in bpo-33169(21).)
* In *note asyncio: a, *note loop.sock_recv(): c11, *note
loop.sock_sendall(): c12, *note loop.sock_accept(): c13, *note
loop.getaddrinfo(): c14, *note loop.getnameinfo(): c15. have been
changed to be proper coroutine methods to match their
documentation. Previously, these methods returned *note
asyncio.Future: bcd. instances. (Contributed by Yury Selivanov in
bpo-32327(22).)
* *note asyncio.Server.sockets: c16. now returns a copy of the
internal list of server sockets, instead of returning it directly.
(Contributed by Yury Selivanov in bpo-32662(23).)
* *note Struct.format: c06. is now a *note str: 447. instance instead
of a *note bytes: 1c2. instance. (Contributed by Victor Stinner in
bpo-21071(24).)
* *note argparse: 6. subparsers can now be made mandatory by passing
‘required=True’ to *note ArgumentParser.add_subparsers(): c17.
(Contributed by Anthony Sottile in bpo-26510(25).)
* *note ast.literal_eval(): c18. is now stricter. Addition and
subtraction of arbitrary numbers are no longer allowed.
(Contributed by Serhiy Storchaka in bpo-31778(26).)
* *note Calendar.itermonthdates: c19. will now consistently raise an
exception when a date falls outside of the ‘0001-01-01’ through
‘9999-12-31’ range. To support applications that cannot tolerate
such exceptions, the new *note Calendar.itermonthdays3: c1a. and
*note Calendar.itermonthdays4: c1b. can be used. The new methods
return tuples and are not restricted by the range supported by
*note datetime.date: 1cd. (Contributed by Alexander Belopolsky in
bpo-28292(27).)
* *note collections.ChainMap: c1c. now preserves the order of the
underlying mappings. (Contributed by Raymond Hettinger in
bpo-32792(28).)
* The ‘submit()’ method of *note
concurrent.futures.ThreadPoolExecutor: 73d. and *note
concurrent.futures.ProcessPoolExecutor: 8ed. now raises a *note
RuntimeError: 195. if called during interpreter shutdown.
(Contributed by Mark Nemec in bpo-33097(29).)
* The *note configparser.ConfigParser: 1c8. constructor now uses
‘read_dict()’ to process the default values, making its behavior
consistent with the rest of the parser. Non-string keys and values
in the defaults dictionary are now being implicitly converted to
strings. (Contributed by James Tocknell in bpo-23835(30).)
* Several undocumented internal imports were removed. One example is
that ‘os.errno’ is no longer available; use ‘import errno’ directly
instead. Note that such undocumented internal imports may be
removed any time without notice, even in micro version releases.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31233
(2) https://bugs.python.org/issue?@action=redirect&bpo=33540
(3) https://bugs.python.org/issue?@action=redirect&bpo=31151
(4) https://bugs.python.org/issue?@action=redirect&bpo=33540
(5) https://bugs.python.org/issue?@action=redirect&bpo=31900
(6) https://bugs.python.org/issue?@action=redirect&bpo=24744
(7) https://bugs.python.org/issue?@action=redirect&bpo=29193
(8) https://bugs.python.org/issue?@action=redirect&bpo=29192
(9) https://bugs.python.org/issue?@action=redirect&bpo=19930
(10) https://bugs.python.org/issue?@action=redirect&bpo=21071
(11) https://bugs.python.org/issue?@action=redirect&bpo=29979
(12) https://bugs.python.org/issue?@action=redirect&bpo=30399
(13) https://bugs.python.org/issue?@action=redirect&bpo=30302
(14) https://unicode.org/reports/tr18/
(15) https://bugs.python.org/issue?@action=redirect&bpo=30349
(16) https://bugs.python.org/issue?@action=redirect&bpo=25054
(17) https://bugs.python.org/issue?@action=redirect&bpo=32308
(18) https://bugs.python.org/issue?@action=redirect&bpo=29995
(19) https://bugs.python.org/issue?@action=redirect&bpo=32121
(20) https://bugs.python.org/issue?@action=redirect&bpo=32331
(21) https://bugs.python.org/issue?@action=redirect&bpo=33169
(22) https://bugs.python.org/issue?@action=redirect&bpo=32327
(23) https://bugs.python.org/issue?@action=redirect&bpo=32662
(24) https://bugs.python.org/issue?@action=redirect&bpo=21071
(25) https://bugs.python.org/issue?@action=redirect&bpo=26510
(26) https://bugs.python.org/issue?@action=redirect&bpo=31778
(27) https://bugs.python.org/issue?@action=redirect&bpo=28292
(28) https://bugs.python.org/issue?@action=redirect&bpo=32792
(29) https://bugs.python.org/issue?@action=redirect&bpo=33097
(30) https://bugs.python.org/issue?@action=redirect&bpo=23835
File: python.info, Node: Changes in the C API<5>, Next: CPython bytecode changes<6>, Prev: Changes in the Python API<7>, Up: Porting to Python 3 7
1.7.17.3 Changes in the C API
.............................
The function *note PySlice_GetIndicesEx(): 3f8. is considered unsafe for
resizable sequences. If the slice indices are not instances of *note
int: 259, but objects that implement the ‘__index__()’ method, the
sequence can be resized after passing its length to
‘PySlice_GetIndicesEx()’. This can lead to returning indices out of the
length of the sequence. For avoiding possible problems use new
functions *note PySlice_Unpack(): 3f9. and *note
PySlice_AdjustIndices(): 3fa. (Contributed by Serhiy Storchaka in
bpo-27867(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27867
File: python.info, Node: CPython bytecode changes<6>, Next: Windows-only Changes<2>, Prev: Changes in the C API<5>, Up: Porting to Python 3 7
1.7.17.4 CPython bytecode changes
.................................
There are two new opcodes: *note LOAD_METHOD: c1f. and ‘CALL_METHOD’.
(Contributed by Yury Selivanov and INADA Naoki in bpo-26110(1).)
The ‘STORE_ANNOTATION’ opcode has been removed. (Contributed by Mark
Shannon in bpo-32550(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26110
(2) https://bugs.python.org/issue?@action=redirect&bpo=32550
File: python.info, Node: Windows-only Changes<2>, Next: Other CPython implementation changes, Prev: CPython bytecode changes<6>, Up: Porting to Python 3 7
1.7.17.5 Windows-only Changes
.............................
The file used to override *note sys.path: 3b0. is now called
‘._pth’ instead of ‘'sys.path'’. See *note Finding
modules: c21. for more information. (Contributed by Steve Dower in
bpo-28137(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28137
File: python.info, Node: Other CPython implementation changes, Prev: Windows-only Changes<2>, Up: Porting to Python 3 7
1.7.17.6 Other CPython implementation changes
.............................................
In preparation for potential future changes to the public CPython
runtime initialization API (see PEP 432(1) for an initial, but somewhat
outdated, draft), CPython’s internal startup and configuration
management logic has been significantly refactored. While these updates
are intended to be entirely transparent to both embedding applications
and users of the regular CPython CLI, they’re being mentioned here as
the refactoring changes the internal order of various operations during
interpreter startup, and hence may uncover previously latent defects,
either in embedding applications, or in CPython itself. (Initially
contributed by Nick Coghlan and Eric Snow as part of bpo-22257(2), and
further updated by Nick, Eric, and Victor Stinner in a number of other
issues). Some known details affected:
* ‘PySys_AddWarnOptionUnicode()’ is not currently usable by embedding
applications due to the requirement to create a Unicode object
prior to calling ‘Py_Initialize’. Use ‘PySys_AddWarnOption()’
instead.
* warnings filters added by an embedding application with
‘PySys_AddWarnOption()’ should now more consistently take
precedence over the default filters set by the interpreter
Due to changes in the way the default warnings filters are configured,
setting *note Py_BytesWarningFlag: 3d0. to a value greater than one is
no longer sufficient to both emit *note BytesWarning: c23. messages and
have them converted to exceptions. Instead, the flag must be set (to
cause the warnings to be emitted in the first place), and an explicit
‘error::BytesWarning’ warnings filter added to convert them to
exceptions.
Due to a change in the way docstrings are handled by the compiler, the
implicit ‘return None’ in a function body consisting solely of a
docstring is now marked as occurring on the same line as the docstring,
not on the function’s header line.
The current exception state has been moved from the frame object to the
co-routine. This simplified the interpreter and fixed a couple of
obscure bugs caused by having swap exception state when entering or
exiting a generator. (Contributed by Mark Shannon in bpo-25612(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0432/
(2) https://bugs.python.org/issue?@action=redirect&bpo=22257
(3) https://bugs.python.org/issue?@action=redirect&bpo=25612
File: python.info, Node: Notable changes in Python 3 7 1, Next: Notable changes in Python 3 7 2, Prev: Porting to Python 3 7, Up: What’s New In Python 3 7
1.7.18 Notable changes in Python 3.7.1
--------------------------------------
Starting in 3.7.1, *note Py_Initialize(): 8ab. now consistently reads
and respects all of the same environment settings as *note Py_Main():
c25. (in earlier Python versions, it respected an ill-defined subset of
those environment variables, while in Python 3.7.0 it didn’t read any of
them due to bpo-34247(1)). If this behavior is unwanted, set *note
Py_IgnoreEnvironmentFlag: 3d4. to 1 before calling *note
Py_Initialize(): 8ab.
In 3.7.1 the C API for Context Variables *note was updated: c26. to use
*note PyObject: 334. pointers. See also bpo-34762(2).
In 3.7.1 the *note tokenize: fb. module now implicitly emits a ‘NEWLINE’
token when provided with input that does not have a trailing new line.
This behavior now matches what the C tokenizer does internally.
(Contributed by Ammar Askar in bpo-33899(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=34247
(2) https://bugs.python.org/issue?@action=redirect&bpo=34762
(3) https://bugs.python.org/issue?@action=redirect&bpo=33899
File: python.info, Node: Notable changes in Python 3 7 2, Next: Notable changes in Python 3 7 6, Prev: Notable changes in Python 3 7 1, Up: What’s New In Python 3 7
1.7.19 Notable changes in Python 3.7.2
--------------------------------------
In 3.7.2, *note venv: 111. on Windows no longer copies the original
binaries, but creates redirector scripts named ‘python.exe’ and
‘pythonw.exe’ instead. This resolves a long standing issue where all
virtual environments would have to be upgraded or recreated with each
Python update. However, note that this release will still require
recreation of virtual environments in order to get the new scripts.
File: python.info, Node: Notable changes in Python 3 7 6, Next: Notable changes in Python 3 7 10, Prev: Notable changes in Python 3 7 2, Up: What’s New In Python 3 7
1.7.20 Notable changes in Python 3.7.6
--------------------------------------
Due to significant security concerns, the 'reuse_address' parameter of
*note asyncio.loop.create_datagram_endpoint(): 72e. is no longer
supported. This is because of the behavior of the socket option
‘SO_REUSEADDR’ in UDP. For more details, see the documentation for
‘loop.create_datagram_endpoint()’. (Contributed by Kyle Stanley,
Antoine Pitrou, and Yury Selivanov in bpo-37228(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37228
File: python.info, Node: Notable changes in Python 3 7 10, Next: Notable changes in Python 3 7 11, Prev: Notable changes in Python 3 7 6, Up: What’s New In Python 3 7
1.7.21 Notable changes in Python 3.7.10
---------------------------------------
Earlier Python versions allowed using both ‘;’ and ‘&’ as query
parameter separators in *note urllib.parse.parse_qs(): 27c. and *note
urllib.parse.parse_qsl(): 27b. Due to security concerns, and to conform
with newer W3C recommendations, this has been changed to allow only a
single separator key, with ‘&’ as the default. This change also affects
‘cgi.parse()’ and ‘cgi.parse_multipart()’ as they use the affected
functions internally. For more details, please see their respective
documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and
Ken Jin in bpo-42967(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42967
File: python.info, Node: Notable changes in Python 3 7 11, Next: Notable security feature in 3 7 14, Prev: Notable changes in Python 3 7 10, Up: What’s New In Python 3 7
1.7.22 Notable changes in Python 3.7.11
---------------------------------------
A security fix alters the *note ftplib.FTP: 8f9. behavior to not trust
the IPv4 address sent from the remote server when setting up a passive
data channel. We reuse the ftp server IP address instead. For unusual
code requiring the old behavior, set a ‘trust_server_pasv_ipv4_address’
attribute on your FTP instance to ‘True’. (See gh-87451(1))
The presence of newline or tab characters in parts of a URL allows for
some forms of attacks. Following the WHATWG specification that updates
RFC 3986, ASCII newline ‘\n’, ‘\r’ and tab ‘\t’ characters are stripped
from the URL by the parser *note urllib.parse(): 10a. preventing such
attacks. The removal characters are controlled by a new module level
variable ‘urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE’. (See gh-88048(2))
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/87451
(2) https://github.com/python/cpython/issues/88048
File: python.info, Node: Notable security feature in 3 7 14, Prev: Notable changes in Python 3 7 11, Up: What’s New In Python 3 7
1.7.23 Notable security feature in 3.7.14
-----------------------------------------
Converting between *note int: 259. and *note str: 447. in bases other
than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10
(decimal) now raises a *note ValueError: 204. if the number of digits in
string form is above a limit to avoid potential denial of service
attacks due to the algorithmic complexity. This is a mitigation for CVE
2020-10735(1). This limit can be configured or disabled by environment
variable, command line flag, or *note sys: d9. APIs. See the *note
integer string conversion length limitation: 5f1. documentation. The
default limit is 4300 digits in string form.
---------- Footnotes ----------
(1) https://www.cve.org/CVERecord?id=CVE-2020-10735
File: python.info, Node: What’s New In Python 3 6, Next: What’s New In Python 3 5, Prev: What’s New In Python 3 7, Up: What’s New in Python
1.8 What’s New In Python 3.6
============================
Editors: Elvis Pranskevichus <>, Yury Selivanov
<>
This article explains the new features in Python 3.6, compared to 3.5.
Python 3.6 was released on December 23, 2016. See the changelog(1) for
a full list of changes.
See also
........
PEP 494(2) - Python 3.6 Release Schedule
* Menu:
* Summary – Release highlights: Summary – Release highlights<6>.
* New Features: New Features<13>.
* Other Language Changes: Other Language Changes<8>.
* New Modules: New Modules<8>.
* Improved Modules: Improved Modules<8>.
* Optimizations: Optimizations<8>.
* Build and C API Changes: Build and C API Changes<2>.
* Other Improvements::
* Deprecated: Deprecated<9>.
* Removed: Removed<9>.
* Porting to Python 3.6: Porting to Python 3 6.
* Notable changes in Python 3.6.2: Notable changes in Python 3 6 2.
* Notable changes in Python 3.6.4: Notable changes in Python 3 6 4.
* Notable changes in Python 3.6.5: Notable changes in Python 3 6 5.
* Notable changes in Python 3.6.7: Notable changes in Python 3 6 7.
* Notable changes in Python 3.6.10: Notable changes in Python 3 6 10.
* Notable changes in Python 3.6.13: Notable changes in Python 3 6 13.
* Notable changes in Python 3.6.14: Notable changes in Python 3 6 14.
---------- Footnotes ----------
(1) https://docs.python.org/3.6/whatsnew/changelog.html
(2) https://peps.python.org/pep-0494/
File: python.info, Node: Summary – Release highlights<6>, Next: New Features<13>, Up: What’s New In Python 3 6
1.8.1 Summary – Release highlights
----------------------------------
New syntax features:
* *note PEP 498: c2f, formatted string literals.
* *note PEP 515: c30, underscores in numeric literals.
* *note PEP 526: c31, syntax for variable annotations.
* *note PEP 525: c32, asynchronous generators.
* *note PEP 530: c33.: asynchronous comprehensions.
New library modules:
* *note secrets: c0.: *note PEP 506 – Adding A Secrets Module To The
Standard Library: c34.
CPython implementation improvements:
* The *note dict: ac6. type has been reimplemented to use a *note
more compact representation: c35. based on a proposal by Raymond
Hettinger(1) and similar to the PyPy dict implementation(2). This
resulted in dictionaries using 20% to 25% less memory when compared
to Python 3.5.
* Customization of class creation has been simplified with the *note
new protocol: c36.
* The class attribute definition order is *note now preserved: c37.
* The order of elements in ‘**kwargs’ now *note corresponds to the
order: c38. in which keyword arguments were passed to the function.
* DTrace and SystemTap *note probing support: c39. has been added.
* The new *note PYTHONMALLOC: c3a. environment variable can now be
used to debug the interpreter memory allocation and access errors.
Significant improvements in the standard library:
* The *note asyncio: a. module has received new features, significant
usability and performance improvements, and a fair amount of bug
fixes. Starting with Python 3.6 the ‘asyncio’ module is no longer
provisional and its API is considered stable.
* A new *note file system path protocol: c3b. has been implemented to
support *note path-like objects: 2a2. All standard library
functions operating on paths have been updated to work with the new
protocol.
* The *note datetime: 30. module has gained support for *note Local
Time Disambiguation: c3c.
* The *note typing: 104. module received a number of *note
improvements: c3d.
* The *note tracemalloc: ff. module has been significantly reworked
and is now used to provide better output for *note ResourceWarning:
246. as well as provide better diagnostics for memory allocation
errors. See the *note PYTHONMALLOC section: c3a. for more
information.
Security improvements:
* The new *note secrets: c0. module has been added to simplify the
generation of cryptographically strong pseudo-random numbers
suitable for managing secrets such as account authentication,
tokens, and similar.
* On Linux, *note os.urandom(): 51e. now blocks until the system
urandom entropy pool is initialized to increase the security. See
the PEP 524(3) for the rationale.
* The *note hashlib: 68. and *note ssl: d0. modules now support
OpenSSL 1.1.0.
* The default settings and feature set of the *note ssl: d0. module
have been improved.
* The *note hashlib: 68. module received support for the BLAKE2,
SHA-3 and SHAKE hash algorithms and the *note scrypt(): c3e. key
derivation function.
Windows improvements:
* *note PEP 528: c3f. and *note PEP 529: c40, Windows filesystem and
console encoding changed to UTF-8.
* The ‘py.exe’ launcher, when used interactively, no longer prefers
Python 2 over Python 3 when the user doesn’t specify a version (via
command line arguments or a config file). Handling of shebang
lines remains unchanged - “python” refers to Python 2 in that case.
* ‘python.exe’ and ‘pythonw.exe’ have been marked as long-path aware,
which means that the 260 character path limit may no longer apply.
See *note removing the MAX_PATH limitation: c41. for details.
* A ‘._pth’ file can be added to force isolated mode and fully
specify all search paths to avoid registry and environment lookup.
See *note the documentation: c21. for more information.
* A ‘python36.zip’ file now works as a landmark to infer *note
PYTHONHOME: 3b8. See *note the documentation: c21. for more
information.
---------- Footnotes ----------
(1)
https://mail.python.org/pipermail/python-dev/2012-December/123028.html
(2)
https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html
(3) https://peps.python.org/pep-0524/
File: python.info, Node: New Features<13>, Next: Other Language Changes<8>, Prev: Summary – Release highlights<6>, Up: What’s New In Python 3 6
1.8.2 New Features
------------------
* Menu:
* PEP 498; Formatted string literals: PEP 498 Formatted string literals.
* PEP 526; Syntax for variable annotations: PEP 526 Syntax for variable annotations.
* PEP 515; Underscores in Numeric Literals: PEP 515 Underscores in Numeric Literals.
* PEP 525; Asynchronous Generators: PEP 525 Asynchronous Generators.
* PEP 530; Asynchronous Comprehensions: PEP 530 Asynchronous Comprehensions.
* PEP 487; Simpler customization of class creation: PEP 487 Simpler customization of class creation.
* PEP 487; Descriptor Protocol Enhancements: PEP 487 Descriptor Protocol Enhancements.
* PEP 519; Adding a file system path protocol: PEP 519 Adding a file system path protocol.
* PEP 495; Local Time Disambiguation: PEP 495 Local Time Disambiguation.
* PEP 529; Change Windows filesystem encoding to UTF-8: PEP 529 Change Windows filesystem encoding to UTF-8.
* PEP 528; Change Windows console encoding to UTF-8: PEP 528 Change Windows console encoding to UTF-8.
* PEP 520; Preserving Class Attribute Definition Order: PEP 520 Preserving Class Attribute Definition Order.
* PEP 468; Preserving Keyword Argument Order: PEP 468 Preserving Keyword Argument Order.
* New dict implementation::
* PEP 523; Adding a frame evaluation API to CPython: PEP 523 Adding a frame evaluation API to CPython.
* PYTHONMALLOC environment variable::
* DTrace and SystemTap probing support::
File: python.info, Node: PEP 498 Formatted string literals, Next: PEP 526 Syntax for variable annotations, Up: New Features<13>
1.8.2.1 PEP 498: Formatted string literals
..........................................
PEP 498(1) introduces a new kind of string literals: 'f-strings', or
*note formatted string literals: 9a9.
Formatted string literals are prefixed with ‘'f'’ and are similar to the
format strings accepted by *note str.format(): 61d. They contain
replacement fields surrounded by curly braces. The replacement fields
are expressions, which are evaluated at run time, and then formatted
using the *note format(): 61b. protocol:
>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'
>>> width = 10
>>> precision = 4
>>> value = decimal.Decimal("12.34567")
>>> f"result: {value:{width}.{precision}}" # nested fields
'result: 12.35'
See also
........
PEP 498(2) – Literal String Interpolation.
PEP written and implemented by Eric V. Smith.
*note Feature documentation: 9a9.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0498/
(2) https://peps.python.org/pep-0498/
File: python.info, Node: PEP 526 Syntax for variable annotations, Next: PEP 515 Underscores in Numeric Literals, Prev: PEP 498 Formatted string literals, Up: New Features<13>
1.8.2.2 PEP 526: Syntax for variable annotations
................................................
PEP 484(1) introduced the standard for type annotations of function
parameters, a.k.a. type hints. This PEP adds syntax to Python for
annotating the types of variables including class variables and instance
variables:
primes: List[int] = []
captain: str # Note: no initial value!
class Starship:
stats: Dict[str, int] = {}
Just as for function annotations, the Python interpreter does not attach
any particular meaning to variable annotations and only stores them in
the ‘__annotations__’ attribute of a class or module.
In contrast to variable declarations in statically typed languages, the
goal of annotation syntax is to provide an easy way to specify
structured type metadata for third party tools and libraries via the
abstract syntax tree and the ‘__annotations__’ attribute.
See also
........
PEP 526(2) – Syntax for variable annotations.
PEP written by Ryan Gonzalez, Philip House, Ivan Levkivskyi, Lisa
Roach, and Guido van Rossum. Implemented by Ivan Levkivskyi.
Tools that use or will use the new syntax: mypy(3), pytype(4), PyCharm,
etc.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0526/
(3) https://www.mypy-lang.org/
(4) https://github.com/google/pytype
File: python.info, Node: PEP 515 Underscores in Numeric Literals, Next: PEP 525 Asynchronous Generators, Prev: PEP 526 Syntax for variable annotations, Up: New Features<13>
1.8.2.3 PEP 515: Underscores in Numeric Literals
................................................
PEP 515(1) adds the ability to use underscores in numeric literals for
improved readability. For example:
>>> 1_000_000_000_000_000
1000000000000000
>>> 0x_FF_FF_FF_FF
4294967295
Single underscores are allowed between digits and after any base
specifier. Leading, trailing, or multiple underscores in a row are not
allowed.
The *note string formatting: 1ea. language also now has support for the
‘'_'’ option to signal the use of an underscore for a thousands
separator for floating-point presentation types and for integer
presentation type ‘'d'’. For integer presentation types ‘'b'’, ‘'o'’,
‘'x'’, and ‘'X'’, underscores will be inserted every 4 digits:
>>> '{:_}'.format(1000000)
'1_000_000'
>>> '{:_x}'.format(0xFFFFFFFF)
'ffff_ffff'
See also
........
PEP 515(2) – Underscores in Numeric Literals
PEP written by Georg Brandl and Serhiy Storchaka.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0515/
(2) https://peps.python.org/pep-0515/
File: python.info, Node: PEP 525 Asynchronous Generators, Next: PEP 530 Asynchronous Comprehensions, Prev: PEP 515 Underscores in Numeric Literals, Up: New Features<13>
1.8.2.4 PEP 525: Asynchronous Generators
........................................
PEP 492(1) introduced support for native coroutines and ‘async’ /
‘await’ syntax to Python 3.5. A notable limitation of the Python 3.5
implementation is that it was not possible to use ‘await’ and ‘yield’ in
the same function body. In Python 3.6 this restriction has been lifted,
making it possible to define 'asynchronous generators':
async def ticker(delay, to):
"""Yield numbers from 0 to *to* every *delay* seconds."""
for i in range(to):
yield i
await asyncio.sleep(delay)
The new syntax allows for faster and more concise code.
See also
........
PEP 525(2) – Asynchronous Generators
PEP written and implemented by Yury Selivanov.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0492/
(2) https://peps.python.org/pep-0525/
File: python.info, Node: PEP 530 Asynchronous Comprehensions, Next: PEP 487 Simpler customization of class creation, Prev: PEP 525 Asynchronous Generators, Up: New Features<13>
1.8.2.5 PEP 530: Asynchronous Comprehensions
............................................
PEP 530(1) adds support for using ‘async for’ in list, set, dict
comprehensions and generator expressions:
result = [i async for i in aiter() if i % 2]
Additionally, ‘await’ expressions are supported in all kinds of
comprehensions:
result = [await fun() for fun in funcs if await condition()]
See also
........
PEP 530(2) – Asynchronous Comprehensions
PEP written and implemented by Yury Selivanov.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0530/
(2) https://peps.python.org/pep-0530/
File: python.info, Node: PEP 487 Simpler customization of class creation, Next: PEP 487 Descriptor Protocol Enhancements, Prev: PEP 530 Asynchronous Comprehensions, Up: New Features<13>
1.8.2.6 PEP 487: Simpler customization of class creation
........................................................
It is now possible to customize subclass creation without using a
metaclass. The new ‘__init_subclass__’ classmethod will be called on
the base class whenever a new subclass is created:
class PluginBase:
subclasses = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
class Plugin1(PluginBase):
pass
class Plugin2(PluginBase):
pass
In order to allow zero-argument *note super(): 4d7. calls to work
correctly from *note __init_subclass__(): 62d. implementations, custom
metaclasses must ensure that the new ‘__classcell__’ namespace entry is
propagated to ‘type.__new__’ (as described in *note Creating the class
object: c4a.).
See also
........
PEP 487(1) – Simpler customization of class creation
PEP written and implemented by Martin Teichmann.
*note Feature documentation: c4b.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0487/
File: python.info, Node: PEP 487 Descriptor Protocol Enhancements, Next: PEP 519 Adding a file system path protocol, Prev: PEP 487 Simpler customization of class creation, Up: New Features<13>
1.8.2.7 PEP 487: Descriptor Protocol Enhancements
.................................................
PEP 487(1) extends the descriptor protocol to include the new optional
*note __set_name__(): c4e. method. Whenever a new class is defined, the
new method will be called on all descriptors included in the definition,
providing them with a reference to the class being defined and the name
given to the descriptor within the class namespace. In other words,
instances of descriptors can now know the attribute name of the
descriptor in the owner class:
class IntField:
def __get__(self, instance, owner):
return instance.__dict__[self.name]
def __set__(self, instance, value):
if not isinstance(value, int):
raise ValueError(f'expecting integer in {self.name}')
instance.__dict__[self.name] = value
# this is the new initializer:
def __set_name__(self, owner, name):
self.name = name
class Model:
int_field = IntField()
See also
........
PEP 487(2) – Simpler customization of class creation
PEP written and implemented by Martin Teichmann.
*note Feature documentation: c4f.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0487/
(2) https://peps.python.org/pep-0487/
File: python.info, Node: PEP 519 Adding a file system path protocol, Next: PEP 495 Local Time Disambiguation, Prev: PEP 487 Descriptor Protocol Enhancements, Up: New Features<13>
1.8.2.8 PEP 519: Adding a file system path protocol
...................................................
File system paths have historically been represented as *note str: 447.
or *note bytes: 1c2. objects. This has led to people who write code
which operate on file system paths to assume that such objects are only
one of those two types (an *note int: 259. representing a file
descriptor does not count as that is not a file path). Unfortunately
that assumption prevents alternative object representations of file
system paths like *note pathlib: a4. from working with pre-existing
code, including Python’s standard library.
To fix this situation, a new interface represented by *note os.PathLike:
c51. has been defined. By implementing the *note __fspath__(): c52.
method, an object signals that it represents a path. An object can then
provide a low-level representation of a file system path as a *note str:
447. or *note bytes: 1c2. object. This means an object is considered
*note path-like: 2a2. if it implements *note os.PathLike: c51. or is a
*note str: 447. or *note bytes: 1c2. object which represents a file
system path. Code can use *note os.fspath(): c53, *note os.fsdecode():
c54, or *note os.fsencode(): c55. to explicitly get a *note str: 447.
and/or *note bytes: 1c2. representation of a path-like object.
The built-in *note open(): 517. function has been updated to accept
*note os.PathLike: c51. objects, as have all relevant functions in the
*note os: a1. and *note os.path: a2. modules, and most other functions
and classes in the standard library. The *note os.DirEntry: 497. class
and relevant classes in *note pathlib: a4. have also been updated to
implement *note os.PathLike: c51.
The hope is that updating the fundamental functions for operating on
file system paths will lead to third-party code to implicitly support
all *note path-like objects: 2a2. without any code changes, or at least
very minimal ones (e.g. calling *note os.fspath(): c53. at the
beginning of code before operating on a path-like object).
Here are some examples of how the new interface allows for *note
pathlib.Path: 22b. to be used more easily and transparently with
pre-existing code:
>>> import pathlib
>>> with open(pathlib.Path("README")) as f:
... contents = f.read()
...
>>> import os.path
>>> os.path.splitext(pathlib.Path("some_file.txt"))
('some_file', '.txt')
>>> os.path.join("/a/b", pathlib.Path("c"))
'/a/b/c'
>>> import os
>>> os.fspath(pathlib.Path("some_file.txt"))
'some_file.txt'
(Implemented by Brett Cannon, Ethan Furman, Dusty Phillips, and Jelle
Zijlstra.)
See also
........
PEP 519(1) – Adding a file system path protocol
PEP written by Brett Cannon and Koos Zevenhoven.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0519/
File: python.info, Node: PEP 495 Local Time Disambiguation, Next: PEP 529 Change Windows filesystem encoding to UTF-8, Prev: PEP 519 Adding a file system path protocol, Up: New Features<13>
1.8.2.9 PEP 495: Local Time Disambiguation
..........................................
In most world locations, there have been and will be times when local
clocks are moved back. In those times, intervals are introduced in
which local clocks show the same time twice in the same day. In these
situations, the information displayed on a local clock (or stored in a
Python datetime instance) is insufficient to identify a particular
moment in time.
PEP 495(1) adds the new 'fold' attribute to instances of *note
datetime.datetime: 1cc. and *note datetime.time: 1ce. classes to
differentiate between two moments in time for which local times are the
same:
>>> u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc)
>>> for i in range(4):
... u = u0 + i*HOUR
... t = u.astimezone(Eastern)
... print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold)
...
04:00:00 UTC = 00:00:00 EDT 0
05:00:00 UTC = 01:00:00 EDT 0
06:00:00 UTC = 01:00:00 EST 1
07:00:00 UTC = 02:00:00 EST 0
The values of the *note fold: c57. attribute have the value ‘0’ for all
instances except those that represent the second (chronologically)
moment in time in an ambiguous case.
See also
........
PEP 495(2) – Local Time Disambiguation
PEP written by Alexander Belopolsky and Tim Peters, implementation
by Alexander Belopolsky.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0495/
(2) https://peps.python.org/pep-0495/
File: python.info, Node: PEP 529 Change Windows filesystem encoding to UTF-8, Next: PEP 528 Change Windows console encoding to UTF-8, Prev: PEP 495 Local Time Disambiguation, Up: New Features<13>
1.8.2.10 PEP 529: Change Windows filesystem encoding to UTF-8
.............................................................
Representing filesystem paths is best performed with str (Unicode)
rather than bytes. However, there are some situations where using bytes
is sufficient and correct.
Prior to Python 3.6, data loss could result when using bytes paths on
Windows. With this change, using bytes to represent paths is now
supported on Windows, provided those bytes are encoded with the encoding
returned by *note sys.getfilesystemencoding(): c59, which now defaults
to ‘'utf-8'’.
Applications that do not use str to represent paths should use *note
os.fsencode(): c55. and *note os.fsdecode(): c54. to ensure their bytes
are correctly encoded. To revert to the previous behaviour, set *note
PYTHONLEGACYWINDOWSFSENCODING: 2b1. or call *note
sys._enablelegacywindowsfsencoding(): 2b0.
See PEP 529(1) for more information and discussion of code modifications
that may be required.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0529/
File: python.info, Node: PEP 528 Change Windows console encoding to UTF-8, Next: PEP 520 Preserving Class Attribute Definition Order, Prev: PEP 529 Change Windows filesystem encoding to UTF-8, Up: New Features<13>
1.8.2.11 PEP 528: Change Windows console encoding to UTF-8
..........................................................
The default console on Windows will now accept all Unicode characters
and provide correctly read str objects to Python code. ‘sys.stdin’,
‘sys.stdout’ and ‘sys.stderr’ now default to utf-8 encoding.
This change only applies when using an interactive console, and not when
redirecting files or pipes. To revert to the previous behaviour for
interactive console use, set *note PYTHONLEGACYWINDOWSSTDIO: c5b.
See also
........
PEP 528(1) – Change Windows console encoding to UTF-8
PEP written and implemented by Steve Dower.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0528/
File: python.info, Node: PEP 520 Preserving Class Attribute Definition Order, Next: PEP 468 Preserving Keyword Argument Order, Prev: PEP 528 Change Windows console encoding to UTF-8, Up: New Features<13>
1.8.2.12 PEP 520: Preserving Class Attribute Definition Order
.............................................................
Attributes in a class definition body have a natural ordering: the same
order in which the names appear in the source. This order is now
preserved in the new class’s *note __dict__: c5d. attribute.
Also, the effective default class 'execution' namespace (returned from
*note type.__prepare__(): c5e.) is now an insertion-order-preserving
mapping.
See also
........
PEP 520(1) – Preserving Class Attribute Definition Order
PEP written and implemented by Eric Snow.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0520/
File: python.info, Node: PEP 468 Preserving Keyword Argument Order, Next: New dict implementation, Prev: PEP 520 Preserving Class Attribute Definition Order, Up: New Features<13>
1.8.2.13 PEP 468: Preserving Keyword Argument Order
...................................................
‘**kwargs’ in a function signature is now guaranteed to be an
insertion-order-preserving mapping.
See also
........
PEP 468(1) – Preserving Keyword Argument Order
PEP written and implemented by Eric Snow.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0468/
File: python.info, Node: New dict implementation, Next: PEP 523 Adding a frame evaluation API to CPython, Prev: PEP 468 Preserving Keyword Argument Order, Up: New Features<13>
1.8.2.14 New dict implementation
................................
The *note dict: ac6. type now uses a “compact” representation based on a
proposal by Raymond Hettinger(1) which was first implemented by PyPy(2).
The memory usage of the new *note dict(): 258. is between 20% and 25%
smaller compared to Python 3.5.
The order-preserving aspect of this new implementation is considered an
implementation detail and should not be relied upon (this may change in
the future, but it is desired to have this new dict implementation in
the language for a few releases before changing the language spec to
mandate order-preserving semantics for all current and future Python
implementations; this also helps preserve backwards-compatibility with
older versions of the language where random iteration order is still in
effect, e.g. Python 3.5).
(Contributed by INADA Naoki in bpo-27350(3). Idea originally suggested
by Raymond Hettinger(4).)
---------- Footnotes ----------
(1)
https://mail.python.org/pipermail/python-dev/2012-December/123028.html
(2)
https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html
(3) https://bugs.python.org/issue?@action=redirect&bpo=27350
(4)
https://mail.python.org/pipermail/python-dev/2012-December/123028.html
File: python.info, Node: PEP 523 Adding a frame evaluation API to CPython, Next: PYTHONMALLOC environment variable, Prev: New dict implementation, Up: New Features<13>
1.8.2.15 PEP 523: Adding a frame evaluation API to CPython
..........................................................
While Python provides extensive support to customize how code executes,
one place it has not done so is in the evaluation of frame objects. If
you wanted some way to intercept frame evaluation in Python there really
wasn’t any way without directly manipulating function pointers for
defined functions.
PEP 523(1) changes this by providing an API to make frame evaluation
pluggable at the C level. This will allow for tools such as debuggers
and JITs to intercept frame evaluation before the execution of Python
code begins. This enables the use of alternative evaluation
implementations for Python code, tracking frame evaluation, etc.
This API is not part of the limited C API and is marked as private to
signal that usage of this API is expected to be limited and only
applicable to very select, low-level use-cases. Semantics of the API
will change with Python as necessary.
See also
........
PEP 523(2) – Adding a frame evaluation API to CPython
PEP written by Brett Cannon and Dino Viehland.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0523/
(2) https://peps.python.org/pep-0523/
File: python.info, Node: PYTHONMALLOC environment variable, Next: DTrace and SystemTap probing support, Prev: PEP 523 Adding a frame evaluation API to CPython, Up: New Features<13>
1.8.2.16 PYTHONMALLOC environment variable
..........................................
The new *note PYTHONMALLOC: c64. environment variable allows setting the
Python memory allocators and installing debug hooks.
It is now possible to install debug hooks on Python memory allocators on
Python compiled in release mode using ‘PYTHONMALLOC=debug’. Effects of
debug hooks:
* Newly allocated memory is filled with the byte ‘0xCB’
* Freed memory is filled with the byte ‘0xDB’
* Detect violations of the Python memory allocator API. For example,
*note PyObject_Free(): c65. called on a memory block allocated by
*note PyMem_Malloc(): c66.
* Detect writes before the start of a buffer (buffer underflows)
* Detect writes after the end of a buffer (buffer overflows)
* Check that the *note GIL: 148. is held when allocator functions of
*note PYMEM_DOMAIN_OBJ: c67. (ex: *note PyObject_Malloc(): c68.)
and *note PYMEM_DOMAIN_MEM: c69. (ex: *note PyMem_Malloc(): c66.)
domains are called.
Checking if the GIL is held is also a new feature of Python 3.6.
See the *note PyMem_SetupDebugHooks(): c6a. function for debug hooks on
Python memory allocators.
It is now also possible to force the usage of the ‘malloc()’ allocator
of the C library for all Python memory allocations using
‘PYTHONMALLOC=malloc’. This is helpful when using external memory
debuggers like Valgrind on a Python compiled in release mode.
On error, the debug hooks on Python memory allocators now use the *note
tracemalloc: ff. module to get the traceback where a memory block was
allocated.
Example of fatal error on buffer overflow using ‘python3.6 -X
tracemalloc=5’ (store 5 frames in traces):
Debug memory block at address p=0x7fbcd41666f8: API 'o'
4 bytes originally requested
The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x02 *** OUCH
at tail+1: 0xfb
at tail+2: 0xfb
at tail+3: 0xfb
at tail+4: 0xfb
at tail+5: 0xfb
at tail+6: 0xfb
at tail+7: 0xfb
The block was made by call #1233329 to debug malloc/realloc.
Data at p: 1a 2b 30 00
Memory block allocated at (most recent call first):
File "test/test_bytes.py", line 323
File "unittest/case.py", line 600
File "unittest/case.py", line 648
File "unittest/suite.py", line 122
File "unittest/suite.py", line 84
Fatal Python error: bad trailing pad byte
Current thread 0x00007fbcdbd32700 (most recent call first):
File "test/test_bytes.py", line 323 in test_hex
File "unittest/case.py", line 600 in run
File "unittest/case.py", line 648 in __call__
File "unittest/suite.py", line 122 in run
File "unittest/suite.py", line 84 in __call__
File "unittest/suite.py", line 122 in run
File "unittest/suite.py", line 84 in __call__
...
(Contributed by Victor Stinner in bpo-26516(1) and bpo-26564(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26516
(2) https://bugs.python.org/issue?@action=redirect&bpo=26564
File: python.info, Node: DTrace and SystemTap probing support, Prev: PYTHONMALLOC environment variable, Up: New Features<13>
1.8.2.17 DTrace and SystemTap probing support
.............................................
Python can now be built ‘--with-dtrace’ which enables static markers for
the following events in the interpreter:
* function call/return
* garbage collection started/finished
* line of code executed.
This can be used to instrument running interpreters in production,
without the need to recompile specific *note debug builds: 1fb. or
providing application-specific profiling/debugging code.
More details in *note Instrumenting CPython with DTrace and SystemTap:
c6c.
The current implementation is tested on Linux and macOS. Additional
markers may be added in the future.
(Contributed by Łukasz Langa in bpo-21590(1), based on patches by Jesús
Cea Avión, David Malcolm, and Nikhil Benesch.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21590
File: python.info, Node: Other Language Changes<8>, Next: New Modules<8>, Prev: New Features<13>, Up: What’s New In Python 3 6
1.8.3 Other Language Changes
----------------------------
Some smaller changes made to the core Python language are:
* A ‘global’ or ‘nonlocal’ statement must now textually appear before
the first use of the affected name in the same scope. Previously
this was a *note SyntaxWarning: 461.
* It is now possible to set a *note special method: c6e. to ‘None’ to
indicate that the corresponding operation is not available. For
example, if a class sets ‘__iter__()’ to ‘None’, the class is not
iterable. (Contributed by Andrew Barnert and Ivan Levkivskyi in
bpo-25958(1).)
* Long sequences of repeated traceback lines are now abbreviated as
‘"[Previous line repeated {count} more times]"’ (see *note
traceback: c6f. for an example). (Contributed by Emanuel Barry in
bpo-26823(2).)
* Import now raises the new exception *note ModuleNotFoundError: b47.
(subclass of *note ImportError: 415.) when it cannot find a module.
Code that currently checks for ImportError (in try-except) will
still work. (Contributed by Eric Snow in bpo-15767(3).)
* Class methods relying on zero-argument ‘super()’ will now work
correctly when called from metaclass methods during class creation.
(Contributed by Martin Teichmann in bpo-23722(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25958
(2) https://bugs.python.org/issue?@action=redirect&bpo=26823
(3) https://bugs.python.org/issue?@action=redirect&bpo=15767
(4) https://bugs.python.org/issue?@action=redirect&bpo=23722
File: python.info, Node: New Modules<8>, Next: Improved Modules<8>, Prev: Other Language Changes<8>, Up: What’s New In Python 3 6
1.8.4 New Modules
-----------------
* Menu:
* secrets::
File: python.info, Node: secrets, Up: New Modules<8>
1.8.4.1 secrets
...............
The main purpose of the new *note secrets: c0. module is to provide an
obvious way to reliably generate cryptographically strong pseudo-random
values suitable for managing secrets, such as account authentication,
tokens, and similar.
Warning: Note that the pseudo-random generators in the *note
random: b8. module should 'NOT' be used for security purposes. Use
*note secrets: c0. on Python 3.6+ and *note os.urandom(): 51e. on
Python 3.5 and earlier.
See also
........
PEP 506(1) – Adding A Secrets Module To The Standard Library
PEP written and implemented by Steven D’Aprano.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0506/
File: python.info, Node: Improved Modules<8>, Next: Optimizations<8>, Prev: New Modules<8>, Up: What’s New In Python 3 6
1.8.5 Improved Modules
----------------------
* Menu:
* array: array<4>.
* ast: ast<4>.
* asyncio: asyncio<9>.
* binascii: binascii<2>.
* cmath::
* collections: collections<4>.
* concurrent.futures: concurrent futures<4>.
* contextlib: contextlib<4>.
* datetime: datetime<5>.
* decimal: decimal<2>.
* distutils: distutils<5>.
* email: email<2>.
* encodings: encodings<2>.
* enum: enum<7>.
* faulthandler: faulthandler<2>.
* fileinput: fileinput<2>.
* hashlib: hashlib<5>.
* http.client: http client<2>.
* idlelib and IDLE: idlelib and IDLE<2>.
* importlib: importlib<6>.
* inspect: inspect<6>.
* json::
* logging: logging<4>.
* math: math<7>.
* multiprocessing: multiprocessing<5>.
* os: os<8>.
* pathlib: pathlib<9>.
* pdb: pdb<5>.
* pickle: pickle<2>.
* pickletools::
* pydoc: pydoc<3>.
* random: random<4>.
* re: re<5>.
* readline::
* rlcompleter::
* shlex: shlex<2>.
* site: site<3>.
* sqlite3: sqlite3<7>.
* socket: socket<7>.
* socketserver: socketserver<2>.
* ssl: ssl<7>.
* statistics: statistics<5>.
* struct::
* subprocess: subprocess<3>.
* sys: sys<10>.
* telnetlib::
* time: time<6>.
* timeit::
* tkinter: tkinter<6>.
* traceback: traceback<4>.
* tracemalloc: tracemalloc<3>.
* typing: typing<9>.
* unicodedata: unicodedata<7>.
* unittest.mock: unittest mock<2>.
* urllib.request: urllib request.
* urllib.robotparser: urllib robotparser.
* venv: venv<5>.
* warnings: warnings<4>.
* winreg::
* winsound::
* xmlrpc.client: xmlrpc client.
* zipfile: zipfile<3>.
* zlib::
File: python.info, Node: array<4>, Next: ast<4>, Up: Improved Modules<8>
1.8.5.1 array
.............
Exhausted iterators of *note array.array: 1a0. will now stay exhausted
even if the iterated array is extended. This is consistent with the
behavior of other mutable sequences.
Contributed by Serhiy Storchaka in bpo-26492(1).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26492
File: python.info, Node: ast<4>, Next: asyncio<9>, Prev: array<4>, Up: Improved Modules<8>
1.8.5.2 ast
...........
The new *note ast.Constant: 2bb. AST node has been added. It can be
used by external AST optimizers for the purposes of constant folding.
Contributed by Victor Stinner in bpo-26146(1).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26146
File: python.info, Node: asyncio<9>, Next: binascii<2>, Prev: ast<4>, Up: Improved Modules<8>
1.8.5.3 asyncio
...............
Starting with Python 3.6 the ‘asyncio’ module is no longer provisional
and its API is considered stable.
Notable changes in the *note asyncio: a. module since Python 3.5.0 (all
backported to 3.5.x due to the provisional status):
* The *note get_event_loop(): 2c4. function has been changed to
always return the currently running loop when called from
coroutines and callbacks. (Contributed by Yury Selivanov in
bpo-28613(1).)
* The *note ensure_future(): c76. function and all functions that use
it, such as *note loop.run_until_complete(): c77, now accept all
kinds of *note awaitable objects: 1ad. (Contributed by Yury
Selivanov.)
* New *note run_coroutine_threadsafe(): 880. function to submit
coroutines to event loops from other threads. (Contributed by
Vincent Michel.)
* New *note Transport.is_closing(): c78. method to check if the
transport is closing or closed. (Contributed by Yury Selivanov.)
* The *note loop.create_server(): b13. method can now accept a list
of hosts. (Contributed by Yann Sionneau.)
* New *note loop.create_future(): c79. method to create Future
objects. This allows alternative event loop implementations, such
as uvloop(2), to provide a faster *note asyncio.Future: bcd.
implementation. (Contributed by Yury Selivanov in bpo-27041(3).)
* New *note loop.get_exception_handler(): c7a. method to get the
current exception handler. (Contributed by Yury Selivanov in
bpo-27040(4).)
* New *note StreamReader.readuntil(): 1b4. method to read data from
the stream until a separator bytes sequence appears. (Contributed
by Mark Korenberg.)
* The performance of *note StreamReader.readexactly(): c7b. has been
improved. (Contributed by Mark Korenberg in bpo-28370(5).)
* The *note loop.getaddrinfo(): c14. method is optimized to avoid
calling the system ‘getaddrinfo’ function if the address is already
resolved. (Contributed by A. Jesse Jiryu Davis.)
* The *note loop.stop(): c7c. method has been changed to stop the
loop immediately after the current iteration. Any new callbacks
scheduled as a result of the last iteration will be discarded.
(Contributed by Guido van Rossum in bpo-25593(6).)
* ‘Future.set_exception’ will now raise *note TypeError: 534. when
passed an instance of the *note StopIteration: bfa. exception.
(Contributed by Chris Angelico in bpo-26221(7).)
* New *note loop.connect_accepted_socket(): b1a. method to be used by
servers that accept connections outside of asyncio, but that use
asyncio to handle them. (Contributed by Jim Fulton in
bpo-27392(8).)
* ‘TCP_NODELAY’ flag is now set for all TCP transports by default.
(Contributed by Yury Selivanov in bpo-27456(9).)
* New *note loop.shutdown_asyncgens(): c7d. to properly close pending
asynchronous generators before closing the loop. (Contributed by
Yury Selivanov in bpo-28003(10).)
* *note Future: bcd. and *note Task: 1be. classes now have an
optimized C implementation which makes asyncio code up to 30%
faster. (Contributed by Yury Selivanov and INADA Naoki in
bpo-26081(11) and bpo-28544(12).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28613
(2) https://github.com/MagicStack/uvloop
(3) https://bugs.python.org/issue?@action=redirect&bpo=27041
(4) https://bugs.python.org/issue?@action=redirect&bpo=27040
(5) https://bugs.python.org/issue?@action=redirect&bpo=28370
(6) https://bugs.python.org/issue?@action=redirect&bpo=25593
(7) https://bugs.python.org/issue?@action=redirect&bpo=26221
(8) https://bugs.python.org/issue?@action=redirect&bpo=27392
(9) https://bugs.python.org/issue?@action=redirect&bpo=27456
(10) https://bugs.python.org/issue?@action=redirect&bpo=28003
(11) https://bugs.python.org/issue?@action=redirect&bpo=26081
(12) https://bugs.python.org/issue?@action=redirect&bpo=28544
File: python.info, Node: binascii<2>, Next: cmath, Prev: asyncio<9>, Up: Improved Modules<8>
1.8.5.4 binascii
................
The *note b2a_base64(): c7f. function now accepts an optional 'newline'
keyword argument to control whether the newline character is appended to
the return value. (Contributed by Victor Stinner in bpo-25357(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25357
File: python.info, Node: cmath, Next: collections<4>, Prev: binascii<2>, Up: Improved Modules<8>
1.8.5.5 cmath
.............
The new *note cmath.tau: c81. ('τ') constant has been added.
(Contributed by Lisa Roach in bpo-12345(1), see PEP 628(2) for details.)
New constants: *note cmath.inf: c82. and *note cmath.nan: c83. to match
*note math.inf: c84. and *note math.nan: 65f, and also *note cmath.infj:
c85. and *note cmath.nanj: c86. to match the format used by complex
repr. (Contributed by Mark Dickinson in bpo-23229(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12345
(2) https://peps.python.org/pep-0628/
(3) https://bugs.python.org/issue?@action=redirect&bpo=23229
File: python.info, Node: collections<4>, Next: concurrent futures<4>, Prev: cmath, Up: Improved Modules<8>
1.8.5.6 collections
...................
The new *note Collection: c88. abstract base class has been added to
represent sized iterable container classes. (Contributed by Ivan
Levkivskyi, docs by Neil Girdhar in bpo-27598(1).)
The new *note Reversible: c89. abstract base class represents iterable
classes that also provide the ‘__reversed__()’ method. (Contributed by
Ivan Levkivskyi in bpo-25987(2).)
The new *note AsyncGenerator: c8a. abstract base class represents
asynchronous generators. (Contributed by Yury Selivanov in
bpo-28720(3).)
The *note namedtuple(): 1ca. function now accepts an optional keyword
argument 'module', which, when specified, is used for the *note
__module__: 36f. attribute of the returned named tuple class.
(Contributed by Raymond Hettinger in bpo-17941(4).)
The 'verbose' and 'rename' arguments for *note namedtuple(): 1ca. are
now keyword-only. (Contributed by Raymond Hettinger in bpo-25628(5).)
Recursive *note collections.deque: 5d8. instances can now be pickled.
(Contributed by Serhiy Storchaka in bpo-26482(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27598
(2) https://bugs.python.org/issue?@action=redirect&bpo=25987
(3) https://bugs.python.org/issue?@action=redirect&bpo=28720
(4) https://bugs.python.org/issue?@action=redirect&bpo=17941
(5) https://bugs.python.org/issue?@action=redirect&bpo=25628
(6) https://bugs.python.org/issue?@action=redirect&bpo=26482
File: python.info, Node: concurrent futures<4>, Next: contextlib<4>, Prev: collections<4>, Up: Improved Modules<8>
1.8.5.7 concurrent.futures
..........................
The *note ThreadPoolExecutor: 73d. class constructor now accepts an
optional 'thread_name_prefix' argument to make it possible to customize
the names of the threads created by the pool. (Contributed by Gregory
P. Smith in bpo-27664(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27664
File: python.info, Node: contextlib<4>, Next: datetime<5>, Prev: concurrent futures<4>, Up: Improved Modules<8>
1.8.5.8 contextlib
..................
The *note contextlib.AbstractContextManager: c8d. class has been added
to provide an abstract base class for context managers. It provides a
sensible default implementation for ‘__enter__()’ which returns ‘self’
and leaves ‘__exit__()’ an abstract method. A matching class has been
added to the *note typing: 104. module as *note typing.ContextManager:
c8e. (Contributed by Brett Cannon in bpo-25609(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25609
File: python.info, Node: datetime<5>, Next: decimal<2>, Prev: contextlib<4>, Up: Improved Modules<8>
1.8.5.9 datetime
................
The *note datetime: 1cc. and *note time: 1ce. classes have the new
‘fold’ attribute used to disambiguate local time when necessary. Many
functions in the *note datetime: 30. have been updated to support local
time disambiguation. See *note Local Time Disambiguation: c3c. section
for more information. (Contributed by Alexander Belopolsky in
bpo-24773(1).)
The *note datetime.strftime(): c90. and *note date.strftime(): c91.
methods now support ISO 8601 date directives ‘%G’, ‘%u’ and ‘%V’.
(Contributed by Ashley Anderson in bpo-12006(2).)
The *note datetime.isoformat(): b30. function now accepts an optional
'timespec' argument that specifies the number of additional components
of the time value to include. (Contributed by Alessandro Cucci and
Alexander Belopolsky in bpo-19475(3).)
The *note datetime.combine(): c92. now accepts an optional 'tzinfo'
argument. (Contributed by Alexander Belopolsky in bpo-27661(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24773
(2) https://bugs.python.org/issue?@action=redirect&bpo=12006
(3) https://bugs.python.org/issue?@action=redirect&bpo=19475
(4) https://bugs.python.org/issue?@action=redirect&bpo=27661
File: python.info, Node: decimal<2>, Next: distutils<5>, Prev: datetime<5>, Up: Improved Modules<8>
1.8.5.10 decimal
................
New *note Decimal.as_integer_ratio(): c94. method that returns a pair
‘(n, d)’ of integers that represent the given *note Decimal: 29f.
instance as a fraction, in lowest terms and with a positive denominator:
>>> Decimal('-3.14').as_integer_ratio()
(-157, 50)
(Contributed by Stefan Krah amd Mark Dickinson in bpo-25928(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25928
File: python.info, Node: distutils<5>, Next: email<2>, Prev: decimal<2>, Up: Improved Modules<8>
1.8.5.11 distutils
..................
The ‘default_format’ attribute has been removed from
‘distutils.command.sdist.sdist’ and the ‘formats’ attribute defaults to
‘['gztar']’. Although not anticipated, any code relying on the presence
of ‘default_format’ may need to be adapted. See bpo-27819(1) for more
details.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27819
File: python.info, Node: email<2>, Next: encodings<2>, Prev: distutils<5>, Up: Improved Modules<8>
1.8.5.12 email
..............
The new email API, enabled via the 'policy' keyword to various
constructors, is no longer provisional. The *note email: 3b.
documentation has been reorganized and rewritten to focus on the new
API, while retaining the old documentation for the legacy API.
(Contributed by R. David Murray in bpo-24277(1).)
The *note email.mime: 45. classes now all accept an optional 'policy'
keyword. (Contributed by Berker Peksag in bpo-27331(2).)
The *note DecodedGenerator: c97. now supports the 'policy' keyword.
There is a new *note policy: 4f. attribute, *note message_factory: c98,
that controls what class is used by default when the parser creates new
message objects. For the *note email.policy.compat32: c99. policy this
is *note Message: 27e, for the new policies it is *note EmailMessage:
27d. (Contributed by R. David Murray in bpo-20476(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24277
(2) https://bugs.python.org/issue?@action=redirect&bpo=27331
(3) https://bugs.python.org/issue?@action=redirect&bpo=20476
File: python.info, Node: encodings<2>, Next: enum<7>, Prev: email<2>, Up: Improved Modules<8>
1.8.5.13 encodings
..................
On Windows, added the ‘'oem'’ encoding to use ‘CP_OEMCP’, and the
‘'ansi'’ alias for the existing ‘'mbcs'’ encoding, which uses the
‘CP_ACP’ code page. (Contributed by Steve Dower in bpo-27959(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27959
File: python.info, Node: enum<7>, Next: faulthandler<2>, Prev: encodings<2>, Up: Improved Modules<8>
1.8.5.14 enum
.............
Two new enumeration base classes have been added to the *note enum: 56.
module: *note Flag: 61f. and ‘IntFlags’. Both are used to define
constants that can be combined using the bitwise operators.
(Contributed by Ethan Furman in bpo-23591(1).)
Many standard library modules have been updated to use the ‘IntFlags’
class for their constants.
The new *note enum.auto: c9c. value can be used to assign values to enum
members automatically:
>>> from enum import Enum, auto
>>> class Color(Enum):
... red = auto()
... blue = auto()
... green = auto()
...
>>> list(Color)
[, , ]
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23591
File: python.info, Node: faulthandler<2>, Next: fileinput<2>, Prev: enum<7>, Up: Improved Modules<8>
1.8.5.15 faulthandler
.....................
On Windows, the *note faulthandler: 58. module now installs a handler
for Windows exceptions: see *note faulthandler.enable(): c9e.
(Contributed by Victor Stinner in bpo-23848(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23848
File: python.info, Node: fileinput<2>, Next: hashlib<5>, Prev: faulthandler<2>, Up: Improved Modules<8>
1.8.5.16 fileinput
..................
*note hook_encoded(): ca0. now supports the 'errors' argument.
(Contributed by Joseph Hackman in bpo-25788(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25788
File: python.info, Node: hashlib<5>, Next: http client<2>, Prev: fileinput<2>, Up: Improved Modules<8>
1.8.5.17 hashlib
................
*note hashlib: 68. supports OpenSSL 1.1.0. The minimum recommend
version is 1.0.2. (Contributed by Christian Heimes in bpo-26470(1).)
BLAKE2 hash functions were added to the module. *note blake2b(): 63e.
and *note blake2s(): 63f. are always available and support the full
feature set of BLAKE2. (Contributed by Christian Heimes in bpo-26798(2)
based on code by Dmitry Chestnykh and Samuel Neves. Documentation
written by Dmitry Chestnykh.)
The SHA-3 hash functions *note sha3_224(): ca2, *note sha3_256(): ca3,
*note sha3_384(): ca4, *note sha3_512(): ca5, and SHAKE hash functions
*note shake_128(): ca6. and *note shake_256(): ca7. were added.
(Contributed by Christian Heimes in bpo-16113(3). Keccak Code Package
by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and
Ronny Van Keer.)
The password-based key derivation function *note scrypt(): c3e. is now
available with OpenSSL 1.1.0 and newer. (Contributed by Christian
Heimes in bpo-27928(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26470
(2) https://bugs.python.org/issue?@action=redirect&bpo=26798
(3) https://bugs.python.org/issue?@action=redirect&bpo=16113
(4) https://bugs.python.org/issue?@action=redirect&bpo=27928
File: python.info, Node: http client<2>, Next: idlelib and IDLE<2>, Prev: hashlib<5>, Up: Improved Modules<8>
1.8.5.18 http.client
....................
*note HTTPConnection.request(): ca9. and *note endheaders(): caa. both
now support chunked encoding request bodies. (Contributed by Demian
Brecht and Rolf Krahl in bpo-12319(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12319
File: python.info, Node: idlelib and IDLE<2>, Next: importlib<6>, Prev: http client<2>, Up: Improved Modules<8>
1.8.5.19 idlelib and IDLE
.........................
The idlelib package is being modernized and refactored to make IDLE look
and work better and to make the code easier to understand, test, and
improve. Part of making IDLE look better, especially on Linux and Mac,
is using ttk widgets, mostly in the dialogs. As a result, IDLE no
longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We
recommend running the latest release of either.
‘Modernizing’ includes renaming and consolidation of idlelib modules.
The renaming of files with partial uppercase names is similar to the
renaming of, for instance, Tkinter and TkFont to tkinter and
tkinter.font in 3.0. As a result, imports of idlelib files that worked
in 3.5 will usually not work in 3.6. At least a module name change will
be needed (see idlelib/README.txt), sometimes more. (Name changes
contributed by Al Swiegart and Terry Reedy in bpo-24225(1). Most
idlelib patches since have been and will be part of the process.)
In compensation, the eventual result with be that some idlelib classes
will be easier to use, with better APIs and docstrings explaining them.
Additional useful information will be added to idlelib when available.
New in 3.6.2:
Multiple fixes for autocompletion. (Contributed by Louie Lu in
bpo-15786(2).)
New in 3.6.3:
Module Browser (on the File menu, formerly called Class Browser), now
displays nested functions and classes in addition to top-level functions
and classes. (Contributed by Guilherme Polo, Cheryl Sabella, and Terry
Jan Reedy in bpo-1612262(3).)
The IDLE features formerly implemented as extensions have been
reimplemented as normal features. Their settings have been moved from
the Extensions tab to other dialog tabs. (Contributed by Charles
Wohlganger and Terry Jan Reedy in bpo-27099(4).)
The Settings dialog (Options, Configure IDLE) has been partly rewritten
to improve both appearance and function. (Contributed by Cheryl Sabella
and Terry Jan Reedy in multiple issues.)
New in 3.6.4:
The font sample now includes a selection of non-Latin characters so that
users can better see the effect of selecting a particular font.
(Contributed by Terry Jan Reedy in bpo-13802(5).) The sample can be
edited to include other characters. (Contributed by Serhiy Storchaka in
bpo-31860(6).)
New in 3.6.6:
Editor code context option revised. Box displays all context lines up
to maxlines. Clicking on a context line jumps the editor to that line.
Context colors for custom themes is added to Highlights tab of Settings
dialog. (Contributed by Cheryl Sabella and Terry Jan Reedy in
bpo-33642(7), bpo-33768(8), and bpo-33679(9).)
On Windows, a new API call tells Windows that tk scales for DPI. On
Windows 8.1+ or 10, with DPI compatibility properties of the Python
binary unchanged, and a monitor resolution greater than 96 DPI, this
should make text and lines sharper. It should otherwise have no effect.
(Contributed by Terry Jan Reedy in bpo-33656(10).)
New in 3.6.7:
Output over N lines (50 by default) is squeezed down to a button. N can
be changed in the PyShell section of the General page of the Settings
dialog. Fewer, but possibly extra long, lines can be squeezed by right
clicking on the output. Squeezed output can be expanded in place by
double-clicking the button or into the clipboard or a separate window by
right-clicking the button. (Contributed by Tal Einat in
bpo-1529353(11).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24225
(2) https://bugs.python.org/issue?@action=redirect&bpo=15786
(3) https://bugs.python.org/issue?@action=redirect&bpo=1612262
(4) https://bugs.python.org/issue?@action=redirect&bpo=27099
(5) https://bugs.python.org/issue?@action=redirect&bpo=13802
(6) https://bugs.python.org/issue?@action=redirect&bpo=31860
(7) https://bugs.python.org/issue?@action=redirect&bpo=33642
(8) https://bugs.python.org/issue?@action=redirect&bpo=33768
(9) https://bugs.python.org/issue?@action=redirect&bpo=33679
(10) https://bugs.python.org/issue?@action=redirect&bpo=33656
(11) https://bugs.python.org/issue?@action=redirect&bpo=1529353
File: python.info, Node: importlib<6>, Next: inspect<6>, Prev: idlelib and IDLE<2>, Up: Improved Modules<8>
1.8.5.20 importlib
..................
Import now raises the new exception *note ModuleNotFoundError: b47.
(subclass of *note ImportError: 415.) when it cannot find a module.
Code that current checks for ‘ImportError’ (in try-except) will still
work. (Contributed by Eric Snow in bpo-15767(1).)
*note importlib.util.LazyLoader: cad. now calls *note create_module():
cae. on the wrapped loader, removing the restriction that *note
importlib.machinery.BuiltinImporter: caf. and *note
importlib.machinery.ExtensionFileLoader: cb0. couldn’t be used with
*note importlib.util.LazyLoader: cad.
*note importlib.util.cache_from_source(): 2f8, *note
importlib.util.source_from_cache(): 515, and *note
importlib.util.spec_from_file_location(): cb1. now accept a *note
path-like object: 2a2.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15767
File: python.info, Node: inspect<6>, Next: json, Prev: importlib<6>, Up: Improved Modules<8>
1.8.5.21 inspect
................
The *note inspect.signature(): 733. function now reports the implicit
‘.0’ parameters generated by the compiler for comprehension and
generator expression scopes as if they were positional-only parameters
called ‘implicit0’. (Contributed by Jelle Zijlstra in bpo-19611(1).)
To reduce code churn when upgrading from Python 2.7 and the legacy
‘inspect.getargspec()’ API, the previously documented deprecation of
*note inspect.getfullargspec(): 734. has been reversed. While this
function is convenient for single/source Python 2/3 code bases, the
richer *note inspect.signature(): 733. interface remains the recommended
approach for new code. (Contributed by Nick Coghlan in bpo-27172(2))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19611
(2) https://bugs.python.org/issue?@action=redirect&bpo=27172
File: python.info, Node: json, Next: logging<4>, Prev: inspect<6>, Up: Improved Modules<8>
1.8.5.22 json
.............
*note json.load(): cb4. and *note json.loads(): 955. now support binary
input. Encoded JSON should be represented using either UTF-8, UTF-16,
or UTF-32. (Contributed by Serhiy Storchaka in bpo-17909(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17909
File: python.info, Node: logging<4>, Next: math<7>, Prev: json, Up: Improved Modules<8>
1.8.5.23 logging
................
The new *note WatchedFileHandler.reopenIfNeeded(): cb6. method has been
added to add the ability to check if the log file needs to be reopened.
(Contributed by Marian Horban in bpo-24884(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24884
File: python.info, Node: math<7>, Next: multiprocessing<5>, Prev: logging<4>, Up: Improved Modules<8>
1.8.5.24 math
.............
The tau ('τ') constant has been added to the *note math: 8e. and *note
cmath: 18. modules. (Contributed by Lisa Roach in bpo-12345(1), see PEP
628(2) for details.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12345
(2) https://peps.python.org/pep-0628/
File: python.info, Node: multiprocessing<5>, Next: os<8>, Prev: math<7>, Up: Improved Modules<8>
1.8.5.25 multiprocessing
........................
*note Proxy Objects: cb9. returned by *note multiprocessing.Manager():
cba. can now be nested. (Contributed by Davin Potts in bpo-6766(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6766
File: python.info, Node: os<8>, Next: pathlib<9>, Prev: multiprocessing<5>, Up: Improved Modules<8>
1.8.5.26 os
...........
See the summary of *note PEP 519: c3b. for details on how the *note os:
a1. and *note os.path: a2. modules now support *note path-like objects:
2a2.
*note scandir(): a5e. now supports *note bytes: 1c2. paths on Windows.
A new *note close(): cbc. method allows explicitly closing a *note
scandir(): a5e. iterator. The *note scandir(): a5e. iterator now
supports the *note context manager: 5d0. protocol. If a ‘scandir()’
iterator is neither exhausted nor explicitly closed a *note
ResourceWarning: 246. will be emitted in its destructor. (Contributed
by Serhiy Storchaka in bpo-25994(1).)
On Linux, *note os.urandom(): 51e. now blocks until the system urandom
entropy pool is initialized to increase the security. See the PEP
524(2) for the rationale.
The Linux ‘getrandom()’ syscall (get random bytes) is now exposed as the
new *note os.getrandom(): cbd. function. (Contributed by Victor
Stinner, part of the PEP 524(3))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25994
(2) https://peps.python.org/pep-0524/
(3) https://peps.python.org/pep-0524/
File: python.info, Node: pathlib<9>, Next: pdb<5>, Prev: os<8>, Up: Improved Modules<8>
1.8.5.27 pathlib
................
*note pathlib: a4. now supports *note path-like objects: 2a2.
(Contributed by Brett Cannon in bpo-27186(1).)
See the summary of *note PEP 519: c3b. for details.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27186
File: python.info, Node: pdb<5>, Next: pickle<2>, Prev: pathlib<9>, Up: Improved Modules<8>
1.8.5.28 pdb
............
The *note Pdb: 921. class constructor has a new optional 'readrc'
argument to control whether ‘.pdbrc’ files should be read.
File: python.info, Node: pickle<2>, Next: pickletools, Prev: pdb<5>, Up: Improved Modules<8>
1.8.5.29 pickle
...............
Objects that need ‘__new__’ called with keyword arguments can now be
pickled using *note pickle protocols: cc1. older than protocol version
4. Protocol version 4 already supports this case. (Contributed by
Serhiy Storchaka in bpo-24164(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24164
File: python.info, Node: pickletools, Next: pydoc<3>, Prev: pickle<2>, Up: Improved Modules<8>
1.8.5.30 pickletools
....................
*note pickletools.dis(): cc3. now outputs the implicit memo index for
the ‘MEMOIZE’ opcode. (Contributed by Serhiy Storchaka in
bpo-25382(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25382
File: python.info, Node: pydoc<3>, Next: random<4>, Prev: pickletools, Up: Improved Modules<8>
1.8.5.31 pydoc
..............
The *note pydoc: b5. module has learned to respect the ‘MANPAGER’
environment variable. (Contributed by Matthias Klose in bpo-8637(1).)
*note help(): 8d6. and *note pydoc: b5. can now list named tuple fields
in the order they were defined rather than alphabetically. (Contributed
by Raymond Hettinger in bpo-24879(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8637
(2) https://bugs.python.org/issue?@action=redirect&bpo=24879
File: python.info, Node: random<4>, Next: re<5>, Prev: pydoc<3>, Up: Improved Modules<8>
1.8.5.32 random
...............
The new *note choices(): cc6. function returns a list of elements of
specified size from the given population with optional weights.
(Contributed by Raymond Hettinger in bpo-18844(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18844
File: python.info, Node: re<5>, Next: readline, Prev: random<4>, Up: Improved Modules<8>
1.8.5.33 re
...........
Added support of modifier spans in regular expressions. Examples:
‘'(?i:p)ython'’ matches ‘'python'’ and ‘'Python'’, but not ‘'PYTHON'’;
‘'(?i)g(?-i:v)r'’ matches ‘'GvR'’ and ‘'gvr'’, but not ‘'GVR'’.
(Contributed by Serhiy Storchaka in bpo-433028(1).)
Match object groups can be accessed by ‘__getitem__’, which is
equivalent to ‘group()’. So ‘mo['name']’ is now equivalent to
‘mo.group('name')’. (Contributed by Eric Smith in bpo-24454(2).)
*note Match: cc8. objects now support *note index-like objects: 718. as
group indices. (Contributed by Jeroen Demeyer and Xiang Zhang in
bpo-27177(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=433028
(2) https://bugs.python.org/issue?@action=redirect&bpo=24454
(3) https://bugs.python.org/issue?@action=redirect&bpo=27177
File: python.info, Node: readline, Next: rlcompleter, Prev: re<5>, Up: Improved Modules<8>
1.8.5.34 readline
.................
Added *note set_auto_history(): cca. to enable or disable automatic
addition of input to the history list. (Contributed by Tyler Crompton
in bpo-26870(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26870
File: python.info, Node: rlcompleter, Next: shlex<2>, Prev: readline, Up: Improved Modules<8>
1.8.5.35 rlcompleter
....................
Private and special attribute names now are omitted unless the prefix
starts with underscores. A space or a colon is added after some
completed keywords. (Contributed by Serhiy Storchaka in bpo-25011(1)
and bpo-25209(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25011
(2) https://bugs.python.org/issue?@action=redirect&bpo=25209
File: python.info, Node: shlex<2>, Next: site<3>, Prev: rlcompleter, Up: Improved Modules<8>
1.8.5.36 shlex
..............
The *note shlex: ccd. has much *note improved shell compatibility: cce.
through the new 'punctuation_chars' argument to control which characters
are treated as punctuation. (Contributed by Vinay Sajip in
bpo-1521950(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1521950
File: python.info, Node: site<3>, Next: sqlite3<7>, Prev: shlex<2>, Up: Improved Modules<8>
1.8.5.37 site
.............
When specifying paths to add to *note sys.path: 3b0. in a ‘.pth’ file,
you may now specify file paths on top of directories (e.g. zip files).
(Contributed by Wolfgang Langner in bpo-26587(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26587
File: python.info, Node: sqlite3<7>, Next: socket<7>, Prev: site<3>, Up: Improved Modules<8>
1.8.5.38 sqlite3
................
*note sqlite3.Cursor.lastrowid: cd1. now supports the ‘REPLACE’
statement. (Contributed by Alex LordThorsen in bpo-16864(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16864
File: python.info, Node: socket<7>, Next: socketserver<2>, Prev: sqlite3<7>, Up: Improved Modules<8>
1.8.5.39 socket
...............
The *note ioctl(): cd3. function now supports the *note
SIO_LOOPBACK_FAST_PATH: cd4. control code. (Contributed by Daniel
Stokes in bpo-26536(1).)
The *note getsockopt(): cd5. constants ‘SO_DOMAIN’, ‘SO_PROTOCOL’,
‘SO_PEERSEC’, and ‘SO_PASSSEC’ are now supported. (Contributed by
Christian Heimes in bpo-26907(2).)
The *note setsockopt(): cd6. now supports the ‘setsockopt(level,
optname, None, optlen: int)’ form. (Contributed by Christian Heimes in
bpo-27744(3).)
The socket module now supports the address family *note AF_ALG: cd7. to
interface with Linux Kernel crypto API. ‘ALG_*’, ‘SOL_ALG’ and *note
sendmsg_afalg(): cd8. were added. (Contributed by Christian Heimes in
bpo-27744(4) with support from Victor Stinner.)
New Linux constants ‘TCP_USER_TIMEOUT’ and ‘TCP_CONGESTION’ were added.
(Contributed by Omar Sandoval, bpo-26273(5)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26536
(2) https://bugs.python.org/issue?@action=redirect&bpo=26907
(3) https://bugs.python.org/issue?@action=redirect&bpo=27744
(4) https://bugs.python.org/issue?@action=redirect&bpo=27744
(5) https://bugs.python.org/issue?@action=redirect&bpo=26273
File: python.info, Node: socketserver<2>, Next: ssl<7>, Prev: socket<7>, Up: Improved Modules<8>
1.8.5.40 socketserver
.....................
Servers based on the *note socketserver: cd. module, including those
defined in *note http.server: 72, *note xmlrpc.server: 12f. and *note
wsgiref.simple_server: 11b, now support the *note context manager: 5d0.
protocol. (Contributed by Aviv Palivoda in bpo-26404(1).)
The ‘wfile’ attribute of *note StreamRequestHandler: cda. classes now
implements the *note io.BufferedIOBase: 690. writable interface. In
particular, calling *note write(): cdb. is now guaranteed to send the
data in full. (Contributed by Martin Panter in bpo-26721(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26404
(2) https://bugs.python.org/issue?@action=redirect&bpo=26721
File: python.info, Node: ssl<7>, Next: statistics<5>, Prev: socketserver<2>, Up: Improved Modules<8>
1.8.5.41 ssl
............
*note ssl: d0. supports OpenSSL 1.1.0. The minimum recommend version is
1.0.2. (Contributed by Christian Heimes in bpo-26470(1).)
3DES has been removed from the default cipher suites and ChaCha20
Poly1305 cipher suites have been added. (Contributed by Christian
Heimes in bpo-27850(2) and bpo-27766(3).)
*note SSLContext: 296. has better default configuration for options and
ciphers. (Contributed by Christian Heimes in bpo-28043(4).)
SSL session can be copied from one client-side connection to another
with the new *note SSLSession: cdd. class. TLS session resumption can
speed up the initial handshake, reduce latency and improve performance
(Contributed by Christian Heimes in bpo-19500(5) based on a draft by
Alex Warhawk.)
The new *note get_ciphers(): cde. method can be used to get a list of
enabled ciphers in order of cipher priority.
All constants and flags have been converted to *note IntEnum: cdf. and
‘IntFlags’. (Contributed by Christian Heimes in bpo-28025(6).)
Server and client-side specific TLS protocols for *note SSLContext: 296.
were added. (Contributed by Christian Heimes in bpo-28085(7).)
Added *note ssl.SSLContext.post_handshake_auth: a30. to enable and *note
ssl.SSLSocket.verify_client_post_handshake(): a31. to initiate TLS 1.3
post-handshake authentication. (Contributed by Christian Heimes in
gh-78851(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26470
(2) https://bugs.python.org/issue?@action=redirect&bpo=27850
(3) https://bugs.python.org/issue?@action=redirect&bpo=27766
(4) https://bugs.python.org/issue?@action=redirect&bpo=28043
(5) https://bugs.python.org/issue?@action=redirect&bpo=19500
(6) https://bugs.python.org/issue?@action=redirect&bpo=28025
(7) https://bugs.python.org/issue?@action=redirect&bpo=28085
(8) https://github.com/python/cpython/issues/78851
File: python.info, Node: statistics<5>, Next: struct, Prev: ssl<7>, Up: Improved Modules<8>
1.8.5.42 statistics
...................
A new *note harmonic_mean(): ce1. function has been added. (Contributed
by Steven D’Aprano in bpo-27181(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27181
File: python.info, Node: struct, Next: subprocess<3>, Prev: statistics<5>, Up: Improved Modules<8>
1.8.5.43 struct
...............
*note struct: d5. now supports IEEE 754 half-precision floats via the
‘'e'’ format specifier. (Contributed by Eli Stevens, Mark Dickinson in
bpo-11734(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11734
File: python.info, Node: subprocess<3>, Next: sys<10>, Prev: struct, Up: Improved Modules<8>
1.8.5.44 subprocess
...................
*note subprocess.Popen: 199. destructor now emits a *note
ResourceWarning: 246. warning if the child process is still running.
Use the context manager protocol (‘with proc: ...’) or explicitly call
the *note wait(): ce4. method to read the exit status of the child
process. (Contributed by Victor Stinner in bpo-26741(1).)
The *note subprocess.Popen: 199. constructor and all functions that pass
arguments through to it now accept 'encoding' and 'errors' arguments.
Specifying either of these will enable text mode for the 'stdin',
'stdout' and 'stderr' streams. (Contributed by Steve Dower in
bpo-6135(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26741
(2) https://bugs.python.org/issue?@action=redirect&bpo=6135
File: python.info, Node: sys<10>, Next: telnetlib, Prev: subprocess<3>, Up: Improved Modules<8>
1.8.5.45 sys
............
The new *note getfilesystemencodeerrors(): ce6. function returns the
name of the error mode used to convert between Unicode filenames and
bytes filenames. (Contributed by Steve Dower in bpo-27781(1).)
On Windows the return value of the *note getwindowsversion(): ce7.
function now includes the 'platform_version' field which contains the
accurate major version, minor version and build number of the current
operating system, rather than the version that is being emulated for the
process (Contributed by Steve Dower in bpo-27932(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27781
(2) https://bugs.python.org/issue?@action=redirect&bpo=27932
File: python.info, Node: telnetlib, Next: time<6>, Prev: sys<10>, Up: Improved Modules<8>
1.8.5.46 telnetlib
..................
‘telnetlib.Telnet’ is now a context manager (contributed by Stéphane
Wirtel in bpo-25485(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25485
File: python.info, Node: time<6>, Next: timeit, Prev: telnetlib, Up: Improved Modules<8>
1.8.5.47 time
.............
The *note struct_time: cea. attributes ‘tm_gmtoff’ and ‘tm_zone’ are now
available on all platforms.
File: python.info, Node: timeit, Next: tkinter<6>, Prev: time<6>, Up: Improved Modules<8>
1.8.5.48 timeit
...............
The new *note Timer.autorange(): cec. convenience method has been added
to call *note Timer.timeit(): ced. repeatedly so that the total run time
is greater or equal to 200 milliseconds. (Contributed by Steven
D’Aprano in bpo-6422(1).)
*note timeit: ef. now warns when there is substantial (4x) variance
between best and worst times. (Contributed by Serhiy Storchaka in
bpo-23552(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6422
(2) https://bugs.python.org/issue?@action=redirect&bpo=23552
File: python.info, Node: tkinter<6>, Next: traceback<4>, Prev: timeit, Up: Improved Modules<8>
1.8.5.49 tkinter
................
Added methods ‘trace_add()’, ‘trace_remove()’ and ‘trace_info()’ in the
‘tkinter.Variable’ class. They replace old methods ‘trace_variable()’,
‘trace()’, ‘trace_vdelete()’ and ‘trace_vinfo()’ that use obsolete Tcl
commands and might not work in future versions of Tcl. (Contributed by
Serhiy Storchaka in bpo-22115(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22115
File: python.info, Node: traceback<4>, Next: tracemalloc<3>, Prev: tkinter<6>, Up: Improved Modules<8>
1.8.5.50 traceback
..................
Both the traceback module and the interpreter’s builtin exception
display now abbreviate long sequences of repeated lines in tracebacks as
shown in the following example:
>>> def f(): f()
...
>>> f()
Traceback (most recent call last):
File "", line 1, in
File "", line 1, in f
File "", line 1, in f
File "", line 1, in f
[Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded
(Contributed by Emanuel Barry in bpo-26823(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26823
File: python.info, Node: tracemalloc<3>, Next: typing<9>, Prev: traceback<4>, Up: Improved Modules<8>
1.8.5.51 tracemalloc
....................
The *note tracemalloc: ff. module now supports tracing memory
allocations in multiple different address spaces.
The new *note DomainFilter: cf1. filter class has been added to filter
block traces by their address space (domain).
(Contributed by Victor Stinner in bpo-26588(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26588
File: python.info, Node: typing<9>, Next: unicodedata<7>, Prev: tracemalloc<3>, Up: Improved Modules<8>
1.8.5.52 typing
...............
Since the *note typing: 104. module is *note provisional: b03, all
changes introduced in Python 3.6 have also been backported to Python
3.5.x.
The *note typing: 104. module has a much improved support for generic
type aliases. For example ‘Dict[str, Tuple[S, T]]’ is now a valid type
annotation. (Contributed by Guido van Rossum in Github #195(1).)
The *note typing.ContextManager: c8e. class has been added for
representing *note contextlib.AbstractContextManager: c8d. (Contributed
by Brett Cannon in bpo-25609(2).)
The *note typing.Collection: cf3. class has been added for representing
*note collections.abc.Collection: c88. (Contributed by Ivan Levkivskyi
in bpo-27598(3).)
The *note typing.ClassVar: 268. type construct has been added to mark
class variables. As introduced in PEP 526(4), a variable annotation
wrapped in ClassVar indicates that a given attribute is intended to be
used as a class variable and should not be set on instances of that
class. (Contributed by Ivan Levkivskyi in Github #280(5).)
A new *note TYPE_CHECKING: cf4. constant that is assumed to be ‘True’ by
the static type checkers, but is ‘False’ at runtime. (Contributed by
Guido van Rossum in Github #230(6).)
A new *note NewType(): cf5. helper function has been added to create
lightweight distinct types for annotations:
from typing import NewType
UserId = NewType('UserId', int)
some_id = UserId(524313)
The static type checker will treat the new type as if it were a subclass
of the original type. (Contributed by Ivan Levkivskyi in Github
#189(7).)
---------- Footnotes ----------
(1) https://github.com/python/typing/pull/195
(2) https://bugs.python.org/issue?@action=redirect&bpo=25609
(3) https://bugs.python.org/issue?@action=redirect&bpo=27598
(4) https://peps.python.org/pep-0526/
(5) https://github.com/python/typing/pull/280
(6) https://github.com/python/typing/issues/230
(7) https://github.com/python/typing/issues/189
File: python.info, Node: unicodedata<7>, Next: unittest mock<2>, Prev: typing<9>, Up: Improved Modules<8>
1.8.5.53 unicodedata
....................
The *note unicodedata: 105. module now uses data from Unicode 9.0.0(1).
(Contributed by Benjamin Peterson.)
---------- Footnotes ----------
(1) https://unicode.org/versions/Unicode9.0.0/
File: python.info, Node: unittest mock<2>, Next: urllib request, Prev: unicodedata<7>, Up: Improved Modules<8>
1.8.5.54 unittest.mock
......................
The *note Mock: a4b. class has the following improvements:
* Two new methods, *note Mock.assert_called(): cf8. and *note
Mock.assert_called_once(): cf9. to check if the mock object was
called. (Contributed by Amit Saha in bpo-26323(1).)
* The *note Mock.reset_mock(): cfa. method now has two optional
keyword only arguments: 'return_value' and 'side_effect'.
(Contributed by Kushal Das in bpo-21271(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26323
(2) https://bugs.python.org/issue?@action=redirect&bpo=21271
File: python.info, Node: urllib request, Next: urllib robotparser, Prev: unittest mock<2>, Up: Improved Modules<8>
1.8.5.55 urllib.request
.......................
If a HTTP request has a file or iterable body (other than a bytes
object) but no ‘Content-Length’ header, rather than throwing an error,
‘AbstractHTTPHandler’ now falls back to use chunked transfer encoding.
(Contributed by Demian Brecht and Rolf Krahl in bpo-12319(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12319
File: python.info, Node: urllib robotparser, Next: venv<5>, Prev: urllib request, Up: Improved Modules<8>
1.8.5.56 urllib.robotparser
...........................
*note RobotFileParser: cfd. now supports the ‘Crawl-delay’ and
‘Request-rate’ extensions. (Contributed by Nikolay Bogoychev in
bpo-16099(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16099
File: python.info, Node: venv<5>, Next: warnings<4>, Prev: urllib robotparser, Up: Improved Modules<8>
1.8.5.57 venv
.............
*note venv: 111. accepts a new parameter ‘--prompt’. This parameter
provides an alternative prefix for the virtual environment. (Proposed
by Łukasz Balcerzak and ported to 3.6 by Stéphane Wirtel in
bpo-22829(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22829
File: python.info, Node: warnings<4>, Next: winreg, Prev: venv<5>, Up: Improved Modules<8>
1.8.5.58 warnings
.................
A new optional 'source' parameter has been added to the *note
warnings.warn_explicit(): d00. function: the destroyed object which
emitted a *note ResourceWarning: 246. A 'source' attribute has also
been added to ‘warnings.WarningMessage’ (contributed by Victor Stinner
in bpo-26568(1) and bpo-26567(2)).
When a *note ResourceWarning: 246. warning is logged, the *note
tracemalloc: ff. module is now used to try to retrieve the traceback
where the destroyed object was allocated.
Example with the script ‘example.py’:
import warnings
def func():
return open(__file__)
f = func()
f = None
Output of the command ‘python3.6 -Wd -X tracemalloc=5 example.py’:
example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'>
f = None
Object allocated at (most recent call first):
File "example.py", lineno 4
return open(__file__)
File "example.py", lineno 6
f = func()
The “Object allocated at” traceback is new and is only displayed if
*note tracemalloc: ff. is tracing Python memory allocations and if the
*note warnings: 112. module was already imported.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26568
(2) https://bugs.python.org/issue?@action=redirect&bpo=26567
File: python.info, Node: winreg, Next: winsound, Prev: warnings<4>, Up: Improved Modules<8>
1.8.5.59 winreg
...............
Added the 64-bit integer type *note REG_QWORD: d02. (Contributed by
Clement Rouault in bpo-23026(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23026
File: python.info, Node: winsound, Next: xmlrpc client, Prev: winreg, Up: Improved Modules<8>
1.8.5.60 winsound
.................
Allowed keyword arguments to be passed to *note Beep: d04, *note
MessageBeep: d05, and *note PlaySound: d06. (bpo-27982(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27982
File: python.info, Node: xmlrpc client, Next: zipfile<3>, Prev: winsound, Up: Improved Modules<8>
1.8.5.61 xmlrpc.client
......................
The *note xmlrpc.client: 12e. module now supports unmarshalling
additional data types used by the Apache XML-RPC implementation for
numerics and ‘None’. (Contributed by Serhiy Storchaka in bpo-26885(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26885
File: python.info, Node: zipfile<3>, Next: zlib, Prev: xmlrpc client, Up: Improved Modules<8>
1.8.5.62 zipfile
................
A new *note ZipInfo.from_file(): d09. class method allows making a *note
ZipInfo: d0a. instance from a filesystem file. A new *note
ZipInfo.is_dir(): d0b. method can be used to check if the *note ZipInfo:
d0a. instance represents a directory. (Contributed by Thomas Kluyver in
bpo-26039(1).)
The *note ZipFile.open(): 417. method can now be used to write data into
a ZIP file, as well as for extracting data. (Contributed by Thomas
Kluyver in bpo-26039(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26039
(2) https://bugs.python.org/issue?@action=redirect&bpo=26039
File: python.info, Node: zlib, Prev: zipfile<3>, Up: Improved Modules<8>
1.8.5.63 zlib
.............
The *note compress(): 63b. and *note decompress(): d0d. functions now
accept keyword arguments. (Contributed by Aviv Palivoda in bpo-26243(1)
and Xiang Zhang in bpo-16764(2) respectively.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26243
(2) https://bugs.python.org/issue?@action=redirect&bpo=16764
File: python.info, Node: Optimizations<8>, Next: Build and C API Changes<2>, Prev: Improved Modules<8>, Up: What’s New In Python 3 6
1.8.6 Optimizations
-------------------
* The Python interpreter now uses a 16-bit wordcode instead of
bytecode which made a number of opcode optimizations possible.
(Contributed by Demur Rumed with input and reviews from Serhiy
Storchaka and Victor Stinner in bpo-26647(1) and bpo-28050(2).)
* The *note asyncio.Future: bcd. class now has an optimized C
implementation. (Contributed by Yury Selivanov and INADA Naoki in
bpo-26081(3).)
* The *note asyncio.Task: 1be. class now has an optimized C
implementation. (Contributed by Yury Selivanov in bpo-28544(4).)
* Various implementation improvements in the *note typing: 104.
module (such as caching of generic types) allow up to 30 times
performance improvements and reduced memory footprint.
* The ASCII decoder is now up to 60 times as fast for error handlers
‘surrogateescape’, ‘ignore’ and ‘replace’ (Contributed by Victor
Stinner in bpo-24870(5)).
* The ASCII and the Latin1 encoders are now up to 3 times as fast for
the error handler ‘surrogateescape’ (Contributed by Victor Stinner
in bpo-25227(6)).
* The UTF-8 encoder is now up to 75 times as fast for error handlers
‘ignore’, ‘replace’, ‘surrogateescape’, ‘surrogatepass’
(Contributed by Victor Stinner in bpo-25267(7)).
* The UTF-8 decoder is now up to 15 times as fast for error handlers
‘ignore’, ‘replace’ and ‘surrogateescape’ (Contributed by Victor
Stinner in bpo-25301(8)).
* ‘bytes % args’ is now up to 2 times faster. (Contributed by Victor
Stinner in bpo-25349(9)).
* ‘bytearray % args’ is now between 2.5 and 5 times faster.
(Contributed by Victor Stinner in bpo-25399(10)).
* Optimize *note bytes.fromhex(): aef. and *note bytearray.fromhex():
af0.: they are now between 2x and 3.5x faster. (Contributed by
Victor Stinner in bpo-25401(11)).
* Optimize ‘bytes.replace(b'', b'.')’ and ‘bytearray.replace(b'',
b'.')’: up to 80% faster. (Contributed by Josh Snider in
bpo-26574(12)).
* Allocator functions of the *note PyMem_Malloc(): c66. domain (*note
PYMEM_DOMAIN_MEM: c69.) now use the *note pymalloc memory
allocator: d0f. instead of ‘malloc()’ function of the C library.
The pymalloc allocator is optimized for objects smaller or equal to
512 bytes with a short lifetime, and use ‘malloc()’ for larger
memory blocks. (Contributed by Victor Stinner in bpo-26249(13)).
* *note pickle.load(): d10. and *note pickle.loads(): d11. are now up
to 10% faster when deserializing many small objects (Contributed by
Victor Stinner in bpo-27056(14)).
* Passing *note keyword arguments: d12. to a function has an overhead
in comparison with passing *note positional arguments: d13. Now in
extension functions implemented with using Argument Clinic this
overhead is significantly decreased. (Contributed by Serhiy
Storchaka in bpo-27574(15)).
* Optimized *note glob(): 2a1. and *note iglob(): 803. functions in
the *note glob: 64. module; they are now about 3–6 times faster.
(Contributed by Serhiy Storchaka in bpo-25596(16)).
* Optimized globbing in *note pathlib: a4. by using *note
os.scandir(): a5e.; it is now about 1.5–4 times faster.
(Contributed by Serhiy Storchaka in bpo-26032(17)).
* *note xml.etree.ElementTree: 125. parsing, iteration and deepcopy
performance has been significantly improved. (Contributed by
Serhiy Storchaka in bpo-25638(18), bpo-25873(19), and
bpo-25869(20).)
* Creation of *note fractions.Fraction: 1e9. instances from floats
and decimals is now 2 to 3 times faster. (Contributed by Serhiy
Storchaka in bpo-25971(21).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26647
(2) https://bugs.python.org/issue?@action=redirect&bpo=28050
(3) https://bugs.python.org/issue?@action=redirect&bpo=26081
(4) https://bugs.python.org/issue?@action=redirect&bpo=28544
(5) https://bugs.python.org/issue?@action=redirect&bpo=24870
(6) https://bugs.python.org/issue?@action=redirect&bpo=25227
(7) https://bugs.python.org/issue?@action=redirect&bpo=25267
(8) https://bugs.python.org/issue?@action=redirect&bpo=25301
(9) https://bugs.python.org/issue?@action=redirect&bpo=25349
(10) https://bugs.python.org/issue?@action=redirect&bpo=25399
(11) https://bugs.python.org/issue?@action=redirect&bpo=25401
(12) https://bugs.python.org/issue?@action=redirect&bpo=26574
(13) https://bugs.python.org/issue?@action=redirect&bpo=26249
(14) https://bugs.python.org/issue?@action=redirect&bpo=27056
(15) https://bugs.python.org/issue?@action=redirect&bpo=27574
(16) https://bugs.python.org/issue?@action=redirect&bpo=25596
(17) https://bugs.python.org/issue?@action=redirect&bpo=26032
(18) https://bugs.python.org/issue?@action=redirect&bpo=25638
(19) https://bugs.python.org/issue?@action=redirect&bpo=25873
(20) https://bugs.python.org/issue?@action=redirect&bpo=25869
(21) https://bugs.python.org/issue?@action=redirect&bpo=25971
File: python.info, Node: Build and C API Changes<2>, Next: Other Improvements, Prev: Optimizations<8>, Up: What’s New In Python 3 6
1.8.7 Build and C API Changes
-----------------------------
* Python now requires some C99 support in the toolchain to build.
Most notably, Python now uses standard integer types and macros in
place of custom macros like ‘PY_LONG_LONG’. For more information,
see PEP 7(1) and bpo-17884(2).
* Cross-compiling CPython with the Android NDK and the Android API
level set to 21 (Android 5.0 Lollipop) or greater runs
successfully. While Android is not yet a supported platform, the
Python test suite runs on the Android emulator with only about 16
tests failures. See the Android meta-issue bpo-26865(3).
* The ‘--enable-optimizations’ configure flag has been added.
Turning it on will activate expensive optimizations like PGO.
(Original patch by Alecsandru Patrascu of Intel in bpo-26359(4).)
* The *note GIL: 148. must now be held when allocator functions of
*note PYMEM_DOMAIN_OBJ: c67. (ex: *note PyObject_Malloc(): c68.)
and *note PYMEM_DOMAIN_MEM: c69. (ex: *note PyMem_Malloc(): c66.)
domains are called.
* New *note Py_FinalizeEx(): d15. API which indicates if flushing
buffered data failed. (Contributed by Martin Panter in
bpo-5319(5).)
* *note PyArg_ParseTupleAndKeywords(): 37e. now supports *note
positional-only parameters: a85. Positional-only parameters are
defined by empty names. (Contributed by Serhiy Storchaka in
bpo-26282(6)).
* ‘PyTraceback_Print’ method now abbreviates long sequences of
repeated lines as ‘"[Previous line repeated {count} more times]"’.
(Contributed by Emanuel Barry in bpo-26823(7).)
* The new *note PyErr_SetImportErrorSubclass(): d16. function allows
for specifying a subclass of *note ImportError: 415. to raise.
(Contributed by Eric Snow in bpo-15767(8).)
* The new *note PyErr_ResourceWarning(): d17. function can be used to
generate a *note ResourceWarning: 246. providing the source of the
resource allocation. (Contributed by Victor Stinner in
bpo-26567(9).)
* The new *note PyOS_FSPath(): d18. function returns the file system
representation of a *note path-like object: 2a2. (Contributed by
Brett Cannon in bpo-27186(10).)
* The *note PyUnicode_FSConverter(): d19. and *note
PyUnicode_FSDecoder(): 574. functions will now accept *note
path-like objects: 2a2.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0007/
(2) https://bugs.python.org/issue?@action=redirect&bpo=17884
(3) https://bugs.python.org/issue?@action=redirect&bpo=26865
(4) https://bugs.python.org/issue?@action=redirect&bpo=26359
(5) https://bugs.python.org/issue?@action=redirect&bpo=5319
(6) https://bugs.python.org/issue?@action=redirect&bpo=26282
(7) https://bugs.python.org/issue?@action=redirect&bpo=26823
(8) https://bugs.python.org/issue?@action=redirect&bpo=15767
(9) https://bugs.python.org/issue?@action=redirect&bpo=26567
(10) https://bugs.python.org/issue?@action=redirect&bpo=27186
File: python.info, Node: Other Improvements, Next: Deprecated<9>, Prev: Build and C API Changes<2>, Up: What’s New In Python 3 6
1.8.8 Other Improvements
------------------------
* When *note -version: d1b. (short form: *note -V: 177.) is supplied
twice, Python prints *note sys.version: 178. for detailed
information.
$ ./python -VV
Python 3.6.0b4+ (3.6:223967b49e49+, Nov 21 2016, 20:55:04)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
File: python.info, Node: Deprecated<9>, Next: Removed<9>, Prev: Other Improvements, Up: What’s New In Python 3 6
1.8.9 Deprecated
----------------
* Menu:
* New Keywords::
* Deprecated Python behavior::
* Deprecated Python modules, functions and methods: Deprecated Python modules functions and methods<2>.
* xml: xml<6>.
* Deprecated functions and types of the C API: Deprecated functions and types of the C API<2>.
* Deprecated Build Options::
File: python.info, Node: New Keywords, Next: Deprecated Python behavior, Up: Deprecated<9>
1.8.9.1 New Keywords
....................
‘async’ and ‘await’ are not recommended to be used as variable, class,
function or module names. Introduced by PEP 492(1) in Python 3.5, they
will become proper keywords in Python 3.7. Starting in Python 3.6, the
use of ‘async’ or ‘await’ as names will generate a *note
DeprecationWarning: 1a5.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0492/
File: python.info, Node: Deprecated Python behavior, Next: Deprecated Python modules functions and methods<2>, Prev: New Keywords, Up: Deprecated<9>
1.8.9.2 Deprecated Python behavior
..................................
Raising the *note StopIteration: bfa. exception inside a generator will
now generate a *note DeprecationWarning: 1a5, and will trigger a *note
RuntimeError: 195. in Python 3.7. See *note PEP 479; Change
StopIteration handling inside generators: d1f. for details.
The ‘__aiter__()’ method is now expected to return an asynchronous
iterator directly instead of returning an awaitable as previously.
Doing the former will trigger a *note DeprecationWarning: 1a5. Backward
compatibility will be removed in Python 3.7. (Contributed by Yury
Selivanov in bpo-27243(1).)
A backslash-character pair that is not a valid escape sequence now
generates a *note DeprecationWarning: 1a5. Although this will
eventually become a *note SyntaxError: 18d, that will not be for several
Python releases. (Contributed by Emanuel Barry in bpo-27364(2).)
When performing a relative import, falling back on ‘__name__’ and
‘__path__’ from the calling module when ‘__spec__’ or ‘__package__’ are
not defined now raises an *note ImportWarning: 4f6. (Contributed by
Rose Ames in bpo-25791(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27243
(2) https://bugs.python.org/issue?@action=redirect&bpo=27364
(3) https://bugs.python.org/issue?@action=redirect&bpo=25791
File: python.info, Node: Deprecated Python modules functions and methods<2>, Next: xml<6>, Prev: Deprecated Python behavior, Up: Deprecated<9>
1.8.9.3 Deprecated Python modules, functions and methods
........................................................
* Menu:
* asynchat::
* asyncore::
* dbm: dbm<4>.
* distutils: distutils<6>.
* grp::
* importlib: importlib<7>.
* os: os<9>.
* re: re<6>.
* ssl: ssl<8>.
* tkinter: tkinter<7>.
* venv: venv<6>.
File: python.info, Node: asynchat, Next: asyncore, Up: Deprecated Python modules functions and methods<2>
1.8.9.4 asynchat
................
The ‘asynchat’ has been deprecated in favor of *note asyncio: a.
(Contributed by Mariatta in bpo-25002(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25002
File: python.info, Node: asyncore, Next: dbm<4>, Prev: asynchat, Up: Deprecated Python modules functions and methods<2>
1.8.9.5 asyncore
................
The ‘asyncore’ has been deprecated in favor of *note asyncio: a.
(Contributed by Mariatta in bpo-25002(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25002
File: python.info, Node: dbm<4>, Next: distutils<6>, Prev: asyncore, Up: Deprecated Python modules functions and methods<2>
1.8.9.6 dbm
...........
Unlike other *note dbm: 31. implementations, the *note dbm.dumb: 32.
module creates databases with the ‘'rw'’ mode and allows modifying the
database opened with the ‘'r'’ mode. This behavior is now deprecated
and will be removed in 3.8. (Contributed by Serhiy Storchaka in
bpo-21708(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21708
File: python.info, Node: distutils<6>, Next: grp, Prev: dbm<4>, Up: Deprecated Python modules functions and methods<2>
1.8.9.7 distutils
.................
The undocumented ‘extra_path’ argument to the ‘distutils.Distribution’
constructor is now considered deprecated and will raise a warning if
set. Support for this parameter will be removed in a future Python
release. See bpo-27919(1) for details.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=27919
File: python.info, Node: grp, Next: importlib<7>, Prev: distutils<6>, Up: Deprecated Python modules functions and methods<2>
1.8.9.8 grp
...........
The support of non-integer arguments in *note getgrgid(): d26. has been
deprecated. (Contributed by Serhiy Storchaka in bpo-26129(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26129
File: python.info, Node: importlib<7>, Next: os<9>, Prev: grp, Up: Deprecated Python modules functions and methods<2>
1.8.9.9 importlib
.................
The *note importlib.machinery.SourceFileLoader.load_module(): d28. and
*note importlib.machinery.SourcelessFileLoader.load_module(): d29.
methods are now deprecated. They were the only remaining
implementations of *note importlib.abc.Loader.load_module(): 866. in
*note importlib: 77. that had not been deprecated in previous versions
of Python in favour of *note importlib.abc.Loader.exec_module(): 867.
The *note importlib.machinery.WindowsRegistryFinder: d2a. class is now
deprecated. As of 3.6.0, it is still added to *note sys.meta_path: d2b.
by default (on Windows), but this may change in future releases.
File: python.info, Node: os<9>, Next: re<6>, Prev: importlib<7>, Up: Deprecated Python modules functions and methods<2>
1.8.9.10 os
...........
Undocumented support of general *note bytes-like objects: d2d. as paths
in *note os: a1. functions, *note compile(): 192. and similar functions
is now deprecated. (Contributed by Serhiy Storchaka in bpo-25791(1) and
bpo-26754(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25791
(2) https://bugs.python.org/issue?@action=redirect&bpo=26754
File: python.info, Node: re<6>, Next: ssl<8>, Prev: os<9>, Up: Deprecated Python modules functions and methods<2>
1.8.9.11 re
...........
Support for inline flags ‘(?letters)’ in the middle of the regular
expression has been deprecated and will be removed in a future Python
version. Flags at the start of a regular expression are still allowed.
(Contributed by Serhiy Storchaka in bpo-22493(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22493
File: python.info, Node: ssl<8>, Next: tkinter<7>, Prev: re<6>, Up: Deprecated Python modules functions and methods<2>
1.8.9.12 ssl
............
OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported.
In the future the *note ssl: d0. module will require at least OpenSSL
1.0.2 or 1.1.0.
SSL-related arguments like ‘certfile’, ‘keyfile’ and ‘check_hostname’ in
*note ftplib: 5e, *note http.client: 6f, *note imaplib: 74, *note
poplib: ac, and *note smtplib: ca. have been deprecated in favor of
‘context’. (Contributed by Christian Heimes in bpo-28022(1).)
A couple of protocols and functions of the *note ssl: d0. module are now
deprecated. Some features will no longer be available in future
versions of OpenSSL. Other features are deprecated in favor of a
different API. (Contributed by Christian Heimes in bpo-28022(2) and
bpo-26470(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=28022
(2) https://bugs.python.org/issue?@action=redirect&bpo=28022
(3) https://bugs.python.org/issue?@action=redirect&bpo=26470
File: python.info, Node: tkinter<7>, Next: venv<6>, Prev: ssl<8>, Up: Deprecated Python modules functions and methods<2>
1.8.9.13 tkinter
................
The ‘tkinter.tix’ module is now deprecated. *note tkinter: f0. users
should use *note tkinter.ttk: f9. instead.
File: python.info, Node: venv<6>, Prev: tkinter<7>, Up: Deprecated Python modules functions and methods<2>
1.8.9.14 venv
.............
The ‘pyvenv’ script has been deprecated in favour of ‘python3 -m venv’.
This prevents confusion as to what Python interpreter ‘pyvenv’ is
connected to and thus what Python interpreter will be used by the
virtual environment. (Contributed by Brett Cannon in bpo-25154(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=25154
File: python.info, Node: xml<6>, Next: Deprecated functions and types of the C API<2>, Prev: Deprecated Python modules functions and methods<2>, Up: Deprecated<9>
1.8.9.15 xml
............
* As mitigation against DTD and external entity retrieval, the *note
xml.dom.minidom: 122. and *note xml.sax: 129. modules no longer
process external entities by default. (Contributed by Christian
Heimes in gh-61441(1).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/61441
File: python.info, Node: Deprecated functions and types of the C API<2>, Next: Deprecated Build Options, Prev: xml<6>, Up: Deprecated<9>
1.8.9.16 Deprecated functions and types of the C API
....................................................
Undocumented functions ‘PyUnicode_AsEncodedObject()’,
‘PyUnicode_AsDecodedObject()’, ‘PyUnicode_AsEncodedUnicode()’ and
‘PyUnicode_AsDecodedUnicode()’ are deprecated now. Use the *note
generic codec based API: d35. instead.
File: python.info, Node: Deprecated Build Options, Prev: Deprecated functions and types of the C API<2>, Up: Deprecated<9>
1.8.9.17 Deprecated Build Options
.................................
The ‘--with-system-ffi’ configure flag is now on by default on non-macOS
UNIX platforms. It may be disabled by using ‘--without-system-ffi’, but
using the flag is deprecated and will not be accepted in Python 3.7.
macOS is unaffected by this change. Note that many OS distributors
already use the ‘--with-system-ffi’ flag when building their system
Python.
File: python.info, Node: Removed<9>, Next: Porting to Python 3 6, Prev: Deprecated<9>, Up: What’s New In Python 3 6
1.8.10 Removed
--------------
* Menu:
* API and Feature Removals: API and Feature Removals<3>.
File: python.info, Node: API and Feature Removals<3>, Up: Removed<9>
1.8.10.1 API and Feature Removals
.................................
* Unknown escapes consisting of ‘'\'’ and an ASCII letter in regular
expressions will now cause an error. In replacement templates for
*note re.sub(): 2a5. they are still allowed, but deprecated. The
*note re.LOCALE: b6c. flag can now only be used with binary
patterns.
* ‘inspect.getmoduleinfo()’ was removed (was deprecated since CPython
3.3). *note inspect.getmodulename(): d39. should be used for
obtaining the module name for a given path. (Contributed by Yury
Selivanov in bpo-13248(1).)
* ‘traceback.Ignore’ class and ‘traceback.usage’,
‘traceback.modname’, ‘traceback.fullmodname’,
‘traceback.find_lines_from_code’, ‘traceback.find_lines’,
‘traceback.find_strings’, ‘traceback.find_executable_lines’ methods
were removed from the *note traceback: fe. module. They were
undocumented methods deprecated since Python 3.2 and equivalent
functionality is available from private methods.
* The ‘tk_menuBar()’ and ‘tk_bindForTraversal()’ dummy methods in
*note tkinter: f0. widget classes were removed (corresponding Tk
commands were obsolete since Tk 4.0).
* The *note open(): 417. method of the *note zipfile.ZipFile: 6c0.
class no longer supports the ‘'U'’ mode (was deprecated since
Python 3.4). Use *note io.TextIOWrapper: 7b6. for reading
compressed text files in *note universal newlines: d3a. mode.
* The undocumented ‘IN’, ‘CDROM’, ‘DLFCN’, ‘TYPES’, ‘CDIO’, and
‘STROPTS’ modules have been removed. They had been available in
the platform specific ‘Lib/plat-*/’ directories, but were
chronically out of date, inconsistently available across platforms,
and unmaintained. The script that created these modules is still
available in the source distribution at Tools/scripts/h2py.py(2).
* The deprecated ‘asynchat.fifo’ class has been removed.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13248
(2)
https://github.com/python/cpython/blob/v3.6.15/Tools/scripts/h2py.py
File: python.info, Node: Porting to Python 3 6, Next: Notable changes in Python 3 6 2, Prev: Removed<9>, Up: What’s New In Python 3 6
1.8.11 Porting to Python 3.6
----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in ‘python’ Command Behavior::
* Changes in the Python API: Changes in the Python API<8>.
* Changes in the C API: Changes in the C API<6>.
* CPython bytecode changes: CPython bytecode changes<7>.
File: python.info, Node: Changes in ‘python’ Command Behavior, Next: Changes in the Python API<8>, Up: Porting to Python 3 6
1.8.11.1 Changes in ‘python’ Command Behavior
.............................................
* The output of a special Python build with defined ‘COUNT_ALLOCS’,
‘SHOW_ALLOC_COUNT’ or ‘SHOW_TRACK_COUNT’ macros is now off by
default. It can be re-enabled using the ‘-X showalloccount’
option. It now outputs to ‘stderr’ instead of ‘stdout’.
(Contributed by Serhiy Storchaka in bpo-23034(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23034
File: python.info, Node: Changes in the Python API<8>, Next: Changes in the C API<6>, Prev: Changes in ‘python’ Command Behavior, Up: Porting to Python 3 6
1.8.11.2 Changes in the Python API
..................................
* *note open(): 517. will no longer allow combining the ‘'U'’ mode
flag with ‘'+'’. (Contributed by Jeff Balogh and John O’Connor in
bpo-2091(1).)
* *note sqlite3: cf. no longer implicitly commits an open transaction
before DDL statements.
* On Linux, *note os.urandom(): 51e. now blocks until the system
urandom entropy pool is initialized to increase the security.
* When *note importlib.abc.Loader.exec_module(): 867. is defined,
*note importlib.abc.Loader.create_module(): cae. must also be
defined.
* *note PyErr_SetImportError(): d3e. now sets *note TypeError: 534.
when its 'msg' argument is not set. Previously only ‘NULL’ was
returned.
* The format of the *note co_lnotab: 2e4. attribute of code objects
changed to support a negative line number delta. By default,
Python does not emit bytecode with a negative line number delta.
Functions using *note frame.f_lineno: 7a7,
‘PyFrame_GetLineNumber()’ or ‘PyCode_Addr2Line()’ are not affected.
Functions directly decoding ‘co_lnotab’ should be updated to use a
signed 8-bit integer type for the line number delta, but this is
only required to support applications using a negative line number
delta. See ‘Objects/lnotab_notes.txt’ for the ‘co_lnotab’ format
and how to decode it, and see the PEP 511(2) for the rationale.
* The functions in the *note compileall: 20. module now return
booleans instead of ‘1’ or ‘0’ to represent success or failure,
respectively. Thanks to booleans being a subclass of integers,
this should only be an issue if you were doing identity checks for
‘1’ or ‘0’. See bpo-25768(3).
* Reading the ‘port’ attribute of *note urllib.parse.urlsplit(): d3f.
and *note urlparse(): 307. results now raises *note ValueError:
204. for out-of-range values, rather than returning *note None:
671. See bpo-20059(4).
* The ‘imp’ module now raises a *note DeprecationWarning: 1a5.
instead of *note PendingDeprecationWarning: 8c7.
* The following modules have had missing APIs added to their
‘__all__’ attributes to match the documented APIs: *note calendar:
14, ‘cgi’, *note csv: 29, *note ElementTree: 125, *note enum: 56,
*note fileinput: 5b, *note ftplib: 5e, *note logging: 87, *note
mailbox: 8b, *note mimetypes: 8f, *note optparse: a0, *note
plistlib: ab, ‘smtpd’, *note subprocess: d6, *note tarfile: de,
*note threading: ed. and *note wave: 113. This means they will
export new symbols when ‘import *’ is used. (Contributed by Joel
Taddei and Jacek Kołodziej in bpo-23883(5).)
* When performing a relative import, if ‘__package__’ does not
compare equal to ‘__spec__.parent’ then *note ImportWarning: 4f6.
is raised. (Contributed by Brett Cannon in bpo-25791(6).)
* When a relative import is performed and no parent package is known,
then *note ImportError: 415. will be raised. Previously, *note
SystemError: 572. could be raised. (Contributed by Brett Cannon in
bpo-18018(7).)
* Servers based on the *note socketserver: cd. module, including
those defined in *note http.server: 72, *note xmlrpc.server: 12f.
and *note wsgiref.simple_server: 11b, now only catch exceptions
derived from *note Exception: 9d9. Therefore if a request handler
raises an exception like *note SystemExit: d40. or *note
KeyboardInterrupt: 9d1, *note handle_error(): d41. is no longer
called, and the exception will stop a single-threaded server.
(Contributed by Martin Panter in bpo-23430(8).)
* ‘spwd.getspnam()’ now raises a *note PermissionError: d42. instead
of *note KeyError: 33f. if the user doesn’t have privileges.
* The *note socket.socket.close(): d43. method now raises an
exception if an error (e.g. ‘EBADF’) was reported by the
underlying system call. (Contributed by Martin Panter in
bpo-26685(9).)
* The 'decode_data' argument for the ‘smtpd.SMTPChannel’ and
‘smtpd.SMTPServer’ constructors is now ‘False’ by default. This
means that the argument passed to ‘process_message()’ is now a
bytes object by default, and ‘process_message()’ will be passed
keyword arguments. Code that has already been updated in
accordance with the deprecation warning generated by 3.5 will not
be affected.
* All optional arguments of the *note dump(): d44, *note dumps():
d45, *note load(): cb4. and *note loads(): 955. functions and *note
JSONEncoder: d46. and *note JSONDecoder: d47. class constructors in
the *note json: 82. module are now *note keyword-only: 2a7.
(Contributed by Serhiy Storchaka in bpo-18726(10).)
* Subclasses of *note type: d48. which don’t override ‘type.__new__’
may no longer use the one-argument form to get the type of an
object.
* As part of PEP 487(11), the handling of keyword arguments passed to
*note type: d48. (other than the metaclass hint, ‘metaclass’) is
now consistently delegated to *note object.__init_subclass__():
62d. This means that ‘type.__new__()’ and ‘type.__init__()’ both
now accept arbitrary keyword arguments, but *note
object.__init_subclass__(): 62d. (which is called from
‘type.__new__()’) will reject them by default. Custom metaclasses
accepting additional keyword arguments will need to adjust their
calls to ‘type.__new__()’ (whether direct or via *note super: 4d7.)
accordingly.
* In ‘distutils.command.sdist.sdist’, the ‘default_format’ attribute
has been removed and is no longer honored. Instead, the gzipped
tarfile format is the default on all platforms and no
platform-specific selection is made. In environments where
distributions are built on Windows and zip distributions are
required, configure the project with a ‘setup.cfg’ file containing
the following:
[sdist]
formats=zip
This behavior has also been backported to earlier Python versions
by Setuptools 26.0.0.
* In the *note urllib.request: 10b. module and the *note
http.client.HTTPConnection.request(): ca9. method, if no
Content-Length header field has been specified and the request body
is a file object, it is now sent with HTTP 1.1 chunked encoding.
If a file object has to be sent to a HTTP 1.0 server, the
Content-Length value now has to be specified by the caller.
(Contributed by Demian Brecht and Rolf Krahl with tweaks from
Martin Panter in bpo-12319(12).)
* The *note DictReader: 9e4. now returns rows of type *note
OrderedDict: 5d7. (Contributed by Steve Holden in bpo-27842(13).)
* The ‘crypt.METHOD_CRYPT’ will no longer be added to ‘crypt.methods’
if unsupported by the platform. (Contributed by Victor Stinner in
bpo-25287(14).)
* The 'verbose' and 'rename' arguments for *note namedtuple(): 1ca.
are now keyword-only. (Contributed by Raymond Hettinger in
bpo-25628(15).)
* On Linux, *note ctypes.util.find_library(): d49. now looks in
‘LD_LIBRARY_PATH’ for shared libraries. (Contributed by Vinay
Sajip in bpo-9998(16).)
* The *note imaplib.IMAP4: 902. class now handles flags containing
the ‘']'’ character in messages sent from the server to improve
real-world compatibility. (Contributed by Lita Cho in
bpo-21815(17).)
* The ‘mmap.write()’ function now returns the number of bytes written
like other write methods. (Contributed by Jakub Stasiak in
bpo-26335(18).)
* The *note pkgutil.iter_modules(): d4a. and *note
pkgutil.walk_packages(): bff. functions now return *note
ModuleInfo: d4b. named tuples. (Contributed by Ramchandra Apte in
bpo-17211(19).)
* *note re.sub(): 2a5. now raises an error for invalid numerical
group references in replacement templates even if the pattern is
not found in the string. The error message for invalid group
references now includes the group index and the position of the
reference. (Contributed by SilentGhost, Serhiy Storchaka in
bpo-25953(20).)
* *note zipfile.ZipFile: 6c0. will now raise *note
NotImplementedError: 22a. for unrecognized compression values.
Previously a plain *note RuntimeError: 195. was raised.
Additionally, calling *note ZipFile: 6c0. methods on a closed
ZipFile or calling the *note write(): d4c. method on a ZipFile
created with mode ‘'r'’ will raise a *note ValueError: 204.
Previously, a *note RuntimeError: 195. was raised in those
scenarios.
* when custom metaclasses are combined with zero-argument *note
super(): 4d7. or direct references from methods to the implicit
‘__class__’ closure variable, the implicit ‘__classcell__’
namespace entry must now be passed up to ‘type.__new__’ for
initialisation. Failing to do so will result in a *note
DeprecationWarning: 1a5. in Python 3.6 and a *note RuntimeError:
195. in Python 3.8.
* With the introduction of *note ModuleNotFoundError: b47, import
system consumers may start expecting import system replacements to
raise that more specific exception when appropriate, rather than
the less-specific *note ImportError: 415. To provide future
compatibility with such consumers, implementers of alternative
import systems that completely replace *note __import__(): 8d3.
will need to update their implementations to raise the new subclass
when a module can’t be found at all. Implementers of compliant
plugins to the default import system shouldn’t need to make any
changes, as the default import system will raise the new subclass
when appropriate.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2091
(2) https://peps.python.org/pep-0511/
(3) https://bugs.python.org/issue?@action=redirect&bpo=25768
(4) https://bugs.python.org/issue?@action=redirect&bpo=20059
(5) https://bugs.python.org/issue?@action=redirect&bpo=23883
(6) https://bugs.python.org/issue?@action=redirect&bpo=25791
(7) https://bugs.python.org/issue?@action=redirect&bpo=18018
(8) https://bugs.python.org/issue?@action=redirect&bpo=23430
(9) https://bugs.python.org/issue?@action=redirect&bpo=26685
(10) https://bugs.python.org/issue?@action=redirect&bpo=18726
(11) https://peps.python.org/pep-0487/
(12) https://bugs.python.org/issue?@action=redirect&bpo=12319
(13) https://bugs.python.org/issue?@action=redirect&bpo=27842
(14) https://bugs.python.org/issue?@action=redirect&bpo=25287
(15) https://bugs.python.org/issue?@action=redirect&bpo=25628
(16) https://bugs.python.org/issue?@action=redirect&bpo=9998
(17) https://bugs.python.org/issue?@action=redirect&bpo=21815
(18) https://bugs.python.org/issue?@action=redirect&bpo=26335
(19) https://bugs.python.org/issue?@action=redirect&bpo=17211
(20) https://bugs.python.org/issue?@action=redirect&bpo=25953
File: python.info, Node: Changes in the C API<6>, Next: CPython bytecode changes<7>, Prev: Changes in the Python API<8>, Up: Porting to Python 3 6
1.8.11.3 Changes in the C API
.............................
* The *note PyMem_Malloc(): c66. allocator family now uses the *note
pymalloc allocator: d0f. rather than the system ‘malloc()’.
Applications calling *note PyMem_Malloc(): c66. without holding the
GIL can now crash. Set the *note PYTHONMALLOC: c64. environment
variable to ‘debug’ to validate the usage of memory allocators in
your application. See bpo-26249(1).
* *note Py_Exit(): d4e. (and the main interpreter) now override the
exit status with 120 if flushing buffered data failed. See
bpo-5319(2).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26249
(2) https://bugs.python.org/issue?@action=redirect&bpo=5319
File: python.info, Node: CPython bytecode changes<7>, Prev: Changes in the C API<6>, Up: Porting to Python 3 6
1.8.11.4 CPython bytecode changes
.................................
There have been several major changes to the *note bytecode: 187. in
Python 3.6.
* The Python interpreter now uses a 16-bit wordcode instead of
bytecode. (Contributed by Demur Rumed with input and reviews from
Serhiy Storchaka and Victor Stinner in bpo-26647(1) and
bpo-28050(2).)
* The new ‘FORMAT_VALUE’ and *note BUILD_STRING: d50. opcodes as part
of the *note formatted string literal: c2f. implementation.
(Contributed by Eric Smith in bpo-25483(3) and Serhiy Storchaka in
bpo-27078(4).)
* The new *note BUILD_CONST_KEY_MAP: d51. opcode to optimize the
creation of dictionaries with constant keys. (Contributed by
Serhiy Storchaka in bpo-27140(5).)
* The function call opcodes have been heavily reworked for better
performance and simpler implementation. The *note MAKE_FUNCTION:
d52, ‘CALL_FUNCTION’, ‘CALL_FUNCTION_KW’ and
‘BUILD_MAP_UNPACK_WITH_CALL’ opcodes have been modified, the new
*note CALL_FUNCTION_EX: d53. and ‘BUILD_TUPLE_UNPACK_WITH_CALL’
have been added, and ‘CALL_FUNCTION_VAR’, ‘CALL_FUNCTION_VAR_KW’
and ‘MAKE_CLOSURE’ opcodes have been removed. (Contributed by
Demur Rumed in bpo-27095(6), and Serhiy Storchaka in bpo-27213(7),
bpo-28257(8).)
* The new *note SETUP_ANNOTATIONS: d54. and ‘STORE_ANNOTATION’
opcodes have been added to support the new *note variable
annotation: d55. syntax. (Contributed by Ivan Levkivskyi in
bpo-27985(9).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=26647
(2) https://bugs.python.org/issue?@action=redirect&bpo=28050
(3) https://bugs.python.org/issue?@action=redirect&bpo=25483
(4) https://bugs.python.org/issue?@action=redirect&bpo=27078
(5) https://bugs.python.org/issue?@action=redirect&bpo=27140
(6) https://bugs.python.org/issue?@action=redirect&bpo=27095
(7) https://bugs.python.org/issue?@action=redirect&bpo=27213
(8) https://bugs.python.org/issue?@action=redirect&bpo=28257
(9) https://bugs.python.org/issue?@action=redirect&bpo=27985
File: python.info, Node: Notable changes in Python 3 6 2, Next: Notable changes in Python 3 6 4, Prev: Porting to Python 3 6, Up: What’s New In Python 3 6
1.8.12 Notable changes in Python 3.6.2
--------------------------------------
* Menu:
* New make regen-all build target::
* Removal of make touch build target::
File: python.info, Node: New make regen-all build target, Next: Removal of make touch build target, Up: Notable changes in Python 3 6 2
1.8.12.1 New ‘make regen-all’ build target
..........................................
To simplify cross-compilation, and to ensure that CPython can reliably
be compiled without requiring an existing version of Python to already
be available, the autotools-based build system no longer attempts to
implicitly recompile generated files based on file modification times.
Instead, a new ‘make regen-all’ command has been added to force
regeneration of these files when desired (e.g. after an initial version
of Python has already been built based on the pregenerated versions).
More selective regeneration targets are also defined - see
Makefile.pre.in(1) for details.
(Contributed by Victor Stinner in bpo-23404(2).)
Added in version 3.6.2.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Makefile.pre.in
(2) https://bugs.python.org/issue?@action=redirect&bpo=23404
File: python.info, Node: Removal of make touch build target, Prev: New make regen-all build target, Up: Notable changes in Python 3 6 2
1.8.12.2 Removal of ‘make touch’ build target
.............................................
The ‘make touch’ build target previously used to request implicit
regeneration of generated files by updating their modification times has
been removed.
It has been replaced by the new ‘make regen-all’ target.
(Contributed by Victor Stinner in bpo-23404(1).)
Changed in version 3.6.2.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23404
File: python.info, Node: Notable changes in Python 3 6 4, Next: Notable changes in Python 3 6 5, Prev: Notable changes in Python 3 6 2, Up: What’s New In Python 3 6
1.8.13 Notable changes in Python 3.6.4
--------------------------------------
The ‘PyExc_RecursionErrorInst’ singleton that was part of the public API
has been removed as its members being never cleared may cause a segfault
during finalization of the interpreter. (Contributed by Xavier de Gaye
in bpo-22898(1) and bpo-30697(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22898
(2) https://bugs.python.org/issue?@action=redirect&bpo=30697
File: python.info, Node: Notable changes in Python 3 6 5, Next: Notable changes in Python 3 6 7, Prev: Notable changes in Python 3 6 4, Up: What’s New In Python 3 6
1.8.14 Notable changes in Python 3.6.5
--------------------------------------
The *note locale.localeconv(): bfe. function now sets temporarily the
‘LC_CTYPE’ locale to the ‘LC_NUMERIC’ locale in some cases.
(Contributed by Victor Stinner in bpo-31900(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31900
File: python.info, Node: Notable changes in Python 3 6 7, Next: Notable changes in Python 3 6 10, Prev: Notable changes in Python 3 6 5, Up: What’s New In Python 3 6
1.8.15 Notable changes in Python 3.6.7
--------------------------------------
*note xml.dom.minidom: 122. and *note xml.sax: 129. modules no longer
process external entities by default. See also gh-61441(1).
In 3.6.7 the *note tokenize: fb. module now implicitly emits a ‘NEWLINE’
token when provided with input that does not have a trailing new line.
This behavior now matches what the C tokenizer does internally.
(Contributed by Ammar Askar in bpo-33899(2).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/61441
(2) https://bugs.python.org/issue?@action=redirect&bpo=33899
File: python.info, Node: Notable changes in Python 3 6 10, Next: Notable changes in Python 3 6 13, Prev: Notable changes in Python 3 6 7, Up: What’s New In Python 3 6
1.8.16 Notable changes in Python 3.6.10
---------------------------------------
Due to significant security concerns, the 'reuse_address' parameter of
*note asyncio.loop.create_datagram_endpoint(): 72e. is no longer
supported. This is because of the behavior of the socket option
‘SO_REUSEADDR’ in UDP. For more details, see the documentation for
‘loop.create_datagram_endpoint()’. (Contributed by Kyle Stanley,
Antoine Pitrou, and Yury Selivanov in bpo-37228(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=37228
File: python.info, Node: Notable changes in Python 3 6 13, Next: Notable changes in Python 3 6 14, Prev: Notable changes in Python 3 6 10, Up: What’s New In Python 3 6
1.8.17 Notable changes in Python 3.6.13
---------------------------------------
Earlier Python versions allowed using both ‘;’ and ‘&’ as query
parameter separators in *note urllib.parse.parse_qs(): 27c. and *note
urllib.parse.parse_qsl(): 27b. Due to security concerns, and to conform
with newer W3C recommendations, this has been changed to allow only a
single separator key, with ‘&’ as the default. This change also affects
‘cgi.parse()’ and ‘cgi.parse_multipart()’ as they use the affected
functions internally. For more details, please see their respective
documentation. (Contributed by Adam Goldschmidt, Senthil Kumaran and
Ken Jin in bpo-42967(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=42967
File: python.info, Node: Notable changes in Python 3 6 14, Prev: Notable changes in Python 3 6 13, Up: What’s New In Python 3 6
1.8.18 Notable changes in Python 3.6.14
---------------------------------------
A security fix alters the *note ftplib.FTP: 8f9. behavior to not trust
the IPv4 address sent from the remote server when setting up a passive
data channel. We reuse the ftp server IP address instead. For unusual
code requiring the old behavior, set a ‘trust_server_pasv_ipv4_address’
attribute on your FTP instance to ‘True’. (See gh-87451(1))
The presence of newline or tab characters in parts of a URL allows for
some forms of attacks. Following the WHATWG specification that updates
RFC 3986, ASCII newline ‘\n’, ‘\r’ and tab ‘\t’ characters are stripped
from the URL by the parser *note urllib.parse(): 10a. preventing such
attacks. The removal characters are controlled by a new module level
variable ‘urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE’. (See gh-88048(2))
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/87451
(2) https://github.com/python/cpython/issues/88048
File: python.info, Node: What’s New In Python 3 5, Next: What’s New In Python 3 4, Prev: What’s New In Python 3 6, Up: What’s New in Python
1.9 What’s New In Python 3.5
============================
Editors: Elvis Pranskevichus <>, Yury Selivanov
<>
This article explains the new features in Python 3.5, compared to 3.4.
Python 3.5 was released on September 13, 2015. See the changelog(1)
for a full list of changes.
See also
........
PEP 478(2) - Python 3.5 Release Schedule
* Menu:
* Summary – Release highlights: Summary – Release highlights<7>.
* New Features: New Features<14>.
* Other Language Changes: Other Language Changes<9>.
* New Modules: New Modules<9>.
* Improved Modules: Improved Modules<9>.
* Other module-level changes::
* Optimizations: Optimizations<9>.
* Build and C API Changes: Build and C API Changes<3>.
* Deprecated: Deprecated<10>.
* Removed: Removed<10>.
* Porting to Python 3.5: Porting to Python 3 5.
* Notable changes in Python 3.5.4: Notable changes in Python 3 5 4.
---------- Footnotes ----------
(1) https://docs.python.org/3.5/whatsnew/changelog.html
(2) https://peps.python.org/pep-0478/
File: python.info, Node: Summary – Release highlights<7>, Next: New Features<14>, Up: What’s New In Python 3 5
1.9.1 Summary – Release highlights
----------------------------------
New syntax features:
* *note PEP 492: d62, coroutines with async and await syntax.
* *note PEP 465: d63, a new matrix multiplication operator: ‘a @ b’.
* *note PEP 448: d64, additional unpacking generalizations.
New library modules:
* *note typing: 104.: *note PEP 484 – Type Hints: d65.
* *note zipapp: 130.: *note PEP 441 Improving Python ZIP Application
Support: d66.
New built-in features:
* ‘bytes % args’, ‘bytearray % args’: *note PEP 461: d67. – Adding
‘%’ formatting to bytes and bytearray.
* New *note bytes.hex(): d68, *note bytearray.hex(): d69. and *note
memoryview.hex(): d6a. methods. (Contributed by Arnon Yaari in
bpo-9951(1).)
* *note memoryview: 464. now supports tuple indexing (including
multi-dimensional). (Contributed by Antoine Pitrou in
bpo-23632(2).)
* Generators have a new ‘gi_yieldfrom’ attribute, which returns the
object being iterated by ‘yield from’ expressions. (Contributed by
Benno Leslie and Yury Selivanov in bpo-24450(3).)
* A new *note RecursionError: d6b. exception is now raised when
maximum recursion depth is reached. (Contributed by Georg Brandl
in bpo-19235(4).)
CPython implementation improvements:
* When the ‘LC_TYPE’ locale is the POSIX locale (‘C’ locale), *note
sys.stdin: 539. and *note sys.stdout: ad6. now use the
‘surrogateescape’ error handler, instead of the ‘strict’ error
handler. (Contributed by Victor Stinner in bpo-19977(5).)
* ‘.pyo’ files are no longer used and have been replaced by a more
flexible scheme that includes the optimization level explicitly in
‘.pyc’ name. (See *note PEP 488 overview: d6c.)
* Builtin and extension modules are now initialized in a multi-phase
process, which is similar to how Python modules are loaded. (See
*note PEP 489 overview: d6d.)
Significant improvements in the standard library:
* *note collections.OrderedDict: 5d7. is now *note implemented in C:
d6e, which makes it 4 to 100 times faster.
* The *note ssl: d0. module gained *note support for Memory BIO: d6f,
which decouples SSL protocol handling from network IO.
* The new *note os.scandir(): a5e. function provides a *note better
and significantly faster way: d70. of directory traversal.
* *note functools.lru_cache(): 9ee. has been mostly *note
reimplemented in C: d71, yielding much better performance.
* The new *note subprocess.run(): b86. function provides a *note
streamlined way to run subprocesses: d72.
* The *note traceback: fe. module has been significantly *note
enhanced: d73. for improved performance and developer convenience.
Security improvements:
* SSLv3 is now disabled throughout the standard library. It can
still be enabled by instantiating a *note ssl.SSLContext: 296.
manually. (See bpo-22638(6) for more details; this change was
backported to CPython 3.4 and 2.7.)
* HTTP cookie parsing is now stricter, in order to protect against
potential injection attacks. (Contributed by Antoine Pitrou in
bpo-22796(7).)
Windows improvements:
* A new installer for Windows has replaced the old MSI. See *note
Using Python on Windows: aa7. for more information.
* Windows builds now use Microsoft Visual C++ 14.0, and extension
modules should use the same.
Please read on for a comprehensive list of user-facing changes,
including many other smaller improvements, CPython optimizations,
deprecations, and potential porting issues.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9951
(2) https://bugs.python.org/issue?@action=redirect&bpo=23632
(3) https://bugs.python.org/issue?@action=redirect&bpo=24450
(4) https://bugs.python.org/issue?@action=redirect&bpo=19235
(5) https://bugs.python.org/issue?@action=redirect&bpo=19977
(6) https://bugs.python.org/issue?@action=redirect&bpo=22638
(7) https://bugs.python.org/issue?@action=redirect&bpo=22796
File: python.info, Node: New Features<14>, Next: Other Language Changes<9>, Prev: Summary – Release highlights<7>, Up: What’s New In Python 3 5
1.9.2 New Features
------------------
* Menu:
* PEP 492 - Coroutines with async and await syntax::
* PEP 465 - A dedicated infix operator for matrix multiplication::
* PEP 448 - Additional Unpacking Generalizations::
* PEP 461 - percent formatting support for bytes and bytearray::
* PEP 484 - Type Hints::
* PEP 471 - os.scandir() function – a better and faster directory iterator: PEP 471 - os scandir function – a better and faster directory iterator.
* PEP 475; Retry system calls failing with EINTR: PEP 475 Retry system calls failing with EINTR.
* PEP 479; Change StopIteration handling inside generators: PEP 479 Change StopIteration handling inside generators.
* PEP 485; A function for testing approximate equality: PEP 485 A function for testing approximate equality.
* PEP 486; Make the Python Launcher aware of virtual environments: PEP 486 Make the Python Launcher aware of virtual environments.
* PEP 488; Elimination of PYO files: PEP 488 Elimination of PYO files.
* PEP 489; Multi-phase extension module initialization: PEP 489 Multi-phase extension module initialization.
File: python.info, Node: PEP 492 - Coroutines with async and await syntax, Next: PEP 465 - A dedicated infix operator for matrix multiplication, Up: New Features<14>
1.9.2.1 PEP 492 - Coroutines with async and await syntax
........................................................
PEP 492(1) greatly improves support for asynchronous programming in
Python by adding *note awaitable objects: 1ad, *note coroutine
functions: d76, *note asynchronous iteration: d77, and *note
asynchronous context managers: 5d3.
Coroutine functions are declared using the new *note async def: 5cd.
syntax:
>>> async def coro():
... return 'spam'
Inside a coroutine function, the new *note await: 1b9. expression can be
used to suspend coroutine execution until the result is available. Any
object can be 'awaited', as long as it implements the *note awaitable:
1ad. protocol by defining the ‘__await__()’ method.
PEP 492 also adds *note async for: aab. statement for convenient
iteration over asynchronous iterables.
An example of a rudimentary HTTP client written using the new syntax:
import asyncio
async def http_get(domain):
reader, writer = await asyncio.open_connection(domain, 80)
writer.write(b'\r\n'.join([
b'GET / HTTP/1.1',
b'Host: %b' % domain.encode('latin-1'),
b'Connection: close',
b'', b''
]))
async for line in reader:
print('>>>', line)
writer.close()
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(http_get('example.com'))
finally:
loop.close()
Similarly to asynchronous iteration, there is a new syntax for
asynchronous context managers. The following script:
import asyncio
async def coro(name, lock):
print('coro {}: waiting for lock'.format(name))
async with lock:
print('coro {}: holding the lock'.format(name))
await asyncio.sleep(1)
print('coro {}: releasing the lock'.format(name))
loop = asyncio.get_event_loop()
lock = asyncio.Lock()
coros = asyncio.gather(coro(1, lock), coro(2, lock))
try:
loop.run_until_complete(coros)
finally:
loop.close()
will output:
coro 2: waiting for lock
coro 2: holding the lock
coro 1: waiting for lock
coro 2: releasing the lock
coro 1: holding the lock
coro 1: releasing the lock
Note that both *note async for: aab. and *note async with: 5d1. can only
be used inside a coroutine function declared with *note async def: 5cd.
Coroutine functions are intended to be run inside a compatible event
loop, such as the *note asyncio loop: d78.
Note:
Changed in version 3.5.2: Starting with CPython 3.5.2, ‘__aiter__’
can directly return *note asynchronous iterators: 1ab. Returning
an *note awaitable: 1ad. object will result in a *note
PendingDeprecationWarning: 8c7.
See more details in the *note Asynchronous Iterators: d79.
documentation section.
See also
........
PEP 492(2) – Coroutines with async and await syntax
PEP written and implemented by Yury Selivanov.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0492/
(2) https://peps.python.org/pep-0492/
File: python.info, Node: PEP 465 - A dedicated infix operator for matrix multiplication, Next: PEP 448 - Additional Unpacking Generalizations, Prev: PEP 492 - Coroutines with async and await syntax, Up: New Features<14>
1.9.2.2 PEP 465 - A dedicated infix operator for matrix multiplication
......................................................................
PEP 465(1) adds the ‘@’ infix operator for matrix multiplication.
Currently, no builtin Python types implement the new operator, however,
it can be implemented by defining ‘__matmul__()’, ‘__rmatmul__()’, and
‘__imatmul__()’ for regular, reflected, and in-place matrix
multiplication. The semantics of these methods is similar to that of
methods defining other infix arithmetic operators.
Matrix multiplication is a notably common operation in many fields of
mathematics, science, engineering, and the addition of ‘@’ allows
writing cleaner code:
S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r)
instead of:
S = dot((dot(H, beta) - r).T,
dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r))
NumPy 1.10 has support for the new operator:
>>> import numpy
>>> x = numpy.ones(3)
>>> x
array([ 1., 1., 1.])
>>> m = numpy.eye(3)
>>> m
array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])
>>> x @ m
array([ 1., 1., 1.])
See also
........
PEP 465(2) – A dedicated infix operator for matrix multiplication
PEP written by Nathaniel J. Smith; implemented by Benjamin
Peterson.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0465/
(2) https://peps.python.org/pep-0465/
File: python.info, Node: PEP 448 - Additional Unpacking Generalizations, Next: PEP 461 - percent formatting support for bytes and bytearray, Prev: PEP 465 - A dedicated infix operator for matrix multiplication, Up: New Features<14>
1.9.2.3 PEP 448 - Additional Unpacking Generalizations
......................................................
PEP 448(1) extends the allowed uses of the ‘*’ iterable unpacking
operator and ‘**’ dictionary unpacking operator. It is now possible to
use an arbitrary number of unpackings in *note function calls: d7c.:
>>> print(*[1], *[2], 3, *[4, 5])
1 2 3 4 5
>>> def fn(a, b, c, d):
... print(a, b, c, d)
...
>>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4})
1 2 3 4
Similarly, tuple, list, set, and dictionary displays allow multiple
unpackings (see *note Expression lists: d7d. and *note Dictionary
displays: d7e.):
>>> *range(4), 4
(0, 1, 2, 3, 4)
>>> [*range(4), 4]
[0, 1, 2, 3, 4]
>>> {*range(4), 4, *(5, 6, 7)}
{0, 1, 2, 3, 4, 5, 6, 7}
>>> {'x': 1, **{'y': 2}}
{'x': 1, 'y': 2}
See also
........
PEP 448(2) – Additional Unpacking Generalizations
PEP written by Joshua Landau; implemented by Neil Girdhar, Thomas
Wouters, and Joshua Landau.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0448/
(2) https://peps.python.org/pep-0448/
File: python.info, Node: PEP 461 - percent formatting support for bytes and bytearray, Next: PEP 484 - Type Hints, Prev: PEP 448 - Additional Unpacking Generalizations, Up: New Features<14>
1.9.2.4 PEP 461 - percent formatting support for bytes and bytearray
....................................................................
PEP 461(1) adds support for the ‘%’ *note interpolation operator: d80.
to *note bytes: 1c2. and *note bytearray: 53a.
While interpolation is usually thought of as a string operation, there
are cases where interpolation on ‘bytes’ or ‘bytearrays’ makes sense,
and the work needed to make up for this missing functionality detracts
from the overall readability of the code. This issue is particularly
important when dealing with wire format protocols, which are often a
mixture of binary and ASCII compatible text.
Examples:
>>> b'Hello %b!' % b'World'
b'Hello World!'
>>> b'x=%i y=%f' % (1, 2.5)
b'x=1 y=2.500000'
Unicode is not allowed for ‘%b’, but it is accepted by ‘%a’ (equivalent
of ‘repr(obj).encode('ascii', 'backslashreplace')’):
>>> b'Hello %b!' % 'World'
Traceback (most recent call last):
File "", line 1, in
TypeError: %b requires bytes, or an object that implements __bytes__, not 'str'
>>> b'price: %a' % '10€'
b"price: '10\\u20ac'"
Note that ‘%s’ and ‘%r’ conversion types, although supported, should
only be used in codebases that need compatibility with Python 2.
See also
........
PEP 461(2) – Adding % formatting to bytes and bytearray
PEP written by Ethan Furman; implemented by Neil Schemenauer and
Ethan Furman.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0461/
(2) https://peps.python.org/pep-0461/
File: python.info, Node: PEP 484 - Type Hints, Next: PEP 471 - os scandir function – a better and faster directory iterator, Prev: PEP 461 - percent formatting support for bytes and bytearray, Up: New Features<14>
1.9.2.5 PEP 484 - Type Hints
............................
Function annotation syntax has been a Python feature since version 3.0 (
PEP 3107(1)), however the semantics of annotations has been left
undefined.
Experience has shown that the majority of function annotation uses were
to provide type hints to function parameters and return values. It
became evident that it would be beneficial for Python users, if the
standard library included the base definitions and tools for type
annotations.
PEP 484(2) introduces a *note provisional module: b03. to provide these
standard definitions and tools, along with some conventions for
situations where annotations are not available.
For example, here is a simple function whose argument and return type
are declared in the annotations:
def greeting(name: str) -> str:
return 'Hello ' + name
While these annotations are available at runtime through the usual
‘__annotations__’ attribute, 'no automatic type checking happens at
runtime'. Instead, it is assumed that a separate off-line type checker
(e.g. mypy(3)) will be used for on-demand source code analysis.
The type system supports unions, generic types, and a special type named
*note Any: 6a8. which is consistent with (i.e. assignable to and from)
all types.
See also
........
* *note typing: 104. module documentation
*
PEP 484(4) – Type Hints
PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz
Langa; implemented by Guido van Rossum.
*
PEP 483(5) – The Theory of Type Hints
PEP written by Guido van Rossum
---------- Footnotes ----------
(1) https://peps.python.org/pep-3107/
(2) https://peps.python.org/pep-0484/
(3) https://mypy-lang.org
(4) https://peps.python.org/pep-0484/
(5) https://peps.python.org/pep-0483/
File: python.info, Node: PEP 471 - os scandir function – a better and faster directory iterator, Next: PEP 475 Retry system calls failing with EINTR, Prev: PEP 484 - Type Hints, Up: New Features<14>
1.9.2.6 PEP 471 - os.scandir() function – a better and faster directory iterator
................................................................................
PEP 471(1) adds a new directory iteration function, *note os.scandir():
a5e, to the standard library. Additionally, *note os.walk(): 4a5. is
now implemented using ‘scandir’, which makes it 3 to 5 times faster on
POSIX systems and 7 to 20 times faster on Windows systems. This is
largely achieved by greatly reducing the number of calls to *note
os.stat(): 49c. required to walk a directory tree.
Additionally, ‘scandir’ returns an iterator, as opposed to returning a
list of file names, which improves memory efficiency when iterating over
very large directories.
The following example shows a simple use of *note os.scandir(): a5e. to
display all the files (excluding directories) in the given 'path' that
don’t start with ‘'.'’. The *note entry.is_file(): d83. call will
generally not make an additional system call:
for entry in os.scandir(path):
if not entry.name.startswith('.') and entry.is_file():
print(entry.name)
See also
........
PEP 471(2) – os.scandir() function – a better and faster directory iterator
PEP written and implemented by Ben Hoyt with the help of Victor
Stinner.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0471/
(2) https://peps.python.org/pep-0471/
File: python.info, Node: PEP 475 Retry system calls failing with EINTR, Next: PEP 479 Change StopIteration handling inside generators, Prev: PEP 471 - os scandir function – a better and faster directory iterator, Up: New Features<14>
1.9.2.7 PEP 475: Retry system calls failing with EINTR
......................................................
An *note errno.EINTR: d86. error code is returned whenever a system
call, that is waiting for I/O, is interrupted by a signal. Previously,
Python would raise *note InterruptedError: d87. in such cases. This
meant that, when writing a Python application, the developer had two
choices:
1. Ignore the ‘InterruptedError’.
2. Handle the ‘InterruptedError’ and attempt to restart the
interrupted system call at every call site.
The first option makes an application fail intermittently. The second
option adds a large amount of boilerplate that makes the code nearly
unreadable. Compare:
print("Hello World")
and:
while True:
try:
print("Hello World")
break
except InterruptedError:
continue
PEP 475(1) implements automatic retry of system calls on ‘EINTR’. This
removes the burden of dealing with ‘EINTR’ or *note InterruptedError:
d87. in user code in most situations and makes Python programs,
including the standard library, more robust. Note that the system call
is only retried if the signal handler does not raise an exception.
Below is a list of functions which are now retried when interrupted by a
signal:
* *note open(): 517. and *note io.open(): 518.;
* functions of the *note faulthandler: 58. module;
* *note os: a1. functions: *note fchdir(): d88, *note fchmod(): 21e,
*note fchown(): d89, *note fdatasync(): d8a, *note fstat(): d8b,
*note fstatvfs(): d8c, *note fsync(): d8d, *note ftruncate(): d8e,
*note mkfifo(): d8f, *note mknod(): d90, *note open(): d91, *note
posix_fadvise(): d92, *note posix_fallocate(): d93, *note pread():
b5f, *note pwrite(): b62, *note read(): d94, *note readv(): b5e,
*note sendfile(): b0f, *note wait3(): d95, *note wait4(): d96,
*note wait(): d97, *note waitid(): d98, *note waitpid(): d99, *note
write(): d9a, *note writev(): b61.;
* special cases: *note os.close(): b74. and *note os.dup2(): b63. now
ignore *note EINTR: d86. errors; the syscall is not retried (see
the PEP for the rationale);
* *note select: c1. functions: *note devpoll.poll(): d9b, *note
epoll.poll(): d9c, *note kqueue.control(): d9d, *note poll.poll():
d9e, *note select(): d9f.;
* methods of the *note socket: da0. class: *note accept(): da1, *note
connect(): da2. (except for non-blocking sockets), *note recv():
da3, *note recvfrom(): da4, *note recvmsg(): da5, *note send():
da6, *note sendall(): da7, *note sendmsg(): 472, *note sendto():
da8.;
* *note signal.sigtimedwait(): da9. and *note signal.sigwaitinfo():
daa.;
* *note time.sleep(): 699.
See also
........
PEP 475(2) – Retry system calls failing with EINTR
PEP and implementation written by Charles-François Natali and
Victor Stinner, with the help of Antoine Pitrou (the French
connection).
---------- Footnotes ----------
(1) https://peps.python.org/pep-0475/
(2) https://peps.python.org/pep-0475/
File: python.info, Node: PEP 479 Change StopIteration handling inside generators, Next: PEP 485 A function for testing approximate equality, Prev: PEP 475 Retry system calls failing with EINTR, Up: New Features<14>
1.9.2.8 PEP 479: Change StopIteration handling inside generators
................................................................
The interaction of generators and *note StopIteration: bfa. in Python
3.4 and earlier was sometimes surprising, and could conceal obscure
bugs. Previously, ‘StopIteration’ raised accidentally inside a
generator function was interpreted as the end of the iteration by the
loop construct driving the generator.
PEP 479(1) changes the behavior of generators: when a ‘StopIteration’
exception is raised inside a generator, it is replaced with a *note
RuntimeError: 195. before it exits the generator frame. The main goal
of this change is to ease debugging in the situation where an unguarded
*note next(): 7d3. call raises ‘StopIteration’ and causes the iteration
controlled by the generator to terminate silently. This is particularly
pernicious in combination with the ‘yield from’ construct.
This is a backwards incompatible change, so to enable the new behavior,
a *note __future__: dac. import is necessary:
>>> from __future__ import generator_stop
>>> def gen():
... next(iter([]))
... yield
...
>>> next(gen())
Traceback (most recent call last):
File "", line 2, in gen
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
RuntimeError: generator raised StopIteration
Without a ‘__future__’ import, a *note PendingDeprecationWarning: 8c7.
will be raised whenever a *note StopIteration: bfa. exception is raised
inside a generator.
See also
........
PEP 479(2) – Change StopIteration handling inside generators
PEP written by Chris Angelico and Guido van Rossum. Implemented by
Chris Angelico, Yury Selivanov and Nick Coghlan.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0479/
(2) https://peps.python.org/pep-0479/
File: python.info, Node: PEP 485 A function for testing approximate equality, Next: PEP 486 Make the Python Launcher aware of virtual environments, Prev: PEP 479 Change StopIteration handling inside generators, Up: New Features<14>
1.9.2.9 PEP 485: A function for testing approximate equality
............................................................
PEP 485(1) adds the *note math.isclose(): daf. and *note
cmath.isclose(): db0. functions which tell whether two values are
approximately equal or “close” to each other. Whether or not two values
are considered close is determined according to given absolute and
relative tolerances. Relative tolerance is the maximum allowed
difference between ‘isclose’ arguments, relative to the larger absolute
value:
>>> import math
>>> a = 5.0
>>> b = 4.99998
>>> math.isclose(a, b, rel_tol=1e-5)
True
>>> math.isclose(a, b, rel_tol=1e-6)
False
It is also possible to compare two values using absolute tolerance,
which must be a non-negative value:
>>> import math
>>> a = 5.0
>>> b = 4.99998
>>> math.isclose(a, b, abs_tol=0.00003)
True
>>> math.isclose(a, b, abs_tol=0.00001)
False
See also
........
PEP 485(2) – A function for testing approximate equality
PEP written by Christopher Barker; implemented by Chris Barker and
Tal Einat.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0485/
(2) https://peps.python.org/pep-0485/
File: python.info, Node: PEP 486 Make the Python Launcher aware of virtual environments, Next: PEP 488 Elimination of PYO files, Prev: PEP 485 A function for testing approximate equality, Up: New Features<14>
1.9.2.10 PEP 486: Make the Python Launcher aware of virtual environments
........................................................................
PEP 486(1) makes the Windows launcher (see PEP 397(2)) aware of an
active virtual environment. When the default interpreter would be used
and the ‘VIRTUAL_ENV’ environment variable is set, the interpreter in
the virtual environment will be used.
See also
........
PEP 486(3) – Make the Python Launcher aware of virtual environments
PEP written and implemented by Paul Moore.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0486/
(2) https://peps.python.org/pep-0397/
(3) https://peps.python.org/pep-0486/
File: python.info, Node: PEP 488 Elimination of PYO files, Next: PEP 489 Multi-phase extension module initialization, Prev: PEP 486 Make the Python Launcher aware of virtual environments, Up: New Features<14>
1.9.2.11 PEP 488: Elimination of PYO files
..........................................
PEP 488(1) does away with the concept of ‘.pyo’ files. This means that
‘.pyc’ files represent both unoptimized and optimized bytecode. To
prevent the need to constantly regenerate bytecode files, ‘.pyc’ files
now have an optional ‘opt-’ tag in their name when the bytecode is
optimized. This has the side-effect of no more bytecode file name
clashes when running under either *note -O: db4. or *note -OO: db5.
Consequently, bytecode files generated from *note -O: db4, and *note
-OO: db5. may now exist simultaneously. *note
importlib.util.cache_from_source(): 2f8. has an updated API to help with
this change.
See also
........
PEP 488(2) – Elimination of PYO files
PEP written and implemented by Brett Cannon.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0488/
(2) https://peps.python.org/pep-0488/
File: python.info, Node: PEP 489 Multi-phase extension module initialization, Prev: PEP 488 Elimination of PYO files, Up: New Features<14>
1.9.2.12 PEP 489: Multi-phase extension module initialization
.............................................................
PEP 489(1) updates extension module initialization to take advantage of
the two step module loading mechanism introduced by PEP 451(2) in Python
3.4.
This change brings the import semantics of extension modules that opt-in
to using the new mechanism much closer to those of Python source and
bytecode modules, including the ability to use any valid identifier as a
module name, rather than being restricted to ASCII.
See also
........
PEP 489(3) – Multi-phase extension module initialization
PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan;
implemented by Petr Viktorin.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0489/
(2) https://peps.python.org/pep-0451/
(3) https://peps.python.org/pep-0489/
File: python.info, Node: Other Language Changes<9>, Next: New Modules<9>, Prev: New Features<14>, Up: What’s New In Python 3 5
1.9.3 Other Language Changes
----------------------------
Some smaller changes made to the core Python language are:
* Added the ‘"namereplace"’ error handlers. The ‘"backslashreplace"’
error handlers now work with decoding and translating.
(Contributed by Serhiy Storchaka in bpo-19676(1) and bpo-22286(2).)
* The *note -b: 5df. option now affects comparisons of *note bytes:
1c2. with *note int: 259. (Contributed by Serhiy Storchaka in
bpo-23681(3).)
* New Kazakh ‘kz1048’ and Tajik ‘koi8_t’ *note codecs: db8.
(Contributed by Serhiy Storchaka in bpo-22682(4) and bpo-22681(5).)
* Property docstrings are now writable. This is especially useful
for *note collections.namedtuple(): 1ca. docstrings. (Contributed
by Berker Peksag in bpo-24064(6).)
* Circular imports involving relative imports are now supported.
(Contributed by Brett Cannon and Antoine Pitrou in bpo-17636(7).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19676
(2) https://bugs.python.org/issue?@action=redirect&bpo=22286
(3) https://bugs.python.org/issue?@action=redirect&bpo=23681
(4) https://bugs.python.org/issue?@action=redirect&bpo=22682
(5) https://bugs.python.org/issue?@action=redirect&bpo=22681
(6) https://bugs.python.org/issue?@action=redirect&bpo=24064
(7) https://bugs.python.org/issue?@action=redirect&bpo=17636
File: python.info, Node: New Modules<9>, Next: Improved Modules<9>, Prev: Other Language Changes<9>, Up: What’s New In Python 3 5
1.9.4 New Modules
-----------------
* Menu:
* typing: typing<10>.
* zipapp: zipapp<2>.
File: python.info, Node: typing<10>, Next: zipapp<2>, Up: New Modules<9>
1.9.4.1 typing
..............
The new *note typing: 104. *note provisional: b03. module provides
standard definitions and tools for function type annotations. See *note
Type Hints: d65. for more information.
File: python.info, Node: zipapp<2>, Prev: typing<10>, Up: New Modules<9>
1.9.4.2 zipapp
..............
The new *note zipapp: 130. module (specified in PEP 441(1)) provides an
API and command line tool for creating executable Python Zip
Applications, which were introduced in Python 2.6 in bpo-1739468(2), but
which were not well publicized, either at the time or since.
With the new module, bundling your application is as simple as putting
all the files, including a ‘__main__.py’ file, into a directory ‘myapp’
and running:
$ python -m zipapp myapp
$ python myapp.pyz
The module implementation has been contributed by Paul Moore in
bpo-23491(3).
See also
........
PEP 441(4) – Improving Python ZIP Application Support
---------- Footnotes ----------
(1) https://peps.python.org/pep-0441/
(2) https://bugs.python.org/issue?@action=redirect&bpo=1739468
(3) https://bugs.python.org/issue?@action=redirect&bpo=23491
(4) https://peps.python.org/pep-0441/
File: python.info, Node: Improved Modules<9>, Next: Other module-level changes, Prev: New Modules<9>, Up: What’s New In Python 3 5
1.9.5 Improved Modules
----------------------
* Menu:
* argparse: argparse<4>.
* asyncio: asyncio<10>.
* bz2::
* cgi::
* cmath: cmath<2>.
* code::
* collections: collections<5>.
* collections.abc: collections abc<3>.
* compileall: compileall<4>.
* concurrent.futures: concurrent futures<5>.
* configparser: configparser<4>.
* contextlib: contextlib<5>.
* csv: csv<3>.
* curses: curses<4>.
* dbm: dbm<5>.
* difflib::
* distutils: distutils<7>.
* doctest: doctest<3>.
* email: email<3>.
* enum: enum<8>.
* faulthandler: faulthandler<3>.
* functools: functools<4>.
* glob: glob<3>.
* gzip: gzip<4>.
* heapq::
* http: http<2>.
* http.client: http client<3>.
* idlelib and IDLE: idlelib and IDLE<3>.
* imaplib: imaplib<2>.
* imghdr::
* importlib: importlib<8>.
* inspect: inspect<7>.
* io: io<5>.
* ipaddress: ipaddress<4>.
* json: json<2>.
* linecache: linecache<2>.
* locale: locale<6>.
* logging: logging<5>.
* lzma::
* math: math<8>.
* multiprocessing: multiprocessing<6>.
* operator: operator<2>.
* os: os<10>.
* pathlib: pathlib<10>.
* pickle: pickle<3>.
* poplib: poplib<2>.
* re: re<7>.
* readline: readline<2>.
* selectors::
* shutil: shutil<5>.
* signal: signal<3>.
* smtpd: smtpd<2>.
* smtplib: smtplib<2>.
* sndhdr::
* socket: socket<8>.
* ssl: ssl<9>.
* sqlite3: sqlite3<8>.
* subprocess: subprocess<4>.
* sys: sys<11>.
* sysconfig: sysconfig<2>.
* tarfile: tarfile<7>.
* threading: threading<6>.
* time: time<7>.
* timeit: timeit<2>.
* tkinter: tkinter<8>.
* traceback: traceback<5>.
* types: types<5>.
* unicodedata: unicodedata<8>.
* unittest: unittest<8>.
* unittest.mock: unittest mock<3>.
* urllib: urllib<2>.
* wsgiref::
* xmlrpc: xmlrpc<2>.
* xml.sax: xml sax.
* zipfile: zipfile<4>.
File: python.info, Node: argparse<4>, Next: asyncio<10>, Up: Improved Modules<9>
1.9.5.1 argparse
................
The *note ArgumentParser: 535. class now allows disabling *note
abbreviated usage: dbe. of long options by setting *note allow_abbrev:
dbf. to ‘False’. (Contributed by Jonathan Paugh, Steven Bethard, paul
j3 and Daniel Eriksson in bpo-14910(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14910
File: python.info, Node: asyncio<10>, Next: bz2, Prev: argparse<4>, Up: Improved Modules<9>
1.9.5.2 asyncio
...............
Since the *note asyncio: a. module is *note provisional: b03, all
changes introduced in Python 3.5 have also been backported to Python
3.4.x.
Notable changes in the *note asyncio: a. module since Python 3.4.0:
* New debugging APIs: *note loop.set_debug(): dc1. and *note
loop.get_debug(): dc2. methods. (Contributed by Victor Stinner.)
* The proactor event loop now supports SSL. (Contributed by Antoine
Pitrou and Victor Stinner in bpo-22560(1).)
* A new *note loop.is_closed(): dc3. method to check if the event
loop is closed. (Contributed by Victor Stinner in bpo-21326(2).)
* A new *note loop.create_task(): 1c0. to conveniently create and
schedule a new *note Task: 1be. for a coroutine. The ‘create_task’
method is also used by all asyncio functions that wrap coroutines
into tasks, such as *note asyncio.wait(): 47b, *note
asyncio.gather(): 5f9, etc. (Contributed by Victor Stinner.)
* A new *note transport.get_write_buffer_limits(): dc4. method to
inquire for 'high-' and 'low-' water limits of the flow control.
(Contributed by Victor Stinner.)
* The ‘async()’ function is deprecated in favor of *note
ensure_future(): c76. (Contributed by Yury Selivanov.)
* New *note loop.set_task_factory(): dc5. and *note
loop.get_task_factory(): dc6. methods to customize the task factory
that *note loop.create_task(): 1c0. method uses. (Contributed by
Yury Selivanov.)
* New *note Queue.join(): dc7. and *note Queue.task_done(): dc8.
queue methods. (Contributed by Victor Stinner.)
* The ‘JoinableQueue’ class was removed, in favor of the *note
asyncio.Queue: a76. class. (Contributed by Victor Stinner.)
Updates in 3.5.1:
* The *note ensure_future(): c76. function and all functions that use
it, such as *note loop.run_until_complete(): c77, now accept all
kinds of *note awaitable objects: 1ad. (Contributed by Yury
Selivanov.)
* New *note run_coroutine_threadsafe(): 880. function to submit
coroutines to event loops from other threads. (Contributed by
Vincent Michel.)
* New *note Transport.is_closing(): c78. method to check if the
transport is closing or closed. (Contributed by Yury Selivanov.)
* The *note loop.create_server(): b13. method can now accept a list
of hosts. (Contributed by Yann Sionneau.)
Updates in 3.5.2:
* New *note loop.create_future(): c79. method to create Future
objects. This allows alternative event loop implementations, such
as uvloop(3), to provide a faster *note asyncio.Future: bcd.
implementation. (Contributed by Yury Selivanov.)
* New *note loop.get_exception_handler(): c7a. method to get the
current exception handler. (Contributed by Yury Selivanov.)
* New *note StreamReader.readuntil(): 1b4. method to read data from
the stream until a separator bytes sequence appears. (Contributed
by Mark Korenberg.)
* The *note loop.create_connection(): 5ff. and *note
loop.create_server(): b13. methods are optimized to avoid calling
the system ‘getaddrinfo’ function if the address is already
resolved. (Contributed by A. Jesse Jiryu Davis.)
* The *note loop.sock_connect(sock, address): dc9. no longer requires
the 'address' to be resolved prior to the call. (Contributed by A.
Jesse Jiryu Davis.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22560
(2) https://bugs.python.org/issue?@action=redirect&bpo=21326
(3) https://github.com/MagicStack/uvloop
File: python.info, Node: bz2, Next: cgi, Prev: asyncio<10>, Up: Improved Modules<9>
1.9.5.3 bz2
...........
The *note BZ2Decompressor.decompress: dcb. method now accepts an
optional 'max_length' argument to limit the maximum size of decompressed
data. (Contributed by Nikolaus Rath in bpo-15955(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15955
File: python.info, Node: cgi, Next: cmath<2>, Prev: bz2, Up: Improved Modules<9>
1.9.5.4 cgi
...........
The ‘FieldStorage’ class now supports the *note context manager: 5d0.
protocol. (Contributed by Berker Peksag in bpo-20289(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20289
File: python.info, Node: cmath<2>, Next: code, Prev: cgi, Up: Improved Modules<9>
1.9.5.5 cmath
.............
A new function *note isclose(): db0. provides a way to test for
approximate equality. (Contributed by Chris Barker and Tal Einat in
bpo-24270(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24270
File: python.info, Node: code, Next: collections<5>, Prev: cmath<2>, Up: Improved Modules<9>
1.9.5.6 code
............
The *note InteractiveInterpreter.showtraceback(): dcf. method now prints
the full chained traceback, just like the interactive interpreter.
(Contributed by Claudiu Popa in bpo-17442(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17442
File: python.info, Node: collections<5>, Next: collections abc<3>, Prev: code, Up: Improved Modules<9>
1.9.5.7 collections
...................
The *note OrderedDict: 5d7. class is now implemented in C, which makes
it 4 to 100 times faster. (Contributed by Eric Snow in bpo-16991(1).)
‘OrderedDict.items()’, ‘OrderedDict.keys()’, ‘OrderedDict.values()’
views now support *note reversed(): 862. iteration. (Contributed by
Serhiy Storchaka in bpo-19505(2).)
The *note deque: 5d8. class now defines *note index(): dd1, *note
insert(): dd2, and *note copy(): dd3, and supports the ‘+’ and ‘*’
operators. This allows deques to be recognized as a *note
MutableSequence: 1a1. and improves their substitutability for lists.
(Contributed by Raymond Hettinger in bpo-23704(3).)
Docstrings produced by *note namedtuple(): 1ca. can now be updated:
Point = namedtuple('Point', ['x', 'y'])
Point.__doc__ += ': Cartesian coordinate'
Point.x.__doc__ = 'abscissa'
Point.y.__doc__ = 'ordinate'
(Contributed by Berker Peksag in bpo-24064(4).)
The *note UserString: dd4. class now implements the ‘__getnewargs__()’,
‘__rmod__()’, *note casefold(): dd5, *note format_map(): dd6, *note
isprintable(): dd7, and *note maketrans(): dd8. methods to match the
corresponding methods of *note str: 447. (Contributed by Joe Jevnik in
bpo-22189(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16991
(2) https://bugs.python.org/issue?@action=redirect&bpo=19505
(3) https://bugs.python.org/issue?@action=redirect&bpo=23704
(4) https://bugs.python.org/issue?@action=redirect&bpo=24064
(5) https://bugs.python.org/issue?@action=redirect&bpo=22189
File: python.info, Node: collections abc<3>, Next: compileall<4>, Prev: collections<5>, Up: Improved Modules<9>
1.9.5.8 collections.abc
.......................
The ‘Sequence.index()’ method now accepts 'start' and 'stop' arguments
to match the corresponding methods of *note tuple: 36b, *note list: 60d,
etc. (Contributed by Devin Jeanpierre in bpo-23086(1).)
A new *note Generator: dda. abstract base class. (Contributed by Stefan
Behnel in bpo-24018(2).)
New *note Awaitable: ddb, *note Coroutine: ddc, *note AsyncIterator:
ddd, and *note AsyncIterable: dde. abstract base classes. (Contributed
by Yury Selivanov in bpo-24184(3).)
For earlier Python versions, a backport of the new ABCs is available in
an external PyPI package(4).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23086
(2) https://bugs.python.org/issue?@action=redirect&bpo=24018
(3) https://bugs.python.org/issue?@action=redirect&bpo=24184
(4) https://pypi.org/project/backports_abc/
File: python.info, Node: compileall<4>, Next: concurrent futures<5>, Prev: collections abc<3>, Up: Improved Modules<9>
1.9.5.9 compileall
..................
A new *note compileall: 20. option, ‘-j N’, allows running 'N' workers
simultaneously to perform parallel bytecode compilation. The *note
compile_dir(): b26. function has a corresponding ‘workers’ parameter.
(Contributed by Claudiu Popa in bpo-16104(1).)
Another new option, ‘-r’, allows controlling the maximum recursion level
for subdirectories. (Contributed by Claudiu Popa in bpo-19628(2).)
The ‘-q’ command line option can now be specified more than once, in
which case all output, including errors, will be suppressed. The
corresponding ‘quiet’ parameter in *note compile_dir(): b26, *note
compile_file(): de0, and *note compile_path(): de1. can now accept an
integer value indicating the level of output suppression. (Contributed
by Thomas Kluyver in bpo-21338(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16104
(2) https://bugs.python.org/issue?@action=redirect&bpo=19628
(3) https://bugs.python.org/issue?@action=redirect&bpo=21338
File: python.info, Node: concurrent futures<5>, Next: configparser<4>, Prev: compileall<4>, Up: Improved Modules<9>
1.9.5.10 concurrent.futures
...........................
The *note Executor.map(): de3. method now accepts a 'chunksize' argument
to allow batching of tasks to improve performance when *note
ProcessPoolExecutor(): 8ed. is used. (Contributed by Dan O’Reilly in
bpo-11271(1).)
The number of workers in the *note ThreadPoolExecutor: 73d. constructor
is optional now. The default value is 5 times the number of CPUs.
(Contributed by Claudiu Popa in bpo-21527(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11271
(2) https://bugs.python.org/issue?@action=redirect&bpo=21527
File: python.info, Node: configparser<4>, Next: contextlib<5>, Prev: concurrent futures<5>, Up: Improved Modules<9>
1.9.5.11 configparser
.....................
*note configparser: 22. now provides a way to customize the conversion
of values by specifying a dictionary of converters in the *note
ConfigParser: 1c8. constructor, or by defining them as methods in
‘ConfigParser’ subclasses. Converters defined in a parser instance are
inherited by its section proxies.
Example:
>>> import configparser
>>> conv = {}
>>> conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()]
>>> cfg = configparser.ConfigParser(converters=conv)
>>> cfg.read_string("""
... [s]
... list = a b c d e f g
... """)
>>> cfg.get('s', 'list')
'a b c d e f g'
>>> cfg.getlist('s', 'list')
['a', 'b', 'c', 'd', 'e', 'f', 'g']
>>> section = cfg['s']
>>> section.getlist('list')
['a', 'b', 'c', 'd', 'e', 'f', 'g']
(Contributed by Łukasz Langa in bpo-18159(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18159
File: python.info, Node: contextlib<5>, Next: csv<3>, Prev: configparser<4>, Up: Improved Modules<9>
1.9.5.12 contextlib
...................
The new *note redirect_stderr(): de6. *note context manager: 5d0.
(similar to *note redirect_stdout(): de7.) makes it easier for utility
scripts to handle inflexible APIs that write their output to *note
sys.stderr: 939. and don’t provide any options to redirect it:
>>> import contextlib, io, logging
>>> f = io.StringIO()
>>> with contextlib.redirect_stderr(f):
... logging.warning('warning')
...
>>> f.getvalue()
'WARNING:root:warning\n'
(Contributed by Berker Peksag in bpo-22389(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22389
File: python.info, Node: csv<3>, Next: curses<4>, Prev: contextlib<5>, Up: Improved Modules<9>
1.9.5.13 csv
............
The *note writerow(): de9. method now supports arbitrary iterables, not
just sequences. (Contributed by Serhiy Storchaka in bpo-23171(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23171
File: python.info, Node: curses<4>, Next: dbm<5>, Prev: csv<3>, Up: Improved Modules<9>
1.9.5.14 curses
...............
The new *note update_lines_cols(): deb. function updates the ‘LINES’ and
‘COLS’ module variables. This is useful for detecting manual screen
resizing. (Contributed by Arnon Yaari in bpo-4254(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4254
File: python.info, Node: dbm<5>, Next: difflib, Prev: curses<4>, Up: Improved Modules<9>
1.9.5.15 dbm
............
*note dumb.open: a96. always creates a new database when the flag has
the value ‘"n"’. (Contributed by Claudiu Popa in bpo-18039(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18039
File: python.info, Node: difflib, Next: distutils<7>, Prev: dbm<5>, Up: Improved Modules<9>
1.9.5.16 difflib
................
The charset of HTML documents generated by *note HtmlDiff.make_file():
dee. can now be customized by using a new 'charset' keyword-only
argument. The default charset of HTML document changed from
‘"ISO-8859-1"’ to ‘"utf-8"’. (Contributed by Berker Peksag in
bpo-2052(1).)
The *note diff_bytes(): def. function can now compare lists of byte
strings. This fixes a regression from Python 2. (Contributed by Terry
J. Reedy and Greg Ward in bpo-17445(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2052
(2) https://bugs.python.org/issue?@action=redirect&bpo=17445
File: python.info, Node: distutils<7>, Next: doctest<3>, Prev: difflib, Up: Improved Modules<9>
1.9.5.17 distutils
..................
Both the ‘build’ and ‘build_ext’ commands now accept a ‘-j’ option to
enable parallel building of extension modules. (Contributed by Antoine
Pitrou in bpo-5309(1).)
The ‘distutils’ module now supports ‘xz’ compression, and can be enabled
by passing ‘xztar’ as an argument to ‘bdist --format’. (Contributed by
Serhiy Storchaka in bpo-16314(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5309
(2) https://bugs.python.org/issue?@action=redirect&bpo=16314
File: python.info, Node: doctest<3>, Next: email<3>, Prev: distutils<7>, Up: Improved Modules<9>
1.9.5.18 doctest
................
The *note DocTestSuite(): df2. function returns an empty *note
unittest.TestSuite: df3. if 'module' contains no docstrings, instead of
raising *note ValueError: 204. (Contributed by Glenn Jones in
bpo-15916(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15916
File: python.info, Node: email<3>, Next: enum<8>, Prev: doctest<3>, Up: Improved Modules<9>
1.9.5.19 email
..............
A new policy option *note Policy.mangle_from_: df5. controls whether or
not lines that start with ‘"From "’ in email bodies are prefixed with a
‘">"’ character by generators. The default is ‘True’ for *note
compat32: c99. and ‘False’ for all other policies. (Contributed by
Milan Oberkirch in bpo-20098(1).)
A new *note Message.get_content_disposition(): df6. method provides easy
access to a canonical value for the ‘Content-Disposition’ header.
(Contributed by Abhilash Raj in bpo-21083(2).)
A new policy option *note EmailPolicy.utf8: df7. can be set to ‘True’ to
encode email headers using the UTF-8 charset instead of using encoded
words. This allows ‘Messages’ to be formatted according to RFC 6532(3)
and used with an SMTP server that supports the RFC 6531(4) ‘SMTPUTF8’
extension. (Contributed by R. David Murray in bpo-24211(5).)
The *note mime.text.MIMEText: df8. constructor now accepts a *note
charset.Charset: df9. instance. (Contributed by Claude Paroz and Berker
Peksag in bpo-16324(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20098
(2) https://bugs.python.org/issue?@action=redirect&bpo=21083
(3) https://datatracker.ietf.org/doc/html/rfc6532.html
(4) https://datatracker.ietf.org/doc/html/rfc6531.html
(5) https://bugs.python.org/issue?@action=redirect&bpo=24211
(6) https://bugs.python.org/issue?@action=redirect&bpo=16324
File: python.info, Node: enum<8>, Next: faulthandler<3>, Prev: email<3>, Up: Improved Modules<9>
1.9.5.20 enum
.............
The *note Enum: 62c. callable has a new parameter 'start' to specify the
initial number of enum values if only 'names' are provided:
>>> Animal = enum.Enum('Animal', 'cat dog', start=10)
>>> Animal.cat
>>> Animal.dog
(Contributed by Ethan Furman in bpo-21706(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21706
File: python.info, Node: faulthandler<3>, Next: functools<4>, Prev: enum<8>, Up: Improved Modules<9>
1.9.5.21 faulthandler
.....................
The *note enable(): c9e, *note register(): dfc, *note dump_traceback():
dfd. and *note dump_traceback_later(): dfe. functions now accept file
descriptors in addition to file-like objects. (Contributed by Wei Wu in
bpo-23566(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23566
File: python.info, Node: functools<4>, Next: glob<3>, Prev: faulthandler<3>, Up: Improved Modules<9>
1.9.5.22 functools
..................
Most of the *note lru_cache(): 9ee. machinery is now implemented in C,
making it significantly faster. (Contributed by Matt Joiner, Alexey
Kachayev, and Serhiy Storchaka in bpo-14373(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14373
File: python.info, Node: glob<3>, Next: gzip<4>, Prev: functools<4>, Up: Improved Modules<9>
1.9.5.23 glob
.............
The *note iglob(): 803. and *note glob(): 2a1. functions now support
recursive search in subdirectories, using the ‘"**"’ pattern.
(Contributed by Serhiy Storchaka in bpo-13968(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13968
File: python.info, Node: gzip<4>, Next: heapq, Prev: glob<3>, Up: Improved Modules<9>
1.9.5.24 gzip
.............
The 'mode' argument of the *note GzipFile: 416. constructor now accepts
‘"x"’ to request exclusive creation. (Contributed by Tim Heaney in
bpo-19222(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19222
File: python.info, Node: heapq, Next: http<2>, Prev: gzip<4>, Up: Improved Modules<9>
1.9.5.25 heapq
..............
Element comparison in *note merge(): e03. can now be customized by
passing a *note key function: e04. in a new optional 'key' keyword
argument, and a new optional 'reverse' keyword argument can be used to
reverse element comparison:
>>> import heapq
>>> a = ['9', '777', '55555']
>>> b = ['88', '6666']
>>> list(heapq.merge(a, b, key=len))
['9', '88', '777', '6666', '55555']
>>> list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True))
['55555', '6666', '777', '88', '9']
(Contributed by Raymond Hettinger in bpo-13742(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13742
File: python.info, Node: http<2>, Next: http client<3>, Prev: heapq, Up: Improved Modules<9>
1.9.5.26 http
.............
A new *note HTTPStatus: 8ff. enum that defines a set of HTTP status
codes, reason phrases and long descriptions written in English.
(Contributed by Demian Brecht in bpo-21793(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21793
File: python.info, Node: http client<3>, Next: idlelib and IDLE<3>, Prev: http<2>, Up: Improved Modules<9>
1.9.5.27 http.client
....................
*note HTTPConnection.getresponse(): e07. now raises a *note
RemoteDisconnected: e08. exception when a remote server connection is
closed unexpectedly. Additionally, if a *note ConnectionError: e09. (of
which ‘RemoteDisconnected’ is a subclass) is raised, the client socket
is now closed automatically, and will reconnect on the next request:
import http.client
conn = http.client.HTTPConnection('www.python.org')
for retries in range(3):
try:
conn.request('GET', '/')
resp = conn.getresponse()
except http.client.RemoteDisconnected:
pass
(Contributed by Martin Panter in bpo-3566(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=3566
File: python.info, Node: idlelib and IDLE<3>, Next: imaplib<2>, Prev: http client<3>, Up: Improved Modules<9>
1.9.5.28 idlelib and IDLE
.........................
Since idlelib implements the IDLE shell and editor and is not intended
for import by other programs, it gets improvements with every release.
See ‘Lib/idlelib/NEWS.txt’ for a cumulative list of changes since 3.4.0,
as well as changes made in future 3.5.x releases. This file is also
available from the IDLE Help ‣ About IDLE dialog.
File: python.info, Node: imaplib<2>, Next: imghdr, Prev: idlelib and IDLE<3>, Up: Improved Modules<9>
1.9.5.29 imaplib
................
The *note IMAP4: 902. class now supports the *note context manager: 5d0.
protocol. When used in a *note with: 5ce. statement, the IMAP4 ‘LOGOUT’
command will be called automatically at the end of the block.
(Contributed by Tarek Ziadé and Serhiy Storchaka in bpo-4972(1).)
The *note imaplib: 74. module now supports RFC 5161(2) (ENABLE
Extension) and RFC 6855(3) (UTF-8 Support) via the *note IMAP4.enable():
e0c. method. A new *note IMAP4.utf8_enabled: e0d. attribute tracks
whether or not RFC 6855(4) support is enabled. (Contributed by Milan
Oberkirch, R. David Murray, and Maciej Szulik in bpo-21800(5).)
The *note imaplib: 74. module now automatically encodes non-ASCII string
usernames and passwords using UTF-8, as recommended by the RFCs.
(Contributed by Milan Oberkirch in bpo-21800(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4972
(2) https://datatracker.ietf.org/doc/html/rfc5161.html
(3) https://datatracker.ietf.org/doc/html/rfc6855.html
(4) https://datatracker.ietf.org/doc/html/rfc6855.html
(5) https://bugs.python.org/issue?@action=redirect&bpo=21800
(6) https://bugs.python.org/issue?@action=redirect&bpo=21800
File: python.info, Node: imghdr, Next: importlib<8>, Prev: imaplib<2>, Up: Improved Modules<9>
1.9.5.30 imghdr
...............
The ‘what()’ function now recognizes the OpenEXR(1) format (contributed
by Martin Vignali and Claudiu Popa in bpo-20295(2)), and the WebP(3)
format (contributed by Fabrice Aneche and Claudiu Popa in bpo-20197(4).)
---------- Footnotes ----------
(1) https://www.openexr.com
(2) https://bugs.python.org/issue?@action=redirect&bpo=20295
(3) https://en.wikipedia.org/wiki/WebP
(4) https://bugs.python.org/issue?@action=redirect&bpo=20197
File: python.info, Node: importlib<8>, Next: inspect<7>, Prev: imghdr, Up: Improved Modules<9>
1.9.5.31 importlib
..................
The *note util.LazyLoader: cad. class allows for lazy loading of modules
in applications where startup time is important. (Contributed by Brett
Cannon in bpo-17621(1).)
The *note abc.InspectLoader.source_to_code(): e10. method is now a
static method. This makes it easier to initialize a module object with
code compiled from a string by running ‘exec(code, module.__dict__)’.
(Contributed by Brett Cannon in bpo-21156(2).)
The new *note util.module_from_spec(): e11. function is now the
preferred way to create a new module. As opposed to creating a *note
types.ModuleType: e12. instance directly, this new function will set the
various import-controlled attributes based on the passed-in spec object.
(Contributed by Brett Cannon in bpo-20383(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17621
(2) https://bugs.python.org/issue?@action=redirect&bpo=21156
(3) https://bugs.python.org/issue?@action=redirect&bpo=20383
File: python.info, Node: inspect<7>, Next: io<5>, Prev: importlib<8>, Up: Improved Modules<9>
1.9.5.32 inspect
................
Both the *note Signature: 1cf. and *note Parameter: 1d0. classes are now
picklable and hashable. (Contributed by Yury Selivanov in bpo-20726(1)
and bpo-20334(2).)
A new *note BoundArguments.apply_defaults(): e14. method provides a way
to set default values for missing arguments:
>>> def foo(a, b='ham', *args): pass
>>> ba = inspect.signature(foo).bind('spam')
>>> ba.apply_defaults()
>>> ba.arguments
OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())])
(Contributed by Yury Selivanov in bpo-24190(3).)
A new class method *note Signature.from_callable(): 735. makes
subclassing of *note Signature: 1cf. easier. (Contributed by Yury
Selivanov and Eric Snow in bpo-17373(4).)
The *note signature(): 733. function now accepts a 'follow_wrapped'
optional keyword argument, which, when set to ‘False’, disables
automatic following of ‘__wrapped__’ links. (Contributed by Yury
Selivanov in bpo-20691(5).)
A set of new functions to inspect *note coroutine functions: d76. and
*note coroutine objects: 48d. has been added: *note iscoroutine(): e15,
*note iscoroutinefunction(): 2e8, *note isawaitable(): e16, *note
getcoroutinelocals(): e17, and *note getcoroutinestate(): e18.
(Contributed by Yury Selivanov in bpo-24017(6) and bpo-24400(7).)
The *note stack(): 64e, *note trace(): 64f, *note getouterframes(): 64c,
and *note getinnerframes(): 64d. functions now return a list of named
tuples. (Contributed by Daniel Shahaf in bpo-16808(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20726
(2) https://bugs.python.org/issue?@action=redirect&bpo=20334
(3) https://bugs.python.org/issue?@action=redirect&bpo=24190
(4) https://bugs.python.org/issue?@action=redirect&bpo=17373
(5) https://bugs.python.org/issue?@action=redirect&bpo=20691
(6) https://bugs.python.org/issue?@action=redirect&bpo=24017
(7) https://bugs.python.org/issue?@action=redirect&bpo=24400
(8) https://bugs.python.org/issue?@action=redirect&bpo=16808
File: python.info, Node: io<5>, Next: ipaddress<4>, Prev: inspect<7>, Up: Improved Modules<9>
1.9.5.33 io
...........
A new *note BufferedIOBase.readinto1(): e1a. method, that uses at most
one call to the underlying raw stream’s *note RawIOBase.read(): e1b. or
*note RawIOBase.readinto(): e1c. methods. (Contributed by Nikolaus Rath
in bpo-20578(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20578
File: python.info, Node: ipaddress<4>, Next: json<2>, Prev: io<5>, Up: Improved Modules<9>
1.9.5.34 ipaddress
..................
Both the *note IPv4Network: 200. and *note IPv6Network: 201. classes now
accept an ‘(address, netmask)’ tuple argument, so as to easily construct
network objects from existing addresses:
>>> import ipaddress
>>> ipaddress.IPv4Network(('127.0.0.0', 8))
IPv4Network('127.0.0.0/8')
>>> ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0'))
IPv4Network('127.0.0.0/8')
(Contributed by Peter Moody and Antoine Pitrou in bpo-16531(1).)
A new ‘reverse_pointer’ attribute for the *note IPv4Network: 200. and
*note IPv6Network: 201. classes returns the name of the reverse DNS PTR
record:
>>> import ipaddress
>>> addr = ipaddress.IPv4Address('127.0.0.1')
>>> addr.reverse_pointer
'1.0.0.127.in-addr.arpa'
>>> addr6 = ipaddress.IPv6Address('::1')
>>> addr6.reverse_pointer
'1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'
(Contributed by Leon Weber in bpo-20480(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16531
(2) https://bugs.python.org/issue?@action=redirect&bpo=20480
File: python.info, Node: json<2>, Next: linecache<2>, Prev: ipaddress<4>, Up: Improved Modules<9>
1.9.5.35 json
.............
The *note json.tool: 83. command line interface now preserves the order
of keys in JSON objects passed in input. The new ‘--sort-keys’ option
can be used to sort the keys alphabetically. (Contributed by Berker
Peksag in bpo-21650(1).)
JSON decoder now raises *note JSONDecodeError: e1f. instead of *note
ValueError: 204. to provide better context information about the error.
(Contributed by Serhiy Storchaka in bpo-19361(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21650
(2) https://bugs.python.org/issue?@action=redirect&bpo=19361
File: python.info, Node: linecache<2>, Next: locale<6>, Prev: json<2>, Up: Improved Modules<9>
1.9.5.36 linecache
..................
A new *note lazycache(): e21. function can be used to capture
information about a non-file-based module to permit getting its lines
later via *note getline(): e22. This avoids doing I/O until a line is
actually needed, without having to carry the module globals around
indefinitely. (Contributed by Robert Collins in bpo-17911(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17911
File: python.info, Node: locale<6>, Next: logging<5>, Prev: linecache<2>, Up: Improved Modules<9>
1.9.5.37 locale
...............
A new *note delocalize(): e24. function can be used to convert a string
into a normalized number string, taking the ‘LC_NUMERIC’ settings into
account:
>>> import locale
>>> locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8')
'de_DE.UTF-8'
>>> locale.delocalize('1.234,56')
'1234.56'
>>> locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8')
'en_US.UTF-8'
>>> locale.delocalize('1,234.56')
'1234.56'
(Contributed by Cédric Krier in bpo-13918(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13918
File: python.info, Node: logging<5>, Next: lzma, Prev: locale<6>, Up: Improved Modules<9>
1.9.5.38 logging
................
All logging methods (*note Logger: b4f. *note log(): e26, *note
exception(): e27, *note critical(): e28, *note debug(): e29, etc.), now
accept exception instances as an 'exc_info' argument, in addition to
boolean values and exception tuples:
>>> import logging
>>> try:
... 1/0
... except ZeroDivisionError as ex:
... logging.error('exception', exc_info=ex)
ERROR:root:exception
(Contributed by Yury Selivanov in bpo-20537(1).)
The *note handlers.HTTPHandler: e2a. class now accepts an optional *note
ssl.SSLContext: 296. instance to configure SSL settings used in an HTTP
connection. (Contributed by Alex Gaynor in bpo-22788(2).)
The *note handlers.QueueListener: e2b. class now takes a
'respect_handler_level' keyword argument which, if set to ‘True’, will
pass messages to handlers taking handler levels into account.
(Contributed by Vinay Sajip.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20537
(2) https://bugs.python.org/issue?@action=redirect&bpo=22788
File: python.info, Node: lzma, Next: math<8>, Prev: logging<5>, Up: Improved Modules<9>
1.9.5.39 lzma
.............
The *note LZMADecompressor.decompress(): e2d. method now accepts an
optional 'max_length' argument to limit the maximum size of decompressed
data. (Contributed by Martin Panter in bpo-15955(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15955
File: python.info, Node: math<8>, Next: multiprocessing<6>, Prev: lzma, Up: Improved Modules<9>
1.9.5.40 math
.............
Two new constants have been added to the *note math: 8e. module: *note
inf: c84. and *note nan: 65f. (Contributed by Mark Dickinson in
bpo-23185(1).)
A new function *note isclose(): daf. provides a way to test for
approximate equality. (Contributed by Chris Barker and Tal Einat in
bpo-24270(2).)
A new *note gcd(): 90f. function has been added. The ‘fractions.gcd()’
function is now deprecated. (Contributed by Mark Dickinson and Serhiy
Storchaka in bpo-22486(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23185
(2) https://bugs.python.org/issue?@action=redirect&bpo=24270
(3) https://bugs.python.org/issue?@action=redirect&bpo=22486
File: python.info, Node: multiprocessing<6>, Next: operator<2>, Prev: math<8>, Up: Improved Modules<9>
1.9.5.41 multiprocessing
........................
*note sharedctypes.synchronized(): e30. objects now support the *note
context manager: 5d0. protocol. (Contributed by Charles-François Natali
in bpo-21565(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21565
File: python.info, Node: operator<2>, Next: os<10>, Prev: multiprocessing<6>, Up: Improved Modules<9>
1.9.5.42 operator
.................
*note attrgetter(): e32, *note itemgetter(): a61, and *note
methodcaller(): e33. objects now support pickling. (Contributed by Josh
Rosenberg and Serhiy Storchaka in bpo-22955(1).)
New *note matmul(): e34. and *note imatmul(): e35. functions to perform
matrix multiplication. (Contributed by Benjamin Peterson in
bpo-21176(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22955
(2) https://bugs.python.org/issue?@action=redirect&bpo=21176
File: python.info, Node: os<10>, Next: pathlib<10>, Prev: operator<2>, Up: Improved Modules<9>
1.9.5.43 os
...........
The new *note scandir(): a5e. function returning an iterator of *note
DirEntry: 497. objects has been added. If possible, *note scandir():
a5e. extracts file attributes while scanning a directory, removing the
need to perform subsequent system calls to determine file type or
attributes, which may significantly improve performance. (Contributed
by Ben Hoyt with the help of Victor Stinner in bpo-22524(1).)
On Windows, a new *note stat_result.st_file_attributes: e37. attribute
is now available. It corresponds to the ‘dwFileAttributes’ member of
the ‘BY_HANDLE_FILE_INFORMATION’ structure returned by
‘GetFileInformationByHandle()’. (Contributed by Ben Hoyt in
bpo-21719(2).)
The *note urandom(): 51e. function now uses the ‘getrandom()’ syscall on
Linux 3.17 or newer, and ‘getentropy()’ on OpenBSD 5.6 and newer,
removing the need to use ‘/dev/urandom’ and avoiding failures due to
potential file descriptor exhaustion. (Contributed by Victor Stinner in
bpo-22181(3).)
New *note get_blocking(): e38. and *note set_blocking(): e39. functions
allow getting and setting a file descriptor’s blocking mode (*note
O_NONBLOCK: e3a.) (Contributed by Victor Stinner in bpo-22054(4).)
The *note truncate(): e3b. and *note ftruncate(): d8e. functions are now
supported on Windows. (Contributed by Steve Dower in bpo-23668(5).)
There is a new *note os.path.commonpath(): e3c. function returning the
longest common sub-path of each passed pathname. Unlike the *note
os.path.commonprefix(): e3d. function, it always returns a valid path:
>>> os.path.commonprefix(['/usr/lib', '/usr/local/lib'])
'/usr/l'
>>> os.path.commonpath(['/usr/lib', '/usr/local/lib'])
'/usr'
(Contributed by Rafik Draoui and Serhiy Storchaka in bpo-10395(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22524
(2) https://bugs.python.org/issue?@action=redirect&bpo=21719
(3) https://bugs.python.org/issue?@action=redirect&bpo=22181
(4) https://bugs.python.org/issue?@action=redirect&bpo=22054
(5) https://bugs.python.org/issue?@action=redirect&bpo=23668
(6) https://bugs.python.org/issue?@action=redirect&bpo=10395
File: python.info, Node: pathlib<10>, Next: pickle<3>, Prev: os<10>, Up: Improved Modules<9>
1.9.5.44 pathlib
................
The new *note Path.samefile(): e3f. method can be used to check whether
the path points to the same file as another path, which can be either
another *note Path: 22b. object, or a string:
>>> import pathlib
>>> p1 = pathlib.Path('/etc/hosts')
>>> p2 = pathlib.Path('/etc/../etc/hosts')
>>> p1.samefile(p2)
True
(Contributed by Vajrasky Kok and Antoine Pitrou in bpo-19775(1).)
The *note Path.mkdir(): e40. method now accepts a new optional
'exist_ok' argument to match ‘mkdir -p’ and *note os.makedirs(): 220.
functionality. (Contributed by Berker Peksag in bpo-21539(2).)
There is a new *note Path.expanduser(): e41. method to expand ‘~’ and
‘~user’ prefixes. (Contributed by Serhiy Storchaka and Claudiu Popa in
bpo-19776(3).)
A new *note Path.home(): e42. class method can be used to get a *note
Path: 22b. instance representing the user’s home directory.
(Contributed by Victor Salgado and Mayank Tripathi in bpo-19777(4).)
New *note Path.write_text(): e43, *note Path.read_text(): e44, *note
Path.write_bytes(): e45, *note Path.read_bytes(): e46. methods to
simplify read/write operations on files.
The following code snippet will create or rewrite existing file
‘~/spam42’:
>>> import pathlib
>>> p = pathlib.Path('~/spam42')
>>> p.expanduser().write_text('ham')
3
(Contributed by Christopher Welborn in bpo-20218(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19775
(2) https://bugs.python.org/issue?@action=redirect&bpo=21539
(3) https://bugs.python.org/issue?@action=redirect&bpo=19776
(4) https://bugs.python.org/issue?@action=redirect&bpo=19777
(5) https://bugs.python.org/issue?@action=redirect&bpo=20218
File: python.info, Node: pickle<3>, Next: poplib<2>, Prev: pathlib<10>, Up: Improved Modules<9>
1.9.5.45 pickle
...............
Nested objects, such as unbound methods or nested classes, can now be
pickled using *note pickle protocols: cc1. older than protocol version
4. Protocol version 4 already supports these cases. (Contributed by
Serhiy Storchaka in bpo-23611(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23611
File: python.info, Node: poplib<2>, Next: re<7>, Prev: pickle<3>, Up: Improved Modules<9>
1.9.5.46 poplib
...............
A new *note POP3.utf8(): e49. command enables RFC 6856(1)
(Internationalized Email) support, if a POP server supports it.
(Contributed by Milan OberKirch in bpo-21804(2).)
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc6856.html
(2) https://bugs.python.org/issue?@action=redirect&bpo=21804
File: python.info, Node: re<7>, Next: readline<2>, Prev: poplib<2>, Up: Improved Modules<9>
1.9.5.47 re
...........
References and conditional references to groups with fixed length are
now allowed in lookbehind assertions:
>>> import re
>>> pat = re.compile(r'(a|b).(?<=\1)c')
>>> pat.match('aac')
<_sre.SRE_Match object; span=(0, 3), match='aac'>
>>> pat.match('bbc')
<_sre.SRE_Match object; span=(0, 3), match='bbc'>
(Contributed by Serhiy Storchaka in bpo-9179(1).)
The number of capturing groups in regular expressions is no longer
limited to 100. (Contributed by Serhiy Storchaka in bpo-22437(2).)
The *note sub(): 2a5. and *note subn(): 2a6. functions now replace
unmatched groups with empty strings instead of raising an exception.
(Contributed by Serhiy Storchaka in bpo-1519638(3).)
The ‘re.error’ exceptions have new attributes, ‘msg’, ‘pattern’, ‘pos’,
‘lineno’, and ‘colno’, that provide better context information about the
error:
>>> re.compile("""
... (?x)
... .++
... """)
Traceback (most recent call last):
...
sre_constants.error: multiple repeat at position 16 (line 3, column 7)
(Contributed by Serhiy Storchaka in bpo-22578(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9179
(2) https://bugs.python.org/issue?@action=redirect&bpo=22437
(3) https://bugs.python.org/issue?@action=redirect&bpo=1519638
(4) https://bugs.python.org/issue?@action=redirect&bpo=22578
File: python.info, Node: readline<2>, Next: selectors, Prev: re<7>, Up: Improved Modules<9>
1.9.5.48 readline
.................
A new *note append_history_file(): e4c. function can be used to append
the specified number of trailing elements in history to the given file.
(Contributed by Bruno Cauet in bpo-22940(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22940
File: python.info, Node: selectors, Next: shutil<5>, Prev: readline<2>, Up: Improved Modules<9>
1.9.5.49 selectors
..................
The new *note DevpollSelector: bd7. supports efficient ‘/dev/poll’
polling on Solaris. (Contributed by Giampaolo Rodola’ in bpo-18931(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18931
File: python.info, Node: shutil<5>, Next: signal<3>, Prev: selectors, Up: Improved Modules<9>
1.9.5.50 shutil
...............
The *note move(): a5b. function now accepts a 'copy_function' argument,
allowing, for example, the *note copy(): a59. function to be used
instead of the default *note copy2(): a5a. if there is a need to ignore
file metadata when moving. (Contributed by Claudiu Popa in
bpo-19840(1).)
The *note make_archive(): 4af. function now supports the 'xztar' format.
(Contributed by Serhiy Storchaka in bpo-5411(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19840
(2) https://bugs.python.org/issue?@action=redirect&bpo=5411
File: python.info, Node: signal<3>, Next: smtpd<2>, Prev: shutil<5>, Up: Improved Modules<9>
1.9.5.51 signal
...............
On Windows, the *note set_wakeup_fd(): b70. function now also supports
socket handles. (Contributed by Victor Stinner in bpo-22018(1).)
Various ‘SIG*’ constants in the *note signal: c6. module have been
converted into *note Enums: 56. This allows meaningful names to be
printed during debugging, instead of integer “magic numbers”.
(Contributed by Giampaolo Rodola’ in bpo-21076(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22018
(2) https://bugs.python.org/issue?@action=redirect&bpo=21076
File: python.info, Node: smtpd<2>, Next: smtplib<2>, Prev: signal<3>, Up: Improved Modules<9>
1.9.5.52 smtpd
..............
Both the ‘SMTPServer’ and ‘SMTPChannel’ classes now accept a
'decode_data' keyword argument to determine if the ‘DATA’ portion of the
SMTP transaction is decoded using the ‘"utf-8"’ codec or is instead
provided to the ‘SMTPServer.process_message()’ method as a byte string.
The default is ‘True’ for backward compatibility reasons, but will
change to ‘False’ in Python 3.6. If 'decode_data' is set to ‘False’,
the ‘process_message’ method must be prepared to accept keyword
arguments. (Contributed by Maciej Szulik in bpo-19662(1).)
The ‘SMTPServer’ class now advertises the ‘8BITMIME’ extension ( RFC
6152(2)) if 'decode_data' has been set ‘True’. If the client specifies
‘BODY=8BITMIME’ on the ‘MAIL’ command, it is passed to
‘SMTPServer.process_message()’ via the 'mail_options' keyword.
(Contributed by Milan Oberkirch and R. David Murray in bpo-21795(3).)
The ‘SMTPServer’ class now also supports the ‘SMTPUTF8’ extension ( RFC
6531(4): Internationalized Email). If the client specified ‘SMTPUTF8
BODY=8BITMIME’ on the ‘MAIL’ command, they are passed to
‘SMTPServer.process_message()’ via the 'mail_options' keyword. It is
the responsibility of the ‘process_message’ method to correctly handle
the ‘SMTPUTF8’ data. (Contributed by Milan Oberkirch in bpo-21725(5).)
It is now possible to provide, directly or via name resolution, IPv6
addresses in the ‘SMTPServer’ constructor, and have it successfully
connect. (Contributed by Milan Oberkirch in bpo-14758(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19662
(2) https://datatracker.ietf.org/doc/html/rfc6152.html
(3) https://bugs.python.org/issue?@action=redirect&bpo=21795
(4) https://datatracker.ietf.org/doc/html/rfc6531.html
(5) https://bugs.python.org/issue?@action=redirect&bpo=21725
(6) https://bugs.python.org/issue?@action=redirect&bpo=14758
File: python.info, Node: smtplib<2>, Next: sndhdr, Prev: smtpd<2>, Up: Improved Modules<9>
1.9.5.53 smtplib
................
A new *note SMTP.auth(): e52. method provides a convenient way to
implement custom authentication mechanisms. (Contributed by Milan
Oberkirch in bpo-15014(1).)
The *note SMTP.set_debuglevel(): e53. method now accepts an additional
debuglevel (2), which enables timestamps in debug messages.
(Contributed by Gavin Chappell and Maciej Szulik in bpo-16914(2).)
Both the *note SMTP.sendmail(): e54. and *note SMTP.send_message(): e55.
methods now support RFC 6531(3) (SMTPUTF8). (Contributed by Milan
Oberkirch and R. David Murray in bpo-22027(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15014
(2) https://bugs.python.org/issue?@action=redirect&bpo=16914
(3) https://datatracker.ietf.org/doc/html/rfc6531.html
(4) https://bugs.python.org/issue?@action=redirect&bpo=22027
File: python.info, Node: sndhdr, Next: socket<8>, Prev: smtplib<2>, Up: Improved Modules<9>
1.9.5.54 sndhdr
...............
The ‘what()’ and ‘whathdr()’ functions now return a *note namedtuple():
1ca. (Contributed by Claudiu Popa in bpo-18615(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18615
File: python.info, Node: socket<8>, Next: ssl<9>, Prev: sndhdr, Up: Improved Modules<9>
1.9.5.55 socket
...............
Functions with timeouts now use a monotonic clock, instead of a system
clock. (Contributed by Victor Stinner in bpo-22043(1).)
A new *note socket.sendfile(): e58. method allows sending a file over a
socket by using the high-performance *note os.sendfile(): b0f. function
on UNIX, resulting in uploads being from 2 to 3 times faster than when
using plain *note socket.send(): da6. (Contributed by Giampaolo Rodola’
in bpo-17552(2).)
The *note socket.sendall(): da7. method no longer resets the socket
timeout every time bytes are received or sent. The socket timeout is
now the maximum total duration to send all data. (Contributed by Victor
Stinner in bpo-23853(3).)
The 'backlog' argument of the *note socket.listen(): e59. method is now
optional. By default it is set to *note SOMAXCONN: e5a. or to ‘128’,
whichever is less. (Contributed by Charles-François Natali in
bpo-21455(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22043
(2) https://bugs.python.org/issue?@action=redirect&bpo=17552
(3) https://bugs.python.org/issue?@action=redirect&bpo=23853
(4) https://bugs.python.org/issue?@action=redirect&bpo=21455
File: python.info, Node: ssl<9>, Next: sqlite3<8>, Prev: socket<8>, Up: Improved Modules<9>
1.9.5.56 ssl
............
* Menu:
* Memory BIO Support::
* Application-Layer Protocol Negotiation Support::
* Other Changes::
File: python.info, Node: Memory BIO Support, Next: Application-Layer Protocol Negotiation Support, Up: ssl<9>
1.9.5.57 Memory BIO Support
...........................
(Contributed by Geert Jansen in bpo-21965(1).)
The new *note SSLObject: b81. class has been added to provide SSL
protocol support for cases when the network I/O capabilities of *note
SSLSocket: 8e9. are not necessary or are suboptimal. ‘SSLObject’
represents an SSL protocol instance, but does not implement any network
I/O methods, and instead provides a memory buffer interface. The new
*note MemoryBIO: e5d. class can be used to pass data between Python and
an SSL protocol instance.
The memory BIO SSL support is primarily intended to be used in
frameworks implementing asynchronous I/O for which *note SSLSocket:
8e9.’s readiness model (“select/poll”) is inefficient.
A new *note SSLContext.wrap_bio(): b82. method can be used to create a
new ‘SSLObject’ instance.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21965
File: python.info, Node: Application-Layer Protocol Negotiation Support, Next: Other Changes, Prev: Memory BIO Support, Up: ssl<9>
1.9.5.58 Application-Layer Protocol Negotiation Support
.......................................................
(Contributed by Benjamin Peterson in bpo-20188(1).)
Where OpenSSL support is present, the *note ssl: d0. module now
implements the 'Application-Layer Protocol Negotiation' TLS extension as
described in RFC 7301(2).
The new *note SSLContext.set_alpn_protocols(): e5f. can be used to
specify which protocols a socket should advertise during the TLS
handshake.
The new *note SSLSocket.selected_alpn_protocol(): e60. returns the
protocol that was selected during the TLS handshake. The *note
HAS_ALPN: e61. flag indicates whether ALPN support is present.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20188
(2) https://datatracker.ietf.org/doc/html/rfc7301.html
File: python.info, Node: Other Changes, Prev: Application-Layer Protocol Negotiation Support, Up: ssl<9>
1.9.5.59 Other Changes
......................
There is a new *note SSLSocket.version(): e63. method to query the
actual protocol version in use. (Contributed by Antoine Pitrou in
bpo-20421(1).)
The *note SSLSocket: 8e9. class now implements a ‘SSLSocket.sendfile()’
method. (Contributed by Giampaolo Rodola’ in bpo-17552(2).)
The ‘SSLSocket.send()’ method now raises either the *note
ssl.SSLWantReadError: e64. or *note ssl.SSLWantWriteError: e65.
exception on a non-blocking socket if the operation would block.
Previously, it would return ‘0’. (Contributed by Nikolaus Rath in
bpo-20951(3).)
The *note cert_time_to_seconds(): e66. function now interprets the input
time as UTC and not as local time, per RFC 5280(4). Additionally, the
return value is always an *note int: 259. (Contributed by Akira Li in
bpo-19940(5).)
New ‘SSLObject.shared_ciphers()’ and *note SSLSocket.shared_ciphers():
e67. methods return the list of ciphers sent by the client during the
handshake. (Contributed by Benjamin Peterson in bpo-23186(6).)
The *note SSLSocket.do_handshake(): e68, *note SSLSocket.read(): e69,
‘SSLSocket.shutdown()’, and *note SSLSocket.write(): e6a. methods of the
*note SSLSocket: 8e9. class no longer reset the socket timeout every
time bytes are received or sent. The socket timeout is now the maximum
total duration of the method. (Contributed by Victor Stinner in
bpo-23853(7).)
The ‘match_hostname()’ function now supports matching of IP addresses.
(Contributed by Antoine Pitrou in bpo-23239(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20421
(2) https://bugs.python.org/issue?@action=redirect&bpo=17552
(3) https://bugs.python.org/issue?@action=redirect&bpo=20951
(4) https://datatracker.ietf.org/doc/html/rfc5280.html
(5) https://bugs.python.org/issue?@action=redirect&bpo=19940
(6) https://bugs.python.org/issue?@action=redirect&bpo=23186
(7) https://bugs.python.org/issue?@action=redirect&bpo=23853
(8) https://bugs.python.org/issue?@action=redirect&bpo=23239
File: python.info, Node: sqlite3<8>, Next: subprocess<4>, Prev: ssl<9>, Up: Improved Modules<9>
1.9.5.60 sqlite3
................
The *note Row: e6c. class now fully supports the sequence protocol, in
particular *note reversed(): 862. iteration and slice indexing.
(Contributed by Claudiu Popa in bpo-10203(1); by Lucas Sinclair, Jessica
McKellar, and Serhiy Storchaka in bpo-13583(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10203
(2) https://bugs.python.org/issue?@action=redirect&bpo=13583
File: python.info, Node: subprocess<4>, Next: sys<11>, Prev: sqlite3<8>, Up: Improved Modules<9>
1.9.5.61 subprocess
...................
The new *note run(): b86. function has been added. It runs the
specified command and returns a *note CompletedProcess: e6e. object,
which describes a finished process. The new API is more consistent and
is the recommended approach to invoking subprocesses in Python code that
does not need to maintain compatibility with earlier Python versions.
(Contributed by Thomas Kluyver in bpo-23342(1).)
Examples:
>>> subprocess.run(["ls", "-l"]) # doesn't capture output
CompletedProcess(args=['ls', '-l'], returncode=0)
>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
>>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n')
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23342
File: python.info, Node: sys<11>, Next: sysconfig<2>, Prev: subprocess<4>, Up: Improved Modules<9>
1.9.5.62 sys
............
A new ‘set_coroutine_wrapper()’ function allows setting a global hook
that will be called whenever a *note coroutine object: 48d. is created
by an *note async def: 5cd. function. A corresponding
‘get_coroutine_wrapper()’ can be used to obtain a currently set wrapper.
Both functions are *note provisional: b03, and are intended for
debugging purposes only. (Contributed by Yury Selivanov in
bpo-24017(1).)
A new *note is_finalizing(): a8f. function can be used to check if the
Python interpreter is *note shutting down: 14e. (Contributed by Antoine
Pitrou in bpo-22696(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24017
(2) https://bugs.python.org/issue?@action=redirect&bpo=22696
File: python.info, Node: sysconfig<2>, Next: tarfile<7>, Prev: sys<11>, Up: Improved Modules<9>
1.9.5.63 sysconfig
..................
The name of the user scripts directory on Windows now includes the first
two components of the Python version. (Contributed by Paul Moore in
bpo-23437(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23437
File: python.info, Node: tarfile<7>, Next: threading<6>, Prev: sysconfig<2>, Up: Improved Modules<9>
1.9.5.64 tarfile
................
The 'mode' argument of the *note open(): e72. function now accepts ‘"x"’
to request exclusive creation. (Contributed by Berker Peksag in
bpo-21717(1).)
The *note TarFile.extractall(): 42a. and *note TarFile.extract(): 42b.
methods now take a keyword argument 'numeric_owner'. If set to ‘True’,
the extracted files and directories will be owned by the numeric ‘uid’
and ‘gid’ from the tarfile. If set to ‘False’ (the default, and the
behavior in versions prior to 3.5), they will be owned by the named user
and group in the tarfile. (Contributed by Michael Vogt and Eric Smith
in bpo-23193(2).)
The *note TarFile.list(): e73. now accepts an optional 'members' keyword
argument that can be set to a subset of the list returned by *note
TarFile.getmembers(): e74. (Contributed by Serhiy Storchaka in
bpo-21549(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21717
(2) https://bugs.python.org/issue?@action=redirect&bpo=23193
(3) https://bugs.python.org/issue?@action=redirect&bpo=21549
File: python.info, Node: threading<6>, Next: time<7>, Prev: tarfile<7>, Up: Improved Modules<9>
1.9.5.65 threading
..................
Both the *note Lock.acquire(): 694. and *note RLock.acquire(): e76.
methods now use a monotonic clock for timeout management. (Contributed
by Victor Stinner in bpo-22043(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22043
File: python.info, Node: time<7>, Next: timeit<2>, Prev: threading<6>, Up: Improved Modules<9>
1.9.5.66 time
.............
The *note monotonic(): 255. function is now always available.
(Contributed by Victor Stinner in bpo-22043(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22043
File: python.info, Node: timeit<2>, Next: tkinter<8>, Prev: time<7>, Up: Improved Modules<9>
1.9.5.67 timeit
...............
A new command line option ‘-u’ or ‘--unit=U’ can be used to specify the
time unit for the timer output. Supported options are ‘usec’, ‘msec’,
or ‘sec’. (Contributed by Julian Gindi in bpo-18983(1).)
The *note timeit(): e79. function has a new 'globals' parameter for
specifying the namespace in which the code will be running.
(Contributed by Ben Roberts in bpo-2527(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18983
(2) https://bugs.python.org/issue?@action=redirect&bpo=2527
File: python.info, Node: tkinter<8>, Next: traceback<5>, Prev: timeit<2>, Up: Improved Modules<9>
1.9.5.68 tkinter
................
The ‘tkinter._fix’ module used for setting up the Tcl/Tk environment on
Windows has been replaced by a private function in the ‘_tkinter’ module
which makes no permanent changes to environment variables. (Contributed
by Zachary Ware in bpo-20035(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20035
File: python.info, Node: traceback<5>, Next: types<5>, Prev: tkinter<8>, Up: Improved Modules<9>
1.9.5.69 traceback
..................
New *note walk_stack(): e7c. and *note walk_tb(): e7d. functions to
conveniently traverse frame and *note traceback objects: af4.
(Contributed by Robert Collins in bpo-17911(1).)
New lightweight classes: *note TracebackException: 25e, *note
StackSummary: e7e, and *note FrameSummary: e7f. (Contributed by Robert
Collins in bpo-17911(2).)
Both the *note print_tb(): e80. and *note print_stack(): e81. functions
now support negative values for the 'limit' argument. (Contributed by
Dmitry Kazakov in bpo-22619(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17911
(2) https://bugs.python.org/issue?@action=redirect&bpo=17911
(3) https://bugs.python.org/issue?@action=redirect&bpo=22619
File: python.info, Node: types<5>, Next: unicodedata<8>, Prev: traceback<5>, Up: Improved Modules<9>
1.9.5.70 types
..............
A new *note coroutine(): e83. function to transform *note generator:
bde. and *note generator-like: dda. objects into *note awaitables: 1ad.
(Contributed by Yury Selivanov in bpo-24017(1).)
A new type called *note CoroutineType: e84, which is used for *note
coroutine: 48d. objects created by *note async def: 5cd. functions.
(Contributed by Yury Selivanov in bpo-24400(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=24017
(2) https://bugs.python.org/issue?@action=redirect&bpo=24400
File: python.info, Node: unicodedata<8>, Next: unittest<8>, Prev: types<5>, Up: Improved Modules<9>
1.9.5.71 unicodedata
....................
The *note unicodedata: 105. module now uses data from Unicode 8.0.0(1).
---------- Footnotes ----------
(1) https://unicode.org/versions/Unicode8.0.0/
File: python.info, Node: unittest<8>, Next: unittest mock<3>, Prev: unicodedata<8>, Up: Improved Modules<9>
1.9.5.72 unittest
.................
The *note TestLoader.loadTestsFromModule(): 291. method now accepts a
keyword-only argument 'pattern' which is passed to ‘load_tests’ as the
third argument. Found packages are now checked for ‘load_tests’
regardless of whether their path matches 'pattern', because it is
impossible for a package name to match the default pattern.
(Contributed by Robert Collins and Barry A. Warsaw in bpo-16662(1).)
Unittest discovery errors now are exposed in the *note
TestLoader.errors: e87. attribute of the *note TestLoader: 290.
instance. (Contributed by Robert Collins in bpo-19746(2).)
A new command line option ‘--locals’ to show local variables in
tracebacks. (Contributed by Robert Collins in bpo-22936(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16662
(2) https://bugs.python.org/issue?@action=redirect&bpo=19746
(3) https://bugs.python.org/issue?@action=redirect&bpo=22936
File: python.info, Node: unittest mock<3>, Next: urllib<2>, Prev: unittest<8>, Up: Improved Modules<9>
1.9.5.73 unittest.mock
......................
The *note Mock: a4b. class has the following improvements:
* The class constructor has a new 'unsafe' parameter, which causes
mock objects to raise *note AttributeError: 348. on attribute names
starting with ‘"assert"’. (Contributed by Kushal Das in
bpo-21238(1).)
* A new *note Mock.assert_not_called(): e89. method to check if the
mock object was called. (Contributed by Kushal Das in
bpo-21262(2).)
The *note MagicMock: e8a. class now supports ‘__truediv__()’,
‘__divmod__()’ and ‘__matmul__()’ operators. (Contributed by Johannes
Baiter in bpo-20968(3), and Håkan Lövdahl in bpo-23581(4) and
bpo-23568(5).)
It is no longer necessary to explicitly pass ‘create=True’ to the *note
patch(): e8b. function when patching builtin names. (Contributed by
Kushal Das in bpo-17660(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21238
(2) https://bugs.python.org/issue?@action=redirect&bpo=21262
(3) https://bugs.python.org/issue?@action=redirect&bpo=20968
(4) https://bugs.python.org/issue?@action=redirect&bpo=23581
(5) https://bugs.python.org/issue?@action=redirect&bpo=23568
(6) https://bugs.python.org/issue?@action=redirect&bpo=17660
File: python.info, Node: urllib<2>, Next: wsgiref, Prev: unittest mock<3>, Up: Improved Modules<9>
1.9.5.74 urllib
...............
A new *note request.HTTPPasswordMgrWithPriorAuth: e8d. class allows HTTP
Basic Authentication credentials to be managed so as to eliminate
unnecessary ‘401’ response handling, or to unconditionally send
credentials on the first request in order to communicate with servers
that return a ‘404’ response instead of a ‘401’ if the ‘Authorization’
header is not sent. (Contributed by Matej Cepl in bpo-19494(1) and
Akshit Khurana in bpo-7159(2).)
A new 'quote_via' argument for the *note parse.urlencode(): e8e.
function provides a way to control the encoding of query parts if
needed. (Contributed by Samwyse and Arnon Yaari in bpo-13866(3).)
The *note request.urlopen(): 295. function accepts an *note
ssl.SSLContext: 296. object as a 'context' argument, which will be used
for the HTTPS connection. (Contributed by Alex Gaynor in bpo-22366(4).)
The *note parse.urljoin(): e8f. was updated to use the RFC 3986(5)
semantics for the resolution of relative URLs, rather than RFC 1808(6)
and RFC 2396(7). (Contributed by Demian Brecht and Senthil Kumaran in
bpo-22118(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19494
(2) https://bugs.python.org/issue?@action=redirect&bpo=7159
(3) https://bugs.python.org/issue?@action=redirect&bpo=13866
(4) https://bugs.python.org/issue?@action=redirect&bpo=22366
(5) https://datatracker.ietf.org/doc/html/rfc3986.html
(6) https://datatracker.ietf.org/doc/html/rfc1808.html
(7) https://datatracker.ietf.org/doc/html/rfc2396.html
(8) https://bugs.python.org/issue?@action=redirect&bpo=22118
File: python.info, Node: wsgiref, Next: xmlrpc<2>, Prev: urllib<2>, Up: Improved Modules<9>
1.9.5.75 wsgiref
................
The 'headers' argument of the *note headers.Headers: e91. class
constructor is now optional. (Contributed by Pablo Torres Navarrete and
SilentGhost in bpo-5800(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5800
File: python.info, Node: xmlrpc<2>, Next: xml sax, Prev: wsgiref, Up: Improved Modules<9>
1.9.5.76 xmlrpc
...............
The *note client.ServerProxy: a56. class now supports the *note context
manager: 5d0. protocol. (Contributed by Claudiu Popa in bpo-20627(1).)
The *note client.ServerProxy: a56. constructor now accepts an optional
*note ssl.SSLContext: 296. instance. (Contributed by Alex Gaynor in
bpo-22960(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=20627
(2) https://bugs.python.org/issue?@action=redirect&bpo=22960
File: python.info, Node: xml sax, Next: zipfile<4>, Prev: xmlrpc<2>, Up: Improved Modules<9>
1.9.5.77 xml.sax
................
SAX parsers now support a character stream of the *note
xmlreader.InputSource: e94. object. (Contributed by Serhiy Storchaka in
bpo-2175(1).)
*note parseString(): e95. now accepts a *note str: 447. instance.
(Contributed by Serhiy Storchaka in bpo-10590(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2175
(2) https://bugs.python.org/issue?@action=redirect&bpo=10590
File: python.info, Node: zipfile<4>, Prev: xml sax, Up: Improved Modules<9>
1.9.5.78 zipfile
................
ZIP output can now be written to unseekable streams. (Contributed by
Serhiy Storchaka in bpo-23252(1).)
The 'mode' argument of *note ZipFile.open(): 417. method now accepts
‘"x"’ to request exclusive creation. (Contributed by Serhiy Storchaka
in bpo-21717(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23252
(2) https://bugs.python.org/issue?@action=redirect&bpo=21717
File: python.info, Node: Other module-level changes, Next: Optimizations<9>, Prev: Improved Modules<9>, Up: What’s New In Python 3 5
1.9.6 Other module-level changes
--------------------------------
Many functions in the *note mmap: 90, ‘ossaudiodev’, *note socket: cc,
*note ssl: d0, and *note codecs: 1b. modules now accept writable *note
bytes-like objects: d2d. (Contributed by Serhiy Storchaka in
bpo-23001(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23001
File: python.info, Node: Optimizations<9>, Next: Build and C API Changes<3>, Prev: Other module-level changes, Up: What’s New In Python 3 5
1.9.7 Optimizations
-------------------
The *note os.walk(): 4a5. function has been sped up by 3 to 5 times on
POSIX systems, and by 7 to 20 times on Windows. This was done using the
new *note os.scandir(): a5e. function, which exposes file information
from the underlying ‘readdir’ or ‘FindFirstFile’/‘FindNextFile’ system
calls. (Contributed by Ben Hoyt with help from Victor Stinner in
bpo-23605(1).)
Construction of ‘bytes(int)’ (filled by zero bytes) is faster and uses
less memory for large objects. ‘calloc()’ is used instead of ‘malloc()’
to allocate memory for these objects. (Contributed by Victor Stinner in
bpo-21233(2).)
Some operations on *note ipaddress: 80. *note IPv4Network: 200. and
*note IPv6Network: 201. have been massively sped up, such as *note
subnets(): e99, *note supernet(): e9a, *note summarize_address_range():
e9b, *note collapse_addresses(): e9c. The speed up can range from 3 to
15 times. (Contributed by Antoine Pitrou, Michel Albert, and Markus in
bpo-21486(3), bpo-21487(4), bpo-20826(5), bpo-23266(6).)
Pickling of *note ipaddress: 80. objects was optimized to produce
significantly smaller output. (Contributed by Serhiy Storchaka in
bpo-23133(7).)
Many operations on *note io.BytesIO: e9d. are now 50% to 100% faster.
(Contributed by Serhiy Storchaka in bpo-15381(8) and David Wilson in
bpo-22003(9).)
The *note marshal.dumps(): e9e. function is now faster: 65–85% with
versions 3 and 4, 20–25% with versions 0 to 2 on typical data, and up to
5 times in best cases. (Contributed by Serhiy Storchaka in
bpo-20416(10) and bpo-23344(11).)
The UTF-32 encoder is now 3 to 7 times faster. (Contributed by Serhiy
Storchaka in bpo-15027(12).)
Regular expressions are now parsed up to 10% faster. (Contributed by
Serhiy Storchaka in bpo-19380(13).)
The *note json.dumps(): d45. function was optimized to run with
‘ensure_ascii=False’ as fast as with ‘ensure_ascii=True’. (Contributed
by Naoki Inada in bpo-23206(14).)
The *note PyObject_IsInstance(): e9f. and *note PyObject_IsSubclass():
ea0. functions have been sped up in the common case that the second
argument has *note type: d48. as its metaclass. (Contributed Georg
Brandl by in bpo-22540(15).)
Method caching was slightly improved, yielding up to 5% performance
improvement in some benchmarks. (Contributed by Antoine Pitrou in
bpo-22847(16).)
Objects from the *note random: b8. module now use 50% less memory on
64-bit builds. (Contributed by Serhiy Storchaka in bpo-23488(17).)
The *note property(): 194. getter calls are up to 25% faster.
(Contributed by Joe Jevnik in bpo-23910(18).)
Instantiation of *note fractions.Fraction: 1e9. is now up to 30% faster.
(Contributed by Stefan Behnel in bpo-22464(19).)
String methods *note find(): ea1, *note rfind(): ea2, *note split():
ea3, *note partition(): ea4. and the *note in: 2ee. string operator are
now significantly faster for searching 1-character substrings.
(Contributed by Serhiy Storchaka in bpo-23573(20).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23605
(2) https://bugs.python.org/issue?@action=redirect&bpo=21233
(3) https://bugs.python.org/issue?@action=redirect&bpo=21486
(4) https://bugs.python.org/issue?@action=redirect&bpo=21487
(5) https://bugs.python.org/issue?@action=redirect&bpo=20826
(6) https://bugs.python.org/issue?@action=redirect&bpo=23266
(7) https://bugs.python.org/issue?@action=redirect&bpo=23133
(8) https://bugs.python.org/issue?@action=redirect&bpo=15381
(9) https://bugs.python.org/issue?@action=redirect&bpo=22003
(10) https://bugs.python.org/issue?@action=redirect&bpo=20416
(11) https://bugs.python.org/issue?@action=redirect&bpo=23344
(12) https://bugs.python.org/issue?@action=redirect&bpo=15027
(13) https://bugs.python.org/issue?@action=redirect&bpo=19380
(14) https://bugs.python.org/issue?@action=redirect&bpo=23206
(15) https://bugs.python.org/issue?@action=redirect&bpo=22540
(16) https://bugs.python.org/issue?@action=redirect&bpo=22847
(17) https://bugs.python.org/issue?@action=redirect&bpo=23488
(18) https://bugs.python.org/issue?@action=redirect&bpo=23910
(19) https://bugs.python.org/issue?@action=redirect&bpo=22464
(20) https://bugs.python.org/issue?@action=redirect&bpo=23573
File: python.info, Node: Build and C API Changes<3>, Next: Deprecated<10>, Prev: Optimizations<9>, Up: What’s New In Python 3 5
1.9.8 Build and C API Changes
-----------------------------
New ‘calloc’ functions were added:
* *note PyMem_RawCalloc(): 38c,
* *note PyMem_Calloc(): ea6,
* *note PyObject_Calloc(): ea7.
(Contributed by Victor Stinner in bpo-21233(1).)
New encoding/decoding helper functions:
* *note Py_DecodeLocale(): bc7. (replaced ‘_Py_char2wchar()’),
* *note Py_EncodeLocale(): bc8. (replaced ‘_Py_wchar2char()’).
(Contributed by Victor Stinner in bpo-18395(2).)
A new *note PyCodec_NameReplaceErrors(): ea8. function to replace the
unicode encode error with ‘\N{...}’ escapes. (Contributed by Serhiy
Storchaka in bpo-19676(3).)
A new *note PyErr_FormatV(): ea9. function similar to *note
PyErr_Format(): eaa, but accepts a ‘va_list’ argument. (Contributed by
Antoine Pitrou in bpo-18711(4).)
A new *note PyExc_RecursionError: eab. exception. (Contributed by Georg
Brandl in bpo-19235(5).)
New *note PyModule_FromDefAndSpec(): eac, *note
PyModule_FromDefAndSpec2(): ead, and *note PyModule_ExecDef(): eae.
functions introduced by PEP 489(6) – multi-phase extension module
initialization. (Contributed by Petr Viktorin in bpo-24268(7).)
New *note PyNumber_MatrixMultiply(): eaf. and *note
PyNumber_InPlaceMatrixMultiply(): eb0. functions to perform matrix
multiplication. (Contributed by Benjamin Peterson in bpo-21176(8). See
also PEP 465(9) for details.)
The *note PyTypeObject.tp_finalize: aa6. slot is now part of the stable
ABI.
Windows builds now require Microsoft Visual C++ 14.0, which is available
as part of Visual Studio 2015(10).
Extension modules now include a platform information tag in their
filename on some platforms (the tag is optional, and CPython will import
extensions without it, although if the tag is present and mismatched,
the extension won’t be loaded):
* On Linux, extension module filenames end with
‘.cpython-m--.pyd’:
* ‘’ is the major number of the Python version; for
Python 3.5 this is ‘3’.
* ‘’ is the minor number of the Python version; for
Python 3.5 this is ‘5’.
* ‘’ is the hardware architecture the extension
module was built to run on. It’s most commonly either ‘i386’
for 32-bit Intel platforms or ‘x86_64’ for 64-bit Intel (and
AMD) platforms.
* ‘’ is always ‘linux-gnu’, except for extensions built to
talk to the 32-bit ABI on 64-bit platforms, in which case it
is ‘linux-gnu32’ (and ‘’ will be ‘x86_64’).
* On Windows, extension module filenames end with
‘.cp-.pyd’:
* ‘’ is the major number of the Python version; for
Python 3.5 this is ‘3’.
* ‘’ is the minor number of the Python version; for
Python 3.5 this is ‘5’.
* ‘’ is the platform the extension module was built
for, either ‘win32’ for Win32, ‘win_amd64’ for Win64,
‘win_ia64’ for Windows Itanium 64, and ‘win_arm’ for Windows
on ARM.
* If built in debug mode, ‘’ will be ‘_d’, otherwise it
will be blank.
* On OS X platforms, extension module filenames now end with
‘-darwin.so’.
* On all other platforms, extension module filenames are the same as
they were with Python 3.4.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=21233
(2) https://bugs.python.org/issue?@action=redirect&bpo=18395
(3) https://bugs.python.org/issue?@action=redirect&bpo=19676
(4) https://bugs.python.org/issue?@action=redirect&bpo=18711
(5) https://bugs.python.org/issue?@action=redirect&bpo=19235
(6) https://peps.python.org/pep-0489/
(7) https://bugs.python.org/issue?@action=redirect&bpo=24268
(8) https://bugs.python.org/issue?@action=redirect&bpo=21176
(9) https://peps.python.org/pep-0465/
(10)
https://visualstudio.microsoft.com/en/vs/older-downloads/#visual-studio-2015-and-other-products
File: python.info, Node: Deprecated<10>, Next: Removed<10>, Prev: Build and C API Changes<3>, Up: What’s New In Python 3 5
1.9.9 Deprecated
----------------
* Menu:
* New Keywords: New Keywords<2>.
* Deprecated Python Behavior: Deprecated Python Behavior<2>.
* Unsupported Operating Systems::
* Deprecated Python modules, functions and methods: Deprecated Python modules functions and methods<3>.
File: python.info, Node: New Keywords<2>, Next: Deprecated Python Behavior<2>, Up: Deprecated<10>
1.9.9.1 New Keywords
....................
‘async’ and ‘await’ are not recommended to be used as variable, class,
function or module names. Introduced by PEP 492(1) in Python 3.5, they
will become proper keywords in Python 3.7.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0492/
File: python.info, Node: Deprecated Python Behavior<2>, Next: Unsupported Operating Systems, Prev: New Keywords<2>, Up: Deprecated<10>
1.9.9.2 Deprecated Python Behavior
..................................
Raising the *note StopIteration: bfa. exception inside a generator will
now generate a silent *note PendingDeprecationWarning: 8c7, which will
become a non-silent deprecation warning in Python 3.6 and will trigger a
*note RuntimeError: 195. in Python 3.7. See *note PEP 479; Change
StopIteration handling inside generators: d1f. for details.
File: python.info, Node: Unsupported Operating Systems, Next: Deprecated Python modules functions and methods<3>, Prev: Deprecated Python Behavior<2>, Up: Deprecated<10>
1.9.9.3 Unsupported Operating Systems
.....................................
Windows XP is no longer supported by Microsoft, thus, per PEP 11(1),
CPython 3.5 is no longer officially supported on this OS.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0011/
File: python.info, Node: Deprecated Python modules functions and methods<3>, Prev: Unsupported Operating Systems, Up: Deprecated<10>
1.9.9.4 Deprecated Python modules, functions and methods
........................................................
The ‘formatter’ module has now graduated to full deprecation and is
still slated for removal in Python 3.6.
The ‘asyncio.async()’ function is deprecated in favor of *note
ensure_future(): c76.
The ‘smtpd’ module has in the past always decoded the DATA portion of
email messages using the ‘utf-8’ codec. This can now be controlled by
the new 'decode_data' keyword to ‘SMTPServer’. The default value is
‘True’, but this default is deprecated. Specify the 'decode_data'
keyword with an appropriate value to avoid the deprecation warning.
Directly assigning values to the *note key: c01, *note value: c02. and
*note coded_value: c03. of *note http.cookies.Morsel: c04. objects is
deprecated. Use the *note set(): c05. method instead. In addition, the
undocumented 'LegalChars' parameter of *note set(): c05. is deprecated,
and is now ignored.
Passing a format string as keyword argument 'format_string' to the *note
format(): c00. method of the *note string.Formatter: eb6. class has been
deprecated. (Contributed by Serhiy Storchaka in bpo-23671(1).)
The ‘platform.dist()’ and ‘platform.linux_distribution()’ functions are
now deprecated. Linux distributions use too many different ways of
describing themselves, so the functionality is left to a package.
(Contributed by Vajrasky Kok and Berker Peksag in bpo-1322(2).)
The previously undocumented ‘from_function’ and ‘from_builtin’ methods
of *note inspect.Signature: 1cf. are deprecated. Use the new *note
Signature.from_callable(): 735. method instead. (Contributed by Yury
Selivanov in bpo-24248(3).)
The ‘inspect.getargspec()’ function is deprecated and scheduled to be
removed in Python 3.6. (See bpo-20438(4) for details.)
The *note inspect: 7e. *note getfullargspec(): 734, *note getcallargs():
eb7, and ‘formatargspec()’ functions are deprecated in favor of the
*note inspect.signature(): 733. API. (Contributed by Yury Selivanov in
bpo-20438(5).)
*note getargvalues(): eb8. and *note formatargvalues(): eb9. functions
were inadvertently marked as deprecated with the release of Python
3.5.0.
Use of *note re.LOCALE: b6c. flag with str patterns or *note re.ASCII:
628. is now deprecated. (Contributed by Serhiy Storchaka in
bpo-22407(6).)
Use of unrecognized special sequences consisting of ‘'\'’ and an ASCII
letter in regular expression patterns and replacement patterns now
raises a deprecation warning and will be forbidden in Python 3.6.
(Contributed by Serhiy Storchaka in bpo-23622(7).)
The undocumented and unofficial 'use_load_tests' default argument of the
*note unittest.TestLoader.loadTestsFromModule(): 291. method now is
deprecated and ignored. (Contributed by Robert Collins and Barry A.
Warsaw in bpo-16662(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23671
(2) https://bugs.python.org/issue?@action=redirect&bpo=1322
(3) https://bugs.python.org/issue?@action=redirect&bpo=24248
(4) https://bugs.python.org/issue?@action=redirect&bpo=20438
(5) https://bugs.python.org/issue?@action=redirect&bpo=20438
(6) https://bugs.python.org/issue?@action=redirect&bpo=22407
(7) https://bugs.python.org/issue?@action=redirect&bpo=23622
(8) https://bugs.python.org/issue?@action=redirect&bpo=16662
File: python.info, Node: Removed<10>, Next: Porting to Python 3 5, Prev: Deprecated<10>, Up: What’s New In Python 3 5
1.9.10 Removed
--------------
* Menu:
* API and Feature Removals: API and Feature Removals<4>.
File: python.info, Node: API and Feature Removals<4>, Up: Removed<10>
1.9.10.1 API and Feature Removals
.................................
The following obsolete and previously deprecated APIs and features have
been removed:
* The ‘__version__’ attribute has been dropped from the email
package. The email code hasn’t been shipped separately from the
stdlib for a long time, and the ‘__version__’ string was not
updated in the last few releases.
* The internal ‘Netrc’ class in the *note ftplib: 5e. module was
deprecated in 3.4, and has now been removed. (Contributed by Matt
Chaput in bpo-6623(1).)
* The concept of ‘.pyo’ files has been removed.
* The JoinableQueue class in the provisional *note asyncio: a. module
was deprecated in 3.4.4 and is now removed. (Contributed by A.
Jesse Jiryu Davis in bpo-23464(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6623
(2) https://bugs.python.org/issue?@action=redirect&bpo=23464
File: python.info, Node: Porting to Python 3 5, Next: Notable changes in Python 3 5 4, Prev: Removed<10>, Up: What’s New In Python 3 5
1.9.11 Porting to Python 3.5
----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in Python behavior: Changes in Python behavior<2>.
* Changes in the Python API: Changes in the Python API<9>.
* Changes in the C API: Changes in the C API<7>.
File: python.info, Node: Changes in Python behavior<2>, Next: Changes in the Python API<9>, Up: Porting to Python 3 5
1.9.11.1 Changes in Python behavior
...................................
* Due to an oversight, earlier Python versions erroneously accepted
the following syntax:
f(1 for x in [1], *args)
f(1 for x in [1], **kwargs)
Python 3.5 now correctly raises a *note SyntaxError: 18d, as
generator expressions must be put in parentheses if not a sole
argument to a function.
File: python.info, Node: Changes in the Python API<9>, Next: Changes in the C API<7>, Prev: Changes in Python behavior<2>, Up: Porting to Python 3 5
1.9.11.2 Changes in the Python API
..................................
* PEP 475(1): System calls are now retried when interrupted by a
signal instead of raising *note InterruptedError: d87. if the
Python signal handler does not raise an exception.
* Before Python 3.5, a *note datetime.time: 1ce. object was
considered to be false if it represented midnight in UTC. This
behavior was considered obscure and error-prone and has been
removed in Python 3.5. See bpo-13936(2) for full details.
* The ‘ssl.SSLSocket.send()’ method now raises either *note
ssl.SSLWantReadError: e64. or *note ssl.SSLWantWriteError: e65. on
a non-blocking socket if the operation would block. Previously, it
would return ‘0’. (Contributed by Nikolaus Rath in bpo-20951(3).)
* The ‘__name__’ attribute of generators is now set from the function
name, instead of being set from the code name. Use
‘gen.gi_code.co_name’ to retrieve the code name. Generators also
have a new ‘__qualname__’ attribute, the qualified name, which is
now used for the representation of a generator (‘repr(gen)’).
(Contributed by Victor Stinner in bpo-21205(4).)
* The deprecated “strict” mode and argument of *note HTMLParser: 874,
‘HTMLParser.error()’, and the ‘HTMLParserError’ exception have been
removed. (Contributed by Ezio Melotti in bpo-15114(5).) The
'convert_charrefs' argument of *note HTMLParser: 874. is now ‘True’
by default. (Contributed by Berker Peksag in bpo-21047(6).)
* Although it is not formally part of the API, it is worth noting for
porting purposes (ie: fixing tests) that error messages that were
previously of the form “‘sometype’ does not support the buffer
protocol” are now of the form “a *note bytes-like object: d2d. is
required, not ‘sometype’”. (Contributed by Ezio Melotti in
bpo-16518(7).)
* If the current directory is set to a directory that no longer
exists then *note FileNotFoundError: 427. will no longer be raised
and instead *note find_spec(): ebf. will return ‘None’ 'without'
caching ‘None’ in *note sys.path_importer_cache: 5e0, which is
different than the typical case (bpo-22834(8)).
* HTTP status code and messages from *note http.client: 6f. and *note
http.server: 72. were refactored into a common *note HTTPStatus:
8ff. enum. The values in *note http.client: 6f. and *note
http.server: 72. remain available for backwards compatibility.
(Contributed by Demian Brecht in bpo-21793(9).)
* When an import loader defines
‘importlib.machinery.Loader.exec_module()’ it is now expected to
also define ‘create_module()’ (raises a *note DeprecationWarning:
1a5. now, will be an error in Python 3.6). If the loader inherits
from *note importlib.abc.Loader: ec0. then there is nothing to do,
else simply define ‘create_module()’ to return ‘None’.
(Contributed by Brett Cannon in bpo-23014(10).)
* The *note re.split(): 2a4. function always ignored empty pattern
matches, so the ‘"x*"’ pattern worked the same as ‘"x+"’, and the
‘"\b"’ pattern never worked. Now *note re.split(): 2a4. raises a
warning if the pattern could match an empty string. For
compatibility, use patterns that never match an empty string (e.g.
‘"x+"’ instead of ‘"x*"’). Patterns that could only match an empty
string (such as ‘"\b"’) now raise an error. (Contributed by Serhiy
Storchaka in bpo-22818(11).)
* The *note http.cookies.Morsel: c04. dict-like interface has been
made self consistent: morsel comparison now takes the *note key:
c01. and *note value: c02. into account, *note copy(): ec1. now
results in a *note Morsel: c04. instance rather than a *note dict:
258, and *note update(): ec2. will now raise an exception if any of
the keys in the update dictionary are invalid. In addition, the
undocumented 'LegalChars' parameter of *note set(): c05. is
deprecated and is now ignored. (Contributed by Demian Brecht in
bpo-2211(12).)
* PEP 488(13) has removed ‘.pyo’ files from Python and introduced the
optional ‘opt-’ tag in ‘.pyc’ file names. The *note
importlib.util.cache_from_source(): 2f8. has gained an
'optimization' parameter to help control the ‘opt-’ tag. Because
of this, the 'debug_override' parameter of the function is now
deprecated. ‘.pyo’ files are also no longer supported as a file
argument to the Python interpreter and thus serve no purpose when
distributed on their own (i.e. sourceless code distribution). Due
to the fact that the magic number for bytecode has changed in
Python 3.5, all old ‘.pyo’ files from previous versions of Python
are invalid regardless of this PEP.
* The *note socket: cc. module now exports the *note
CAN_RAW_FD_FRAMES: ec3. constant on linux 3.6 and greater.
* The *note ssl.cert_time_to_seconds(): e66. function now interprets
the input time as UTC and not as local time, per RFC 5280(14).
Additionally, the return value is always an *note int: 259.
(Contributed by Akira Li in bpo-19940(15).)
* The ‘pygettext.py’ Tool now uses the standard +NNNN format for
timezones in the POT-Creation-Date header.
* The *note smtplib: ca. module now uses *note sys.stderr: 939.
instead of the previous module-level ‘stderr’ variable for debug
output. If your (test) program depends on patching the
module-level variable to capture the debug output, you will need to
update it to capture sys.stderr instead.
* The *note str.startswith(): ec4. and *note str.endswith(): ec5.
methods no longer return ‘True’ when finding the empty string and
the indexes are completely out of range. (Contributed by Serhiy
Storchaka in bpo-24284(16).)
* The *note inspect.getdoc(): 9f9. function now returns documentation
strings inherited from base classes. Documentation strings no
longer need to be duplicated if the inherited documentation is
appropriate. To suppress an inherited string, an empty string must
be specified (or the documentation may be filled in). This change
affects the output of the *note pydoc: b5. module and the *note
help(): 8d6. function. (Contributed by Serhiy Storchaka in
bpo-15582(17).)
* Nested *note functools.partial(): 410. calls are now flattened. If
you were relying on the previous behavior, you can now either add
an attribute to a *note functools.partial(): 410. object or you can
create a subclass of *note functools.partial(): 410. (Contributed
by Alexander Belopolsky in bpo-7830(18).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0475/
(2) https://bugs.python.org/issue?@action=redirect&bpo=13936
(3) https://bugs.python.org/issue?@action=redirect&bpo=20951
(4) https://bugs.python.org/issue?@action=redirect&bpo=21205
(5) https://bugs.python.org/issue?@action=redirect&bpo=15114
(6) https://bugs.python.org/issue?@action=redirect&bpo=21047
(7) https://bugs.python.org/issue?@action=redirect&bpo=16518
(8) https://bugs.python.org/issue?@action=redirect&bpo=22834
(9) https://bugs.python.org/issue?@action=redirect&bpo=21793
(10) https://bugs.python.org/issue?@action=redirect&bpo=23014
(11) https://bugs.python.org/issue?@action=redirect&bpo=22818
(12) https://bugs.python.org/issue?@action=redirect&bpo=2211
(13) https://peps.python.org/pep-0488/
(14) https://datatracker.ietf.org/doc/html/rfc5280.html
(15) https://bugs.python.org/issue?@action=redirect&bpo=19940
(16) https://bugs.python.org/issue?@action=redirect&bpo=24284
(17) https://bugs.python.org/issue?@action=redirect&bpo=15582
(18) https://bugs.python.org/issue?@action=redirect&bpo=7830
File: python.info, Node: Changes in the C API<7>, Prev: Changes in the Python API<9>, Up: Porting to Python 3 5
1.9.11.3 Changes in the C API
.............................
* The undocumented ‘format’ member of the (non-public)
‘PyMemoryViewObject’ structure has been removed. All extensions
relying on the relevant parts in ‘memoryobject.h’ must be rebuilt.
* The ‘PyMemAllocator’ structure was renamed to *note
PyMemAllocatorEx: ec7. and a new ‘calloc’ field was added.
* Removed non-documented macro ‘PyObject_REPR()’ which leaked
references. Use format character ‘%R’ in *note
PyUnicode_FromFormat(): 385.-like functions to format the *note
repr(): 7f9. of the object. (Contributed by Serhiy Storchaka in
bpo-22453(1).)
* Because the lack of the *note __module__: 36f. attribute breaks
pickling and introspection, a deprecation warning is now raised for
builtin types without the *note __module__: 36f. attribute. This
will be an *note AttributeError: 348. in the future. (Contributed
by Serhiy Storchaka in bpo-20204(2).)
* As part of the PEP 492(3) implementation, the ‘tp_reserved’ slot of
*note PyTypeObject: aa5. was replaced with a *note tp_as_async:
ec8. slot. Refer to *note Coroutine Objects: ec9. for new types,
structures and functions.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=22453
(2) https://bugs.python.org/issue?@action=redirect&bpo=20204
(3) https://peps.python.org/pep-0492/
File: python.info, Node: Notable changes in Python 3 5 4, Prev: Porting to Python 3 5, Up: What’s New In Python 3 5
1.9.12 Notable changes in Python 3.5.4
--------------------------------------
* Menu:
* New make regen-all build target: New make regen-all build target<2>.
* Removal of make touch build target: Removal of make touch build target<2>.
File: python.info, Node: New make regen-all build target<2>, Next: Removal of make touch build target<2>, Up: Notable changes in Python 3 5 4
1.9.12.1 New ‘make regen-all’ build target
..........................................
To simplify cross-compilation, and to ensure that CPython can reliably
be compiled without requiring an existing version of Python to already
be available, the autotools-based build system no longer attempts to
implicitly recompile generated files based on file modification times.
Instead, a new ‘make regen-all’ command has been added to force
regeneration of these files when desired (e.g. after an initial version
of Python has already been built based on the pregenerated versions).
More selective regeneration targets are also defined - see
Makefile.pre.in(1) for details.
(Contributed by Victor Stinner in bpo-23404(2).)
Added in version 3.5.4.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Makefile.pre.in
(2) https://bugs.python.org/issue?@action=redirect&bpo=23404
File: python.info, Node: Removal of make touch build target<2>, Prev: New make regen-all build target<2>, Up: Notable changes in Python 3 5 4
1.9.12.2 Removal of ‘make touch’ build target
.............................................
The ‘make touch’ build target previously used to request implicit
regeneration of generated files by updating their modification times has
been removed.
It has been replaced by the new ‘make regen-all’ target.
(Contributed by Victor Stinner in bpo-23404(1).)
Changed in version 3.5.4.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23404
File: python.info, Node: What’s New In Python 3 4, Next: What’s New In Python 3 3, Prev: What’s New In Python 3 5, Up: What’s New in Python
1.10 What’s New In Python 3.4
=============================
Author: R. David Murray <> (Editor)
This article explains the new features in Python 3.4, compared to 3.3.
Python 3.4 was released on March 16, 2014. For full details, see the
changelog(1).
See also
........
PEP 429(2) – Python 3.4 Release Schedule
* Menu:
* Summary – Release Highlights: Summary – Release Highlights<3>.
* New Features: New Features<15>.
* New Modules: New Modules<10>.
* Improved Modules: Improved Modules<10>.
* CPython Implementation Changes::
* Deprecated: Deprecated<11>.
* Removed: Removed<11>.
* Porting to Python 3.4: Porting to Python 3 4.
* Changed in 3.4.3: Changed in 3 4 3.
---------- Footnotes ----------
(1) https://docs.python.org/3.4/whatsnew/changelog.html
(2) https://peps.python.org/pep-0429/
File: python.info, Node: Summary – Release Highlights<3>, Next: New Features<15>, Up: What’s New In Python 3 4
1.10.1 Summary – Release Highlights
-----------------------------------
New syntax features:
* No new syntax features were added in Python 3.4.
Other new features:
* *note pip should always be available: ed0. ( PEP 453(1)).
* *note Newly created file descriptors are non-inheritable: ed1. (
PEP 446(2)).
* command line option for *note isolated mode: ed2. (bpo-16499(3)).
* *note improvements in the handling of codecs: ed3. that are not
text encodings (multiple issues).
* *note A ModuleSpec Type: ed4. for the Import System ( PEP 451(4)).
(Affects importer authors.)
* The *note marshal: 8d. format has been made *note more compact and
efficient: ed5. (bpo-16475(5)).
New library modules:
* *note asyncio: a.: *note New provisional API for asynchronous IO:
ed6. ( PEP 3156(6)).
* *note ensurepip: 55.: *note Bootstrapping the pip installer: ed7. (
PEP 453(7)).
* *note enum: 56.: *note Support for enumeration types: ed8. ( PEP
435(8)).
* *note pathlib: a4.: *note Object-oriented filesystem paths: ed9. (
PEP 428(9)).
* *note selectors: c2.: *note High-level and efficient I/O
multiplexing: eda, built upon the *note select: c1. module
primitives (part of PEP 3156(10)).
* *note statistics: d2.: A basic *note numerically stable statistics
library: edb. ( PEP 450(11)).
* *note tracemalloc: ff.: *note Trace Python memory allocations: edc.
( PEP 454(12)).
Significantly improved library modules:
* *note Single-dispatch generic functions: edd. in *note functools:
5f. ( PEP 443(13)).
* New *note pickle: a6. *note protocol 4: ede. ( PEP 3154(14)).
* *note multiprocessing: 94. now has *note an option to avoid using
os.fork on Unix: edf. (bpo-8713(15)).
* *note email: 3b. has a new submodule, *note contentmanager: 3d, and
a new *note Message: 27e. subclass (‘EmailMessage’) that *note
simplify MIME handling: ee0. (bpo-18891(16)).
* The *note inspect: 7e. and *note pydoc: b5. modules are now capable
of correct introspection of a much wider variety of callable
objects, which improves the output of the Python *note help(): 8d6.
system.
* The *note ipaddress: 80. module API has been declared stable
Security improvements:
* *note Secure and interchangeable hash algorithm: ee1. ( PEP
456(17)).
* *note Make newly created file descriptors non-inheritable: ed1. (
PEP 446(18)) to avoid leaking file descriptors to child processes.
* New command line option for *note isolated mode: ed2,
(bpo-16499(19)).
* *note multiprocessing: 94. now has *note an option to avoid using
os.fork on Unix: edf. 'spawn' and 'forkserver' are more secure
because they avoid sharing data with child processes.
* *note multiprocessing: 94. child processes on Windows no longer
inherit all of the parent’s inheritable handles, only the necessary
ones.
* A new *note hashlib.pbkdf2_hmac(): 50a. function provides the
PKCS#5 password-based key derivation function 2(20).
* *note TLSv1.1 and TLSv1.2 support: ee2. for *note ssl: d0.
* *note Retrieving certificates from the Windows system cert store
support: ee3. for *note ssl: d0.
* *note Server-side SNI (Server Name Indication) support: ee4. for
*note ssl: d0.
* The *note ssl.SSLContext: 296. class has a *note lot of
improvements: ee5.
* All modules in the standard library that support SSL now support
server certificate verification, including hostname matching
(‘ssl.match_hostname()’) and CRLs (Certificate Revocation lists,
see *note ssl.SSLContext.load_verify_locations(): ee6.).
CPython implementation improvements:
* *note Safe object finalization: ee7. ( PEP 442(21)).
* Leveraging PEP 442(22), in most cases *note module globals are no
longer set to None during finalization: ee7. (bpo-18214(23)).
* *note Configurable memory allocators: ee8. ( PEP 445(24)).
* *note Argument Clinic: ee9. ( PEP 436(25)).
Please read on for a comprehensive list of user-facing changes,
including many other smaller improvements, CPython optimizations,
deprecations, and potential porting issues.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0453/
(2) https://peps.python.org/pep-0446/
(3) https://bugs.python.org/issue?@action=redirect&bpo=16499
(4) https://peps.python.org/pep-0451/
(5) https://bugs.python.org/issue?@action=redirect&bpo=16475
(6) https://peps.python.org/pep-3156/
(7) https://peps.python.org/pep-0453/
(8) https://peps.python.org/pep-0435/
(9) https://peps.python.org/pep-0428/
(10) https://peps.python.org/pep-3156/
(11) https://peps.python.org/pep-0450/
(12) https://peps.python.org/pep-0454/
(13) https://peps.python.org/pep-0443/
(14) https://peps.python.org/pep-3154/
(15) https://bugs.python.org/issue?@action=redirect&bpo=8713
(16) https://bugs.python.org/issue?@action=redirect&bpo=18891
(17) https://peps.python.org/pep-0456/
(18) https://peps.python.org/pep-0446/
(19) https://bugs.python.org/issue?@action=redirect&bpo=16499
(20) https://en.wikipedia.org/wiki/PBKDF2
(21) https://peps.python.org/pep-0442/
(22) https://peps.python.org/pep-0442/
(23) https://bugs.python.org/issue?@action=redirect&bpo=18214
(24) https://peps.python.org/pep-0445/
(25) https://peps.python.org/pep-0436/
File: python.info, Node: New Features<15>, Next: New Modules<10>, Prev: Summary – Release Highlights<3>, Up: What’s New In Python 3 4
1.10.2 New Features
-------------------
* Menu:
* PEP 453; Explicit Bootstrapping of PIP in Python Installations: PEP 453 Explicit Bootstrapping of PIP in Python Installations.
* PEP 446; Newly Created File Descriptors Are Non-Inheritable: PEP 446 Newly Created File Descriptors Are Non-Inheritable.
* Improvements to Codec Handling::
* PEP 451; A ModuleSpec Type for the Import System: PEP 451 A ModuleSpec Type for the Import System.
* Other Language Changes: Other Language Changes<10>.
File: python.info, Node: PEP 453 Explicit Bootstrapping of PIP in Python Installations, Next: PEP 446 Newly Created File Descriptors Are Non-Inheritable, Up: New Features<15>
1.10.2.1 PEP 453: Explicit Bootstrapping of PIP in Python Installations
.......................................................................
* Menu:
* Bootstrapping pip By Default::
* Documentation Changes::
File: python.info, Node: Bootstrapping pip By Default, Next: Documentation Changes, Up: PEP 453 Explicit Bootstrapping of PIP in Python Installations
1.10.2.2 Bootstrapping pip By Default
.....................................
The new *note ensurepip: 55. module (defined in PEP 453(1)) provides a
standard cross-platform mechanism to bootstrap the pip installer into
Python installations and virtual environments. The version of ‘pip’
included with Python 3.4.0 is ‘pip’ 1.5.4, and future 3.4.x maintenance
releases will update the bundled version to the latest version of ‘pip’
that is available at the time of creating the release candidate.
By default, the commands ‘pipX’ and ‘pipX.Y’ will be installed on all
platforms (where X.Y stands for the version of the Python installation),
along with the ‘pip’ Python package and its dependencies. On Windows
and in virtual environments on all platforms, the unversioned ‘pip’
command will also be installed. On other platforms, the system wide
unversioned ‘pip’ command typically refers to the separately installed
Python 2 version.
The ‘pyvenv’ command line utility and the *note venv: 111. module make
use of the *note ensurepip: 55. module to make ‘pip’ readily available
in virtual environments. When using the command line utility, ‘pip’ is
installed by default, while when using the *note venv: 111. module *note
API: eed. installation of ‘pip’ must be requested explicitly.
For CPython *note source builds on POSIX systems: eee, the ‘make
install’ and ‘make altinstall’ commands bootstrap ‘pip’ by default.
This behaviour can be controlled through configure options, and
overridden through Makefile options.
On Windows and Mac OS X, the CPython installers now default to
installing ‘pip’ along with CPython itself (users may opt out of
installing it during the installation process). Window users will need
to opt in to the automatic ‘PATH’ modifications to have ‘pip’ available
from the command line by default, otherwise it can still be accessed
through the Python launcher for Windows as ‘py -m pip’.
As discussed in the PEP(2) platform packagers may choose not to install
these commands by default, as long as, when invoked, they provide clear
and simple directions on how to install them on that platform (usually
using the system package manager).
Note: To avoid conflicts between parallel Python 2 and Python 3
installations, only the versioned ‘pip3’ and ‘pip3.4’ commands are
bootstrapped by default when ‘ensurepip’ is invoked directly - the
‘--default-pip’ option is needed to also request the unversioned
‘pip’ command. ‘pyvenv’ and the Windows installer ensure that the
unqualified ‘pip’ command is made available in those environments,
and ‘pip’ can always be invoked via the ‘-m’ switch rather than
directly to avoid ambiguity on systems with multiple Python
installations.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0453/
(2)
https://peps.python.org/pep-0453/#recommendations-for-downstream-distributors
File: python.info, Node: Documentation Changes, Prev: Bootstrapping pip By Default, Up: PEP 453 Explicit Bootstrapping of PIP in Python Installations
1.10.2.3 Documentation Changes
..............................
As part of this change, the *note Installing Python Modules: ef0. and
distributing-index sections of the documentation have been completely
redesigned as short getting started and FAQ documents. Most packaging
documentation has now been moved out to the Python Packaging Authority
maintained Python Packaging User Guide(1) and the documentation of the
individual projects.
However, as this migration is currently still incomplete, the legacy
versions of those guides remaining available as *note Building C and C++
Extensions with setuptools: ef1. and *note Building C and C++ Extensions
with setuptools: ef2.
See also
........
PEP 453(2) – Explicit bootstrapping of pip in Python installations
PEP written by Donald Stufft and Nick Coghlan, implemented by
Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily.
---------- Footnotes ----------
(1) https://packaging.python.org
(2) https://peps.python.org/pep-0453/
File: python.info, Node: PEP 446 Newly Created File Descriptors Are Non-Inheritable, Next: Improvements to Codec Handling, Prev: PEP 453 Explicit Bootstrapping of PIP in Python Installations, Up: New Features<15>
1.10.2.4 PEP 446: Newly Created File Descriptors Are Non-Inheritable
....................................................................
PEP 446(1) makes newly created file descriptors *note non-inheritable:
ef4. In general, this is the behavior an application will want: when
launching a new process, having currently open files also open in the
new process can lead to all sorts of hard to find bugs, and potentially
to security issues.
However, there are occasions when inheritance is desired. To support
these cases, the following new functions and methods are available:
* *note os.get_inheritable(): ef5, *note os.set_inheritable(): ef6.
* *note os.get_handle_inheritable(): ef7, *note
os.set_handle_inheritable(): ef8.
* *note socket.socket.get_inheritable(): ef9, *note
socket.socket.set_inheritable(): efa.
See also
........
PEP 446(2) – Make newly created file descriptors non-inheritable
PEP written and implemented by Victor Stinner.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0446/
(2) https://peps.python.org/pep-0446/
File: python.info, Node: Improvements to Codec Handling, Next: PEP 451 A ModuleSpec Type for the Import System, Prev: PEP 446 Newly Created File Descriptors Are Non-Inheritable, Up: New Features<15>
1.10.2.5 Improvements to Codec Handling
.......................................
Since it was first introduced, the *note codecs: 1b. module has always
been intended to operate as a type-neutral dynamic encoding and decoding
system. However, its close coupling with the Python text model,
especially the type restricted convenience methods on the builtin *note
str: 447, *note bytes: 1c2. and *note bytearray: 53a. types, has
historically obscured that fact.
As a key step in clarifying the situation, the *note codecs.encode():
efc. and *note codecs.decode(): efd. convenience functions are now
properly documented in Python 2.7, 3.3 and 3.4. These functions have
existed in the *note codecs: 1b. module (and have been covered by the
regression test suite) since Python 2.4, but were previously only
discoverable through runtime introspection.
Unlike the convenience methods on *note str: 447, *note bytes: 1c2. and
*note bytearray: 53a, the *note codecs: 1b. convenience functions
support arbitrary codecs in both Python 2 and Python 3, rather than
being limited to Unicode text encodings (in Python 3) or ‘basestring’
<-> ‘basestring’ conversions (in Python 2).
In Python 3.4, the interpreter is able to identify the known non-text
encodings provided in the standard library and direct users towards
these general purpose convenience functions when appropriate:
>>> b"abcdef".decode("hex")
Traceback (most recent call last):
File "", line 1, in
LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs
>>> "hello".encode("rot13")
Traceback (most recent call last):
File "", line 1, in
LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs
>>> open("foo.txt", encoding="hex")
Traceback (most recent call last):
File "", line 1, in
LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs
In a related change, whenever it is feasible without breaking backwards
compatibility, exceptions raised during encoding and decoding operations
are wrapped in a chained exception of the same type that mentions the
name of the codec responsible for producing the error:
>>> import codecs
>>> codecs.decode(b"abcdefgh", "hex")
Traceback (most recent call last):
File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode
return (binascii.a2b_hex(input), len(input))
binascii.Error: Non-hexadecimal digit found
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found)
>>> codecs.encode("hello", "bz2")
Traceback (most recent call last):
File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode
return (bz2.compress(input), len(input))
File "/usr/lib/python3.4/bz2.py", line 498, in compress
return comp.compress(data) + comp.flush()
TypeError: 'str' does not support the buffer interface
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface)
Finally, as the examples above show, these improvements have permitted
the restoration of the convenience aliases for the non-Unicode codecs
that were themselves restored in Python 3.2. This means that encoding
binary data to and from its hexadecimal representation (for example) can
now be written as:
>>> from codecs import encode, decode
>>> encode(b"hello", "hex")
b'68656c6c6f'
>>> decode(b"68656c6c6f", "hex")
b'hello'
The binary and text transforms provided in the standard library are
detailed in *note Binary Transforms: efe. and *note Text Transforms:
eff.
(Contributed by Nick Coghlan in bpo-7475(1), bpo-17827(2), bpo-17828(3)
and bpo-19619(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=7475
(2) https://bugs.python.org/issue?@action=redirect&bpo=17827
(3) https://bugs.python.org/issue?@action=redirect&bpo=17828
(4) https://bugs.python.org/issue?@action=redirect&bpo=19619
File: python.info, Node: PEP 451 A ModuleSpec Type for the Import System, Next: Other Language Changes<10>, Prev: Improvements to Codec Handling, Up: New Features<15>
1.10.2.6 PEP 451: A ModuleSpec Type for the Import System
.........................................................
PEP 451(1) provides an encapsulation of the information about a module
that the import machinery will use to load it (that is, a module
specification). This helps simplify both the import implementation and
several import-related APIs. The change is also a stepping stone for
several future import-related improvements(2).
The public-facing changes from the PEP are entirely backward-compatible.
Furthermore, they should be transparent to everyone but importer
authors. Key finder and loader methods have been deprecated, but they
will continue working. New importers should use the new methods
described in the PEP. Existing importers should be updated to implement
the new methods. See the *note Deprecated: f01. section for a list of
methods that should be replaced and their replacements.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0451/
(2)
https://mail.python.org/pipermail/python-dev/2013-November/130111.html
File: python.info, Node: Other Language Changes<10>, Prev: PEP 451 A ModuleSpec Type for the Import System, Up: New Features<15>
1.10.2.7 Other Language Changes
...............................
Some smaller changes made to the core Python language are:
* Unicode database updated to UCD version 6.3.
* *note min(): f03. and *note max(): f04. now accept a 'default'
keyword-only argument that can be used to specify the value they
return if the iterable they are evaluating has no elements.
(Contributed by Julian Berman in bpo-18111(1).)
* Module objects are now *note weakly referenceable: f05.
* Module ‘__file__’ attributes (and related values) should now always
contain absolute paths by default, with the sole exception of
‘__main__.__file__’ when a script has been executed directly using
a relative path. (Contributed by Brett Cannon in bpo-18416(2).)
* All the UTF-* codecs (except UTF-7) now reject surrogates during
both encoding and decoding unless the ‘surrogatepass’ error handler
is used, with the exception of the UTF-16 decoder (which accepts
valid surrogate pairs) and the UTF-16 encoder (which produces them
while encoding non-BMP characters). (Contributed by Victor
Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in bpo-12892(3).)
* New German EBCDIC *note codec: db8. ‘cp273’. (Contributed by
Michael Bierenfeld and Andrew Kuchling in bpo-1097797(4).)
* New Ukrainian *note codec: db8. ‘cp1125’. (Contributed by Serhiy
Storchaka in bpo-19668(5).)
* *note bytes: 1c2.join() and *note bytearray: 53a.join() now accept
arbitrary buffer objects as arguments. (Contributed by Antoine
Pitrou in bpo-15958(6).)
* The *note int: 259. constructor now accepts any object that has an
‘__index__’ method for its 'base' argument. (Contributed by Mark
Dickinson in bpo-16772(7).)
* Frame objects now have a *note clear(): 282. method that clears all
references to local variables from the frame. (Contributed by
Antoine Pitrou in bpo-17934(8).)
* *note memoryview: 464. is now registered as a *note Sequence: 1e,
and supports the *note reversed(): 862. builtin. (Contributed by
Nick Coghlan and Claudiu Popa in bpo-18690(9) and bpo-19078(10).)
* Signatures reported by *note help(): 8d6. have been modified and
improved in several cases as a result of the introduction of
Argument Clinic and other changes to the *note inspect: 7e. and
*note pydoc: b5. modules.
* *note __length_hint__(): f06. is now part of the formal language
specification (see PEP 424(11)). (Contributed by Armin Ronacher in
bpo-16148(12).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18111
(2) https://bugs.python.org/issue?@action=redirect&bpo=18416
(3) https://bugs.python.org/issue?@action=redirect&bpo=12892
(4) https://bugs.python.org/issue?@action=redirect&bpo=1097797
(5) https://bugs.python.org/issue?@action=redirect&bpo=19668
(6) https://bugs.python.org/issue?@action=redirect&bpo=15958
(7) https://bugs.python.org/issue?@action=redirect&bpo=16772
(8) https://bugs.python.org/issue?@action=redirect&bpo=17934
(9) https://bugs.python.org/issue?@action=redirect&bpo=18690
(10) https://bugs.python.org/issue?@action=redirect&bpo=19078
(11) https://peps.python.org/pep-0424/
(12) https://bugs.python.org/issue?@action=redirect&bpo=16148
File: python.info, Node: New Modules<10>, Next: Improved Modules<10>, Prev: New Features<15>, Up: What’s New In Python 3 4
1.10.3 New Modules
------------------
* Menu:
* asyncio: asyncio<11>.
* ensurepip: ensurepip<2>.
* enum: enum<9>.
* pathlib: pathlib<11>.
* selectors: selectors<2>.
* statistics: statistics<6>.
* tracemalloc: tracemalloc<4>.
File: python.info, Node: asyncio<11>, Next: ensurepip<2>, Up: New Modules<10>
1.10.3.1 asyncio
................
The new *note asyncio: a. module (defined in PEP 3156(1)) provides a
standard pluggable event loop model for Python, providing solid
asynchronous IO support in the standard library, and making it easier
for other event loop implementations to interoperate with the standard
library and each other.
For Python 3.4, this module is considered a *note provisional API: b03.
See also
........
PEP 3156(2) – Asynchronous IO Support Rebooted: the “asyncio” Module
PEP written and implementation led by Guido van Rossum.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3156/
(2) https://peps.python.org/pep-3156/
File: python.info, Node: ensurepip<2>, Next: enum<9>, Prev: asyncio<11>, Up: New Modules<10>
1.10.3.2 ensurepip
..................
The new *note ensurepip: 55. module is the primary infrastructure for
the PEP 453(1) implementation. In the normal course of events end users
will not need to interact with this module, but it can be used to
manually bootstrap ‘pip’ if the automated bootstrapping into an
installation or virtual environment was declined.
*note ensurepip: 55. includes a bundled copy of ‘pip’, up-to-date as of
the first release candidate of the release of CPython with which it
ships (this applies to both maintenance releases and feature releases).
‘ensurepip’ does not access the internet. If the installation has
internet access, after ‘ensurepip’ is run the bundled ‘pip’ can be used
to upgrade ‘pip’ to a more recent release than the bundled one. (Note
that such an upgraded version of ‘pip’ is considered to be a separately
installed package and will not be removed if Python is uninstalled.)
The module is named 'ensure'pip because if called when ‘pip’ is already
installed, it does nothing. It also has an ‘--upgrade’ option that will
cause it to install the bundled copy of ‘pip’ if the existing installed
version of ‘pip’ is older than the bundled copy.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0453/
File: python.info, Node: enum<9>, Next: pathlib<11>, Prev: ensurepip<2>, Up: New Modules<10>
1.10.3.3 enum
.............
The new *note enum: 56. module (defined in PEP 435(1)) provides a
standard implementation of enumeration types, allowing other modules
(such as *note socket: cc.) to provide more informative error messages
and better debugging support by replacing opaque integer constants with
backwards compatible enumeration values.
See also
........
PEP 435(2) – Adding an Enum type to the Python standard library
PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman,
implemented by Ethan Furman.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0435/
(2) https://peps.python.org/pep-0435/
File: python.info, Node: pathlib<11>, Next: selectors<2>, Prev: enum<9>, Up: New Modules<10>
1.10.3.4 pathlib
................
The new *note pathlib: a4. module offers classes representing filesystem
paths with semantics appropriate for different operating systems. Path
classes are divided between 'pure paths', which provide purely
computational operations without I/O, and 'concrete paths', which
inherit from pure paths but also provide I/O operations.
For Python 3.4, this module is considered a *note provisional API: b03.
See also
........
PEP 428(1) – The pathlib module – object-oriented filesystem paths
PEP written and implemented by Antoine Pitrou.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0428/
File: python.info, Node: selectors<2>, Next: statistics<6>, Prev: pathlib<11>, Up: New Modules<10>
1.10.3.5 selectors
..................
The new *note selectors: c2. module (created as part of implementing PEP
3156(1)) allows high-level and efficient I/O multiplexing, built upon
the *note select: c1. module primitives.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3156/
File: python.info, Node: statistics<6>, Next: tracemalloc<4>, Prev: selectors<2>, Up: New Modules<10>
1.10.3.6 statistics
...................
The new *note statistics: d2. module (defined in PEP 450(1)) offers some
core statistics functionality directly in the standard library. This
module supports calculation of the mean, median, mode, variance and
standard deviation of a data series.
See also
........
PEP 450(2) – Adding A Statistics Module To The Standard Library
PEP written and implemented by Steven D’Aprano
---------- Footnotes ----------
(1) https://peps.python.org/pep-0450/
(2) https://peps.python.org/pep-0450/
File: python.info, Node: tracemalloc<4>, Prev: statistics<6>, Up: New Modules<10>
1.10.3.7 tracemalloc
....................
The new *note tracemalloc: ff. module (defined in PEP 454(1)) is a debug
tool to trace memory blocks allocated by Python. It provides the
following information:
* Trace where an object was allocated
* Statistics on allocated memory blocks per filename and per line
number: total size, number and average size of allocated memory
blocks
* Compute the differences between two snapshots to detect memory
leaks
See also
........
PEP 454(2) – Add a new tracemalloc module to trace Python memory allocations
PEP written and implemented by Victor Stinner
---------- Footnotes ----------
(1) https://peps.python.org/pep-0454/
(2) https://peps.python.org/pep-0454/
File: python.info, Node: Improved Modules<10>, Next: CPython Implementation Changes, Prev: New Modules<10>, Up: What’s New In Python 3 4
1.10.4 Improved Modules
-----------------------
* Menu:
* abc::
* aifc: aifc<2>.
* argparse: argparse<5>.
* audioop::
* base64: base64<3>.
* collections: collections<6>.
* colorsys::
* contextlib: contextlib<6>.
* dbm: dbm<6>.
* dis: dis<4>.
* doctest: doctest<4>.
* email: email<4>.
* filecmp::
* functools: functools<5>.
* gc: gc<5>.
* glob: glob<4>.
* hashlib: hashlib<6>.
* hmac: hmac<3>.
* html::
* http: http<3>.
* idlelib and IDLE: idlelib and IDLE<4>.
* importlib: importlib<9>.
* inspect: inspect<8>.
* ipaddress: ipaddress<5>.
* logging: logging<6>.
* marshal: marshal<2>.
* mmap: mmap<3>.
* multiprocessing: multiprocessing<7>.
* operator: operator<3>.
* os: os<11>.
* pdb: pdb<6>.
* pickle: pickle<4>.
* plistlib: plistlib<2>.
* poplib: poplib<3>.
* pprint: pprint<4>.
* pty::
* pydoc: pydoc<4>.
* re: re<8>.
* resource::
* select::
* shelve: shelve<2>.
* shutil: shutil<6>.
* smtpd: smtpd<3>.
* smtplib: smtplib<3>.
* socket: socket<9>.
* sqlite3: sqlite3<9>.
* ssl: ssl<10>.
* stat::
* struct: struct<2>.
* subprocess: subprocess<5>.
* sunau: sunau<2>.
* sys: sys<12>.
* tarfile: tarfile<8>.
* textwrap::
* threading: threading<7>.
* traceback: traceback<6>.
* types: types<6>.
* urllib: urllib<3>.
* unittest: unittest<9>.
* venv: venv<7>.
* wave: wave<2>.
* weakref: weakref<2>.
* xml.etree: xml etree<2>.
* zipfile: zipfile<5>.
File: python.info, Node: abc, Next: aifc<2>, Up: Improved Modules<10>
1.10.4.1 abc
............
New function *note abc.get_cache_token(): f11. can be used to know when
to invalidate caches that are affected by changes in the object graph.
(Contributed by Łukasz Langa in bpo-16832(1).)
New class *note ABC: f12. has *note ABCMeta: f13. as its meta class.
Using ‘ABC’ as a base class has essentially the same effect as
specifying ‘metaclass=abc.ABCMeta’, but is simpler to type and easier to
read. (Contributed by Bruno Dupuis in bpo-16049(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16832
(2) https://bugs.python.org/issue?@action=redirect&bpo=16049
File: python.info, Node: aifc<2>, Next: argparse<5>, Prev: abc, Up: Improved Modules<10>
1.10.4.2 aifc
.............
The ‘getparams()’ method now returns a namedtuple rather than a plain
tuple. (Contributed by Claudiu Popa in bpo-17818(1).)
‘aifc.open()’ now supports the context management protocol: when used in
a *note with: 5ce. block, the ‘close()’ method of the returned object
will be called automatically at the end of the block. (Contributed by
Serhiy Storchacha in bpo-16486(2).)
The ‘writeframesraw()’ and ‘writeframes()’ methods now accept any *note
bytes-like object: d2d. (Contributed by Serhiy Storchaka in
bpo-8311(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17818
(2) https://bugs.python.org/issue?@action=redirect&bpo=16486
(3) https://bugs.python.org/issue?@action=redirect&bpo=8311
File: python.info, Node: argparse<5>, Next: audioop, Prev: aifc<2>, Up: Improved Modules<10>
1.10.4.3 argparse
.................
The *note FileType: f16. class now accepts 'encoding' and 'errors'
arguments, which are passed through to *note open(): 517. (Contributed
by Lucas Maystre in bpo-11175(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11175
File: python.info, Node: audioop, Next: base64<3>, Prev: argparse<5>, Up: Improved Modules<10>
1.10.4.4 audioop
................
‘audioop’ now supports 24-bit samples. (Contributed by Serhiy Storchaka
in bpo-12866(1).)
New ‘byteswap()’ function converts big-endian samples to little-endian
and vice versa. (Contributed by Serhiy Storchaka in bpo-19641(2).)
All ‘audioop’ functions now accept any *note bytes-like object: d2d.
Strings are not accepted: they didn’t work before, now they raise an
error right away. (Contributed by Serhiy Storchaka in bpo-16685(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12866
(2) https://bugs.python.org/issue?@action=redirect&bpo=19641
(3) https://bugs.python.org/issue?@action=redirect&bpo=16685
File: python.info, Node: base64<3>, Next: collections<6>, Prev: audioop, Up: Improved Modules<10>
1.10.4.5 base64
...............
The encoding and decoding functions in *note base64: e. now accept any
*note bytes-like object: d2d. in cases where it previously required a
*note bytes: 1c2. or *note bytearray: 53a. instance. (Contributed by
Nick Coghlan in bpo-17839(1).)
New functions *note a85encode(): f19, *note a85decode(): f1a, *note
b85encode(): f1b, and *note b85decode(): f1c. provide the ability to
encode and decode binary data from and to ‘Ascii85’ and the
git/mercurial ‘Base85’ formats, respectively. The ‘a85’ functions have
options that can be used to make them compatible with the variants of
the ‘Ascii85’ encoding, including the Adobe variant. (Contributed by
Martin Morrison, the Mercurial project, Serhiy Storchaka, and Antoine
Pitrou in bpo-17618(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17839
(2) https://bugs.python.org/issue?@action=redirect&bpo=17618
File: python.info, Node: collections<6>, Next: colorsys, Prev: base64<3>, Up: Improved Modules<10>
1.10.4.6 collections
....................
The *note ChainMap.new_child(): f1e. method now accepts an 'm' argument
specifying the child map to add to the chain. This allows an existing
mapping and/or a custom mapping type to be used for the child.
(Contributed by Vinay Sajip in bpo-16613(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16613
File: python.info, Node: colorsys, Next: contextlib<6>, Prev: collections<6>, Up: Improved Modules<10>
1.10.4.7 colorsys
.................
The number of digits in the coefficients for the RGB — YIQ conversions
have been expanded so that they match the FCC NTSC versions. The change
in results should be less than 1% and may better match results found
elsewhere. (Contributed by Brian Landers and Serhiy Storchaka in
bpo-14323(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14323
File: python.info, Node: contextlib<6>, Next: dbm<6>, Prev: colorsys, Up: Improved Modules<10>
1.10.4.8 contextlib
...................
The new *note contextlib.suppress: f21. context manager helps to clarify
the intent of code that deliberately suppresses exceptions from a single
statement. (Contributed by Raymond Hettinger in bpo-15806(1) and Zero
Piraeus in bpo-19266(2).)
The new *note contextlib.redirect_stdout(): de7. context manager makes
it easier for utility scripts to handle inflexible APIs that write their
output to *note sys.stdout: ad6. and don’t provide any options to
redirect it. Using the context manager, the *note sys.stdout: ad6.
output can be redirected to any other stream or, in conjunction with
*note io.StringIO: f22, to a string. The latter can be especially
useful, for example, to capture output from a function that was written
to implement a command line interface. It is recommended only for
utility scripts because it affects the global state of *note sys.stdout:
ad6. (Contributed by Raymond Hettinger in bpo-15805(3).)
The *note contextlib: 23. documentation has also been updated to include
a *note discussion: f23. of the differences between single use, reusable
and reentrant context managers.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15806
(2) https://bugs.python.org/issue?@action=redirect&bpo=19266
(3) https://bugs.python.org/issue?@action=redirect&bpo=15805
File: python.info, Node: dbm<6>, Next: dis<4>, Prev: contextlib<6>, Up: Improved Modules<10>
1.10.4.9 dbm
............
*note dbm.open(): f25. objects now support the context management
protocol. When used in a *note with: 5ce. statement, the ‘close’ method
of the database object will be called automatically at the end of the
block. (Contributed by Claudiu Popa and Nick Coghlan in bpo-19282(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19282
File: python.info, Node: dis<4>, Next: doctest<4>, Prev: dbm<6>, Up: Improved Modules<10>
1.10.4.10 dis
.............
Functions *note show_code(): f27, *note dis(): b34, ‘distb()’, and *note
disassemble(): f28. now accept a keyword-only 'file' argument that
controls where they write their output.
The *note dis: 38. module is now built around an *note Instruction: 1dc.
class that provides object oriented access to the details of each
individual bytecode operation.
A new method, *note get_instructions(): 1db, provides an iterator that
emits the Instruction stream for a given piece of Python code. Thus it
is now possible to write a program that inspects and manipulates a
bytecode object in ways different from those provided by the *note dis:
38. module itself. For example:
>>> import dis
>>> for instr in dis.get_instructions(lambda x: x + 1):
... print(instr.opname)
LOAD_FAST
LOAD_CONST
BINARY_ADD
RETURN_VALUE
The various display tools in the *note dis: 38. module have been
rewritten to use these new components.
In addition, a new application-friendly class *note Bytecode: f29.
provides an object-oriented API for inspecting bytecode in both in
human-readable form and for iterating over instructions. The *note
Bytecode: f29. constructor takes the same arguments that
‘get_instruction()’ does (plus an optional 'current_offset'), and the
resulting object can be iterated to produce *note Instruction: 1dc.
objects. But it also has a *note dis: f2a. method, equivalent to
calling *note dis: b34. on the constructor argument, but returned as a
multi-line string:
>>> bytecode = dis.Bytecode(lambda x: x + 1, current_offset=3)
>>> for instr in bytecode:
... print('{} ({})'.format(instr.opname, instr.opcode))
LOAD_FAST (124)
LOAD_CONST (100)
BINARY_ADD (23)
RETURN_VALUE (83)
>>> bytecode.dis().splitlines()
[' 1 0 LOAD_FAST 0 (x)',
' --> 3 LOAD_CONST 1 (1)',
' 6 BINARY_ADD',
' 7 RETURN_VALUE']
*note Bytecode: f29. also has a class method, *note from_traceback():
f2b, that provides the ability to manipulate a traceback (that is,
‘print(Bytecode.from_traceback(tb).dis())’ is equivalent to
‘distb(tb)’).
(Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in
bpo-11816(1) and Claudiu Popa in bpo-17916(2).)
New function *note stack_effect(): f2c. computes the effect on the
Python stack of a given opcode and argument, information that is not
otherwise available. (Contributed by Larry Hastings in bpo-19722(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11816
(2) https://bugs.python.org/issue?@action=redirect&bpo=17916
(3) https://bugs.python.org/issue?@action=redirect&bpo=19722
File: python.info, Node: doctest<4>, Next: email<4>, Prev: dis<4>, Up: Improved Modules<10>
1.10.4.11 doctest
.................
A new *note option flag: f2e, *note FAIL_FAST: f2f, halts test running
as soon as the first failure is detected. (Contributed by R. David
Murray and Daniel Urban in bpo-16522(1).)
The *note doctest: 3a. command line interface now uses *note argparse:
6, and has two new options, ‘-o’ and ‘-f’. ‘-o’ allows *note doctest
options: f2e. to be specified on the command line, and ‘-f’ is a
shorthand for ‘-o FAIL_FAST’ (to parallel the similar option supported
by the *note unittest: 106. CLI). (Contributed by R. David Murray in
bpo-11390(2).)
*note doctest: 3a. will now find doctests in extension module ‘__doc__’
strings. (Contributed by Zachary Ware in bpo-3158(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16522
(2) https://bugs.python.org/issue?@action=redirect&bpo=11390
(3) https://bugs.python.org/issue?@action=redirect&bpo=3158
File: python.info, Node: email<4>, Next: filecmp, Prev: doctest<4>, Up: Improved Modules<10>
1.10.4.12 email
...............
*note as_string(): f31. now accepts a 'policy' argument to override the
default policy of the message when generating a string representation of
it. This means that ‘as_string’ can now be used in more circumstances,
instead of having to create and use a *note generator: 40. in order to
pass formatting parameters to its ‘flatten’ method. (Contributed by R.
David Murray in bpo-18600(1).)
New method *note as_bytes(): f32. added to produce a bytes
representation of the message in a fashion similar to how ‘as_string’
produces a string representation. It does not accept the 'maxheaderlen'
argument, but does accept the 'unixfrom' and 'policy' arguments. The
*note Message: 27e. *note __bytes__(): f33. method calls it, meaning
that ‘bytes(mymsg)’ will now produce the intuitive result: a bytes
object containing the fully formatted message. (Contributed by R. David
Murray in bpo-18600(2).)
The *note Message.set_param(): f34. message now accepts a 'replace'
keyword argument. When specified, the associated header will be updated
without changing its location in the list of headers. For backward
compatibility, the default is ‘False’. (Contributed by R. David Murray
in bpo-18891(3).) A pair of new subclasses of *note Message: 27e. have
been added (*note EmailMessage: 27d. and *note MIMEPart: f35.), along
with a new sub-module, *note contentmanager: 3d. and a new *note policy:
4f. attribute *note content_manager: f36. All documentation is
currently in the new module, which is being added as part of email’s new
*note provisional API: b03. These classes provide a number of new
methods that make extracting content from and inserting content into
email messages much easier. For details, see the *note contentmanager:
3d. documentation and the *note email; Examples: f37. These API
additions complete the bulk of the work that was planned as part of the
email6 project. The currently provisional API is scheduled to become
final in Python 3.5 (possibly with a few minor additions in the area of
error handling). (Contributed by R. David Murray in bpo-18891(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18600
(2) https://bugs.python.org/issue?@action=redirect&bpo=18600
(3) https://bugs.python.org/issue?@action=redirect&bpo=18891
(4) https://bugs.python.org/issue?@action=redirect&bpo=18891
File: python.info, Node: filecmp, Next: functools<5>, Prev: email<4>, Up: Improved Modules<10>
1.10.4.13 filecmp
.................
A new *note clear_cache(): f39. function provides the ability to clear
the *note filecmp: 5a. comparison cache, which uses *note os.stat():
49c. information to determine if the file has changed since the last
compare. This can be used, for example, if the file might have been
changed and re-checked in less time than the resolution of a particular
filesystem’s file modification time field. (Contributed by Mark Levitt
in bpo-18149(1).)
New module attribute *note DEFAULT_IGNORES: f3a. provides the list of
directories that are used as the default value for the 'ignore'
parameter of the *note dircmp(): f3b. function. (Contributed by Eli
Bendersky in bpo-15442(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18149
(2) https://bugs.python.org/issue?@action=redirect&bpo=15442
File: python.info, Node: functools<5>, Next: gc<5>, Prev: filecmp, Up: Improved Modules<10>
1.10.4.14 functools
...................
The new *note partialmethod(): a7b. descriptor brings partial argument
application to descriptors, just as *note partial(): 410. provides for
normal callables. The new descriptor also makes it easier to get
arbitrary callables (including *note partial(): 410. instances) to
behave like normal instance methods when included in a class definition.
(Contributed by Alon Horev and Nick Coghlan in bpo-4331(1).) The new
*note singledispatch(): 635. decorator brings support for
single-dispatch generic functions to the Python standard library. Where
object oriented programming focuses on grouping multiple operations on a
common set of data into a class, a generic function focuses on grouping
multiple implementations of an operation that allows it to work with
'different' kinds of data.
See also
........
PEP 443(2) – Single-dispatch generic functions
PEP written and implemented by Łukasz Langa.
*note total_ordering(): f3d. now supports a return value of *note
NotImplemented: 7cd. from the underlying comparison function.
(Contributed by Katie Miller in bpo-10042(3).)
A pure-python version of the *note partial(): 410. function is now in
the stdlib; in CPython it is overridden by the C accelerated version,
but it is available for other implementations to use. (Contributed by
Brian Thorne in bpo-12428(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4331
(2) https://peps.python.org/pep-0443/
(3) https://bugs.python.org/issue?@action=redirect&bpo=10042
(4) https://bugs.python.org/issue?@action=redirect&bpo=12428
File: python.info, Node: gc<5>, Next: glob<4>, Prev: functools<5>, Up: Improved Modules<10>
1.10.4.15 gc
............
New function *note get_stats(): f3f. returns a list of three
per-generation dictionaries containing the collections statistics since
interpreter startup. (Contributed by Antoine Pitrou in bpo-16351(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16351
File: python.info, Node: glob<4>, Next: hashlib<6>, Prev: gc<5>, Up: Improved Modules<10>
1.10.4.16 glob
..............
A new function *note escape(): f41. provides a way to escape special
characters in a filename so that they do not become part of the globbing
expansion but are instead matched literally. (Contributed by Serhiy
Storchaka in bpo-8402(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8402
File: python.info, Node: hashlib<6>, Next: hmac<3>, Prev: glob<4>, Up: Improved Modules<10>
1.10.4.17 hashlib
.................
A new *note hashlib.pbkdf2_hmac(): 50a. function provides the PKCS#5
password-based key derivation function 2(1). (Contributed by Christian
Heimes in bpo-18582(2).)
The *note name: f43. attribute of *note hashlib: 68. hash objects is now
a formally supported interface. It has always existed in CPython’s
*note hashlib: 68. (although it did not return lower case names for all
supported hashes), but it was not a public interface and so some other
Python implementations have not previously supported it. (Contributed
by Jason R. Coombs in bpo-18532(3).)
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/PBKDF2
(2) https://bugs.python.org/issue?@action=redirect&bpo=18582
(3) https://bugs.python.org/issue?@action=redirect&bpo=18532
File: python.info, Node: hmac<3>, Next: html, Prev: hashlib<6>, Up: Improved Modules<10>
1.10.4.18 hmac
..............
*note hmac: 6a. now accepts ‘bytearray’ as well as ‘bytes’ for the 'key'
argument to the *note new(): a9f. function, and the 'msg' parameter to
both the *note new(): a9f. function and the *note update(): f45. method
now accepts any type supported by the *note hashlib: 68. module.
(Contributed by Jonas Borgström in bpo-18240(1).)
The 'digestmod' argument to the *note hmac.new(): a9f. function may now
be any hash digest name recognized by *note hashlib: 68. In addition,
the current behavior in which the value of 'digestmod' defaults to ‘MD5’
is deprecated: in a future version of Python there will be no default
value. (Contributed by Christian Heimes in bpo-17276(2).)
With the addition of *note block_size: f46. and *note name: f47.
attributes (and the formal documentation of the *note digest_size: f48.
attribute), the *note hmac: 6a. module now conforms fully to the PEP
247(3) API. (Contributed by Christian Heimes in bpo-18775(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18240
(2) https://bugs.python.org/issue?@action=redirect&bpo=17276
(3) https://peps.python.org/pep-0247/
(4) https://bugs.python.org/issue?@action=redirect&bpo=18775
File: python.info, Node: html, Next: http<3>, Prev: hmac<3>, Up: Improved Modules<10>
1.10.4.19 html
..............
New function *note unescape(): 957. function converts HTML5 character
references to the corresponding Unicode characters. (Contributed by
Ezio Melotti in bpo-2927(1).)
*note HTMLParser: 874. accepts a new keyword argument 'convert_charrefs'
that, when ‘True’, automatically converts all character references. For
backward-compatibility, its value defaults to ‘False’, but it will
change to ‘True’ in a future version of Python, so you are invited to
set it explicitly and update your code to use this new feature.
(Contributed by Ezio Melotti in bpo-13633(2).)
The 'strict' argument of *note HTMLParser: 874. is now deprecated.
(Contributed by Ezio Melotti in bpo-15114(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2927
(2) https://bugs.python.org/issue?@action=redirect&bpo=13633
(3) https://bugs.python.org/issue?@action=redirect&bpo=15114
File: python.info, Node: http<3>, Next: idlelib and IDLE<4>, Prev: html, Up: Improved Modules<10>
1.10.4.20 http
..............
*note send_error(): f4b. now accepts an optional additional 'explain'
parameter which can be used to provide an extended error description,
overriding the hardcoded default if there is one. This extended error
description will be formatted using the ‘error_message_format’ attribute
and sent as the body of the error response. (Contributed by Karl Cow in
bpo-12921(1).)
The *note http.server: 72. *note command line interface: f4c. now has a
‘-b/--bind’ option that causes the server to listen on a specific
address. (Contributed by Malte Swart in bpo-17764(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12921
(2) https://bugs.python.org/issue?@action=redirect&bpo=17764
File: python.info, Node: idlelib and IDLE<4>, Next: importlib<9>, Prev: http<3>, Up: Improved Modules<10>
1.10.4.21 idlelib and IDLE
..........................
Since idlelib implements the IDLE shell and editor and is not intended
for import by other programs, it gets improvements with every release.
See ‘Lib/idlelib/NEWS.txt’ for a cumulative list of changes since 3.3.0,
as well as changes made in future 3.4.x releases. This file is also
available from the IDLE Help ‣ About IDLE dialog.
File: python.info, Node: importlib<9>, Next: inspect<8>, Prev: idlelib and IDLE<4>, Up: Improved Modules<10>
1.10.4.22 importlib
...................
The *note InspectLoader: f4f. ABC defines a new method, *note
source_to_code(): e10. that accepts source data and a path and returns a
code object. The default implementation is equivalent to ‘compile(data,
path, 'exec', dont_inherit=True)’. (Contributed by Eric Snow and Brett
Cannon in bpo-15627(1).)
*note InspectLoader: f4f. also now has a default implementation for the
*note get_code(): f50. method. However, it will normally be desirable
to override the default implementation for performance reasons.
(Contributed by Brett Cannon in bpo-18072(2).)
The *note reload(): 514. function has been moved from ‘imp’ to *note
importlib: 77. as part of the ‘imp’ module deprecation. (Contributed by
Berker Peksag in bpo-18193(3).)
*note importlib.util: 7d. now has a *note MAGIC_NUMBER: 50e. attribute
providing access to the bytecode version number. This replaces the
‘get_magic()’ function in the deprecated ‘imp’ module. (Contributed by
Brett Cannon in bpo-18192(4).)
New *note importlib.util: 7d. functions *note cache_from_source(): 2f8.
and *note source_from_cache(): 515. replace the same-named functions in
the deprecated ‘imp’ module. (Contributed by Brett Cannon in
bpo-18194(5).)
The *note importlib: 77. bootstrap *note NamespaceLoader: f51. now
conforms to the *note InspectLoader: f4f. ABC, which means that ‘runpy’
and ‘python -m’ can now be used with namespace packages. (Contributed
by Brett Cannon in bpo-18058(6).)
*note importlib.util: 7d. has a new function *note decode_source(): f52.
that decodes source from bytes using universal newline processing. This
is useful for implementing *note InspectLoader.get_source(): f53.
methods.
*note importlib.machinery.ExtensionFileLoader: cb0. now has a *note
get_filename(): f54. method. This was inadvertently omitted in the
original implementation. (Contributed by Eric Snow in bpo-19152(7).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15627
(2) https://bugs.python.org/issue?@action=redirect&bpo=18072
(3) https://bugs.python.org/issue?@action=redirect&bpo=18193
(4) https://bugs.python.org/issue?@action=redirect&bpo=18192
(5) https://bugs.python.org/issue?@action=redirect&bpo=18194
(6) https://bugs.python.org/issue?@action=redirect&bpo=18058
(7) https://bugs.python.org/issue?@action=redirect&bpo=19152
File: python.info, Node: inspect<8>, Next: ipaddress<5>, Prev: importlib<9>, Up: Improved Modules<10>
1.10.4.23 inspect
.................
The *note inspect: 7e. module now offers a basic *note command line
interface: f56. to quickly display source code and other information for
modules, classes and functions. (Contributed by Claudiu Popa and Nick
Coghlan in bpo-18626(1).)
*note unwrap(): f57. makes it easy to unravel wrapper function chains
created by *note functools.wraps(): f58. (and any other API that sets
the ‘__wrapped__’ attribute on a wrapper function). (Contributed by
Daniel Urban, Aaron Iles and Nick Coghlan in bpo-13266(2).)
As part of the implementation of the new *note enum: 56. module, the
*note inspect: 7e. module now has substantially better support for
custom ‘__dir__’ methods and dynamic class attributes provided through
metaclasses. (Contributed by Ethan Furman in bpo-18929(3) and
bpo-19030(4).)
*note getfullargspec(): 734. and ‘getargspec()’ now use the *note
signature(): 733. API. This allows them to support a much broader range
of callables, including those with ‘__signature__’ attributes, those
with metadata provided by argument clinic, *note functools.partial():
410. objects and more. Note that, unlike *note signature(): 733, these
functions still ignore ‘__wrapped__’ attributes, and report the already
bound first argument for bound methods, so it is still necessary to
update your code to use *note signature(): 733. directly if those
features are desired. (Contributed by Yury Selivanov in bpo-17481(5).)
*note signature(): 733. now supports duck types of CPython functions,
which adds support for functions compiled with Cython. (Contributed by
Stefan Behnel and Yury Selivanov in bpo-17159(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18626
(2) https://bugs.python.org/issue?@action=redirect&bpo=13266
(3) https://bugs.python.org/issue?@action=redirect&bpo=18929
(4) https://bugs.python.org/issue?@action=redirect&bpo=19030
(5) https://bugs.python.org/issue?@action=redirect&bpo=17481
(6) https://bugs.python.org/issue?@action=redirect&bpo=17159
File: python.info, Node: ipaddress<5>, Next: logging<6>, Prev: inspect<8>, Up: Improved Modules<10>
1.10.4.24 ipaddress
...................
*note ipaddress: 80. was added to the standard library in Python 3.3 as
a *note provisional API: b03. With the release of Python 3.4, this
qualification has been removed: *note ipaddress: 80. is now considered a
stable API, covered by the normal standard library requirements to
maintain backwards compatibility.
A new *note is_global: f5a. property is ‘True’ if an address is globally
routeable. (Contributed by Peter Moody in bpo-17400(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17400
File: python.info, Node: logging<6>, Next: marshal<2>, Prev: ipaddress<5>, Up: Improved Modules<10>
1.10.4.25 logging
.................
The *note TimedRotatingFileHandler: f5c. has a new 'atTime' parameter
that can be used to specify the time of day when rollover should happen.
(Contributed by Ronald Oussoren in bpo-9556(1).)
*note SocketHandler: f5d. and *note DatagramHandler: f5e. now support
Unix domain sockets (by setting 'port' to ‘None’). (Contributed by
Vinay Sajip in commit ce46195b56a9.)
*note fileConfig(): b51. now accepts a *note
configparser.RawConfigParser: f5f. subclass instance for the 'fname'
parameter. This facilitates using a configuration file when logging
configuration is just a part of the overall application configuration,
or where the application modifies the configuration before passing it to
*note fileConfig(): b51. (Contributed by Vinay Sajip in bpo-16110(2).)
Logging configuration data received from a socket via the *note
logging.config.listen(): f60. function can now be validated before being
processed by supplying a verification function as the argument to the
new 'verify' keyword argument. (Contributed by Vinay Sajip in
bpo-15452(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9556
(2) https://bugs.python.org/issue?@action=redirect&bpo=16110
(3) https://bugs.python.org/issue?@action=redirect&bpo=15452
File: python.info, Node: marshal<2>, Next: mmap<3>, Prev: logging<6>, Up: Improved Modules<10>
1.10.4.26 marshal
.................
The default *note marshal: 8d. version has been bumped to 3. The code
implementing the new version restores the Python2 behavior of recording
only one copy of interned strings and preserving the interning on
deserialization, and extends this “one copy” ability to any object type
(including handling recursive references). This reduces both the size
of ‘.pyc’ files and the amount of memory a module occupies in memory
when it is loaded from a ‘.pyc’ (or ‘.pyo’) file. (Contributed by
Kristján Valur Jónsson in bpo-16475(1), with additional speedups by
Antoine Pitrou in bpo-19219(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16475
(2) https://bugs.python.org/issue?@action=redirect&bpo=19219
File: python.info, Node: mmap<3>, Next: multiprocessing<7>, Prev: marshal<2>, Up: Improved Modules<10>
1.10.4.27 mmap
..............
mmap objects are now *note weakly referenceable: f05. (Contributed by
Valerie Lambert in bpo-4885(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4885
File: python.info, Node: multiprocessing<7>, Next: operator<3>, Prev: mmap<3>, Up: Improved Modules<10>
1.10.4.28 multiprocessing
.........................
On Unix two new *note start methods: 2cb, ‘spawn’ and ‘forkserver’, have
been added for starting processes using *note multiprocessing: 94.
These make the mixing of processes with threads more robust, and the
‘spawn’ method matches the semantics that multiprocessing has always
used on Windows. New function *note get_all_start_methods(): f64.
reports all start methods available on the platform, *note
get_start_method(): f65. reports the current start method, and *note
set_start_method(): 2ca. sets the start method. (Contributed by Richard
Oudkerk in bpo-8713(1).)
*note multiprocessing: 94. also now has the concept of a ‘context’,
which determines how child processes are created. New function *note
get_context(): 2c9. returns a context that uses a specified start
method. It has the same API as the *note multiprocessing: 94. module
itself, so you can use it to create *note Pool: f66.s and other objects
that will operate within that context. This allows a framework and an
application or different parts of the same application to use
multiprocessing without interfering with each other. (Contributed by
Richard Oudkerk in bpo-18999(2).)
Except when using the old 'fork' start method, child processes no longer
inherit unneeded handles/file descriptors from their parents (part of
bpo-8713(3)).
*note multiprocessing: 94. now relies on *note runpy: be. (which
implements the ‘-m’ switch) to initialise ‘__main__’ appropriately in
child processes when using the ‘spawn’ or ‘forkserver’ start methods.
This resolves some edge cases where combining multiprocessing, the ‘-m’
command line switch, and explicit relative imports could cause obscure
failures in child processes. (Contributed by Nick Coghlan in
bpo-19946(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8713
(2) https://bugs.python.org/issue?@action=redirect&bpo=18999
(3) https://bugs.python.org/issue?@action=redirect&bpo=8713
(4) https://bugs.python.org/issue?@action=redirect&bpo=19946
File: python.info, Node: operator<3>, Next: os<11>, Prev: multiprocessing<7>, Up: Improved Modules<10>
1.10.4.29 operator
..................
New function *note length_hint(): f68. provides an implementation of the
specification for how the *note __length_hint__(): f06. special method
should be used, as part of the PEP 424(1) formal specification of this
language feature. (Contributed by Armin Ronacher in bpo-16148(2).)
There is now a pure-python version of the *note operator: 9f. module
available for reference and for use by alternate implementations of
Python. (Contributed by Zachary Ware in bpo-16694(3).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0424/
(2) https://bugs.python.org/issue?@action=redirect&bpo=16148
(3) https://bugs.python.org/issue?@action=redirect&bpo=16694
File: python.info, Node: os<11>, Next: pdb<6>, Prev: operator<3>, Up: Improved Modules<10>
1.10.4.30 os
............
There are new functions to get and set the *note inheritable flag: ef4.
of a file descriptor (*note os.get_inheritable(): ef5, *note
os.set_inheritable(): ef6.) or a Windows handle (*note
os.get_handle_inheritable(): ef7, *note os.set_handle_inheritable():
ef8.).
New function *note cpu_count(): 1c5. reports the number of CPUs
available on the platform on which Python is running (or ‘None’ if the
count can’t be determined). The *note multiprocessing.cpu_count(): f6a.
function is now implemented in terms of this function). (Contributed by
Trent Nelson, Yogesh Chaudhari, Victor Stinner, and Charles-François
Natali in bpo-17914(1).)
*note os.path.samestat(): f6b. is now available on the Windows platform
(and the *note os.path.samefile(): f6c. implementation is now shared
between Unix and Windows). (Contributed by Brian Curtin in
bpo-11939(2).)
*note os.path.ismount(): a11. now recognizes volumes mounted below a
drive root on Windows. (Contributed by Tim Golden in bpo-9035(3).)
*note os.open(): d91. supports two new flags on platforms that provide
them, *note O_PATH: f6d. (un-opened file descriptor), and *note
O_TMPFILE: f6e. (unnamed temporary file; as of 3.4.0 release available
only on Linux systems with a kernel version of 3.11 or newer that have
uapi headers). (Contributed by Christian Heimes in bpo-18673(4) and
Benjamin Peterson, respectively.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17914
(2) https://bugs.python.org/issue?@action=redirect&bpo=11939
(3) https://bugs.python.org/issue?@action=redirect&bpo=9035
(4) https://bugs.python.org/issue?@action=redirect&bpo=18673
File: python.info, Node: pdb<6>, Next: pickle<4>, Prev: os<11>, Up: Improved Modules<10>
1.10.4.31 pdb
.............
*note pdb: a5. has been enhanced to handle generators, *note yield: 9cd,
and ‘yield from’ in a more useful fashion. This is especially helpful
when debugging *note asyncio: a. based programs. (Contributed by Andrew
Svetlov and Xavier de Gaye in bpo-16596(1).)
The ‘print’ command has been removed from *note pdb: a5, restoring
access to the Python *note print(): f70. function from the pdb command
line. Python2’s ‘pdb’ did not have a ‘print’ command; instead, entering
‘print’ executed the ‘print’ statement. In Python3 ‘print’ was
mistakenly made an alias for the pdb *note p: f71. command. ‘p’,
however, prints the ‘repr’ of its argument, not the ‘str’ like the
Python2 ‘print’ command did. Worse, the Python3 ‘pdb print’ command
shadowed the Python3 ‘print’ function, making it inaccessible at the
‘pdb’ prompt. (Contributed by Connor Osborn in bpo-18764(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16596
(2) https://bugs.python.org/issue?@action=redirect&bpo=18764
File: python.info, Node: pickle<4>, Next: plistlib<2>, Prev: pdb<6>, Up: Improved Modules<10>
1.10.4.32 pickle
................
*note pickle: a6. now supports (but does not use by default) a new
pickle protocol, protocol 4. This new protocol addresses a number of
issues that were present in previous protocols, such as the
serialization of nested classes, very large strings and containers, and
classes whose ‘__new__()’ method takes keyword-only arguments. It also
provides some efficiency improvements.
See also
........
PEP 3154(1) – Pickle protocol 4
PEP written by Antoine Pitrou and implemented by Alexandre
Vassalotti.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3154/
File: python.info, Node: plistlib<2>, Next: poplib<3>, Prev: pickle<4>, Up: Improved Modules<10>
1.10.4.33 plistlib
..................
*note plistlib: ab. now has an API that is similar to the standard
pattern for stdlib serialization protocols, with new *note load(): 94f,
*note dump(): 951, *note loads(): 950, and *note dumps(): 952.
functions. (The older API is now deprecated.) In addition to the
already supported XML plist format (*note FMT_XML: f74.), it also now
supports the binary plist format (*note FMT_BINARY: f75.). (Contributed
by Ronald Oussoren and others in bpo-14455(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14455
File: python.info, Node: poplib<3>, Next: pprint<4>, Prev: plistlib<2>, Up: Improved Modules<10>
1.10.4.34 poplib
................
Two new methods have been added to *note poplib: ac.: *note capa(): f77,
which returns the list of capabilities advertised by the POP server, and
*note stls(): f78, which switches a clear-text POP3 session into an
encrypted POP3 session if the POP server supports it. (Contributed by
Lorenzo Catucci in bpo-4473(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4473
File: python.info, Node: pprint<4>, Next: pty, Prev: poplib<3>, Up: Improved Modules<10>
1.10.4.35 pprint
................
The *note pprint: ae. module’s *note PrettyPrinter: f7a. class and its
*note pformat(): f7b, and *note pprint(): 824. functions have a new
option, 'compact', that controls how the output is formatted. Currently
setting 'compact' to ‘True’ means that sequences will be printed with as
many sequence elements as will fit within 'width' on each (indented)
line. (Contributed by Serhiy Storchaka in bpo-19132(1).)
Long strings are now wrapped using Python’s normal line continuation
syntax. (Contributed by Antoine Pitrou in bpo-17150(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19132
(2) https://bugs.python.org/issue?@action=redirect&bpo=17150
File: python.info, Node: pty, Next: pydoc<4>, Prev: pprint<4>, Up: Improved Modules<10>
1.10.4.36 pty
.............
*note pty.spawn(): f7d. now returns the status value from *note
os.waitpid(): d99. on the child process, instead of ‘None’.
(Contributed by Gregory P. Smith.)
File: python.info, Node: pydoc<4>, Next: re<8>, Prev: pty, Up: Improved Modules<10>
1.10.4.37 pydoc
...............
The *note pydoc: b5. module is now based directly on the *note
inspect.signature(): 733. introspection API, allowing it to provide
signature information for a wider variety of callable objects. This
change also means that ‘__wrapped__’ attributes are now taken into
account when displaying help information. (Contributed by Larry
Hastings in bpo-19674(1).)
The *note pydoc: b5. module no longer displays the ‘self’ parameter for
already bound methods. Instead, it aims to always display the exact
current signature of the supplied callable. (Contributed by Larry
Hastings in bpo-20710(2).)
In addition to the changes that have been made to *note pydoc: b5.
directly, its handling of custom ‘__dir__’ methods and various
descriptor behaviours has also been improved substantially by the
underlying changes in the *note inspect: 7e. module.
As the *note help(): 8d6. builtin is based on *note pydoc: b5, the above
changes also affect the behaviour of *note help(): 8d6.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19674
(2) https://bugs.python.org/issue?@action=redirect&bpo=20710
File: python.info, Node: re<8>, Next: resource, Prev: pydoc<4>, Up: Improved Modules<10>
1.10.4.38 re
............
New *note fullmatch(): f80. function and ‘regex.fullmatch()’ method
anchor the pattern at both ends of the string to match. This provides a
way to be explicit about the goal of the match, which avoids a class of
subtle bugs where ‘$’ characters get lost during code changes or the
addition of alternatives to an existing regular expression.
(Contributed by Matthew Barnett in bpo-16203(1).)
The repr of *note regex objects: f81. now includes the pattern and the
flags; the repr of *note match objects: f82. now includes the start,
end, and the part of the string that matched. (Contributed by Hugo
Lopes Tavares and Serhiy Storchaka in bpo-13592(2) and bpo-17087(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16203
(2) https://bugs.python.org/issue?@action=redirect&bpo=13592
(3) https://bugs.python.org/issue?@action=redirect&bpo=17087
File: python.info, Node: resource, Next: select, Prev: re<8>, Up: Improved Modules<10>
1.10.4.39 resource
..................
New *note prlimit(): f84. function, available on Linux platforms with a
kernel version of 2.6.36 or later and glibc of 2.13 or later, provides
the ability to query or set the resource limits for processes other than
the one making the call. (Contributed by Christian Heimes in
bpo-16595(1).)
On Linux kernel version 2.6.36 or later, there are also some new Linux
specific constants: *note RLIMIT_MSGQUEUE: f85, *note RLIMIT_NICE: f86,
*note RLIMIT_RTPRIO: f87, *note RLIMIT_RTTIME: f88, and *note
RLIMIT_SIGPENDING: f89. (Contributed by Christian Heimes in
bpo-19324(2).)
On FreeBSD version 9 and later, there some new FreeBSD specific
constants: *note RLIMIT_SBSIZE: f8a, *note RLIMIT_SWAP: f8b, and *note
RLIMIT_NPTS: f8c. (Contributed by Claudiu Popa in bpo-19343(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16595
(2) https://bugs.python.org/issue?@action=redirect&bpo=19324
(3) https://bugs.python.org/issue?@action=redirect&bpo=19343
File: python.info, Node: select, Next: shelve<2>, Prev: resource, Up: Improved Modules<10>
1.10.4.40 select
................
*note epoll: f8e. objects now support the context management protocol.
When used in a *note with: 5ce. statement, the *note close(): f8f.
method will be called automatically at the end of the block.
(Contributed by Serhiy Storchaka in bpo-16488(1).)
*note devpoll: f90. objects now have *note fileno(): f91. and *note
close(): f92. methods, as well as a new attribute *note closed: f93.
(Contributed by Victor Stinner in bpo-18794(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16488
(2) https://bugs.python.org/issue?@action=redirect&bpo=18794
File: python.info, Node: shelve<2>, Next: shutil<6>, Prev: select, Up: Improved Modules<10>
1.10.4.41 shelve
................
*note Shelf: f95. instances may now be used in *note with: 5ce.
statements, and will be automatically closed at the end of the ‘with’
block. (Contributed by Filip Gruszczyński in bpo-13896(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13896
File: python.info, Node: shutil<6>, Next: smtpd<3>, Prev: shelve<2>, Up: Improved Modules<10>
1.10.4.42 shutil
................
*note copyfile(): a58. now raises a specific *note Error: f97. subclass,
*note SameFileError: f98, when the source and destination are the same
file, which allows an application to take appropriate action on this
specific error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in
bpo-1492704(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1492704
File: python.info, Node: smtpd<3>, Next: smtplib<3>, Prev: shutil<6>, Up: Improved Modules<10>
1.10.4.43 smtpd
...............
The ‘SMTPServer’ and ‘SMTPChannel’ classes now accept a 'map' keyword
argument which, if specified, is passed in to ‘asynchat.async_chat’ as
its 'map' argument. This allows an application to avoid affecting the
global socket map. (Contributed by Vinay Sajip in bpo-11959(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11959
File: python.info, Node: smtplib<3>, Next: socket<9>, Prev: smtpd<3>, Up: Improved Modules<10>
1.10.4.44 smtplib
.................
*note SMTPException: f9b. is now a subclass of *note OSError: 413, which
allows both socket level errors and SMTP protocol level errors to be
caught in one try/except statement by code that only cares whether or
not an error occurred. (Contributed by Ned Jackson Lovely in
bpo-2118(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2118
File: python.info, Node: socket<9>, Next: sqlite3<9>, Prev: smtplib<3>, Up: Improved Modules<10>
1.10.4.45 socket
................
The socket module now supports the *note CAN_BCM: f9d. protocol on
platforms that support it. (Contributed by Brian Thorne in
bpo-15359(1).)
Socket objects have new methods to get or set their *note inheritable
flag: ef4, *note get_inheritable(): ef9. and *note set_inheritable():
efa.
The ‘socket.AF_*’ and ‘socket.SOCK_*’ constants are now enumeration
values using the new *note enum: 56. module. This allows meaningful
names to be printed during debugging, instead of integer “magic
numbers”.
The *note AF_LINK: f9e. constant is now available on BSD and OSX.
*note inet_pton(): 962. and *note inet_ntop(): f9f. are now supported on
Windows. (Contributed by Atsuo Ishimoto in bpo-7171(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15359
(2) https://bugs.python.org/issue?@action=redirect&bpo=7171
File: python.info, Node: sqlite3<9>, Next: ssl<10>, Prev: socket<9>, Up: Improved Modules<10>
1.10.4.46 sqlite3
.................
A new boolean parameter to the *note connect(): 2aa. function, 'uri',
can be used to indicate that the 'database' parameter is a ‘uri’ (see
the SQLite URI documentation(1)). (Contributed by poq in bpo-13773(2).)
---------- Footnotes ----------
(1) https://www.sqlite.org/uri.html
(2) https://bugs.python.org/issue?@action=redirect&bpo=13773
File: python.info, Node: ssl<10>, Next: stat, Prev: sqlite3<9>, Up: Improved Modules<10>
1.10.4.47 ssl
.............
*note PROTOCOL_TLSv1_1: fa2. and *note PROTOCOL_TLSv1_2: fa3. (TLSv1.1
and TLSv1.2 support) have been added; support for these protocols is
only available if Python is linked with OpenSSL 1.0.1 or later.
(Contributed by Michele Orrù and Antoine Pitrou in bpo-16692(1).) New
function *note create_default_context(): 155. provides a standard way to
obtain an *note SSLContext: 296. whose settings are intended to be a
reasonable balance between compatibility and security. These settings
are more stringent than the defaults provided by the *note SSLContext:
296. constructor, and may be adjusted in the future, without prior
deprecation, if best-practice security requirements change. The new
recommended best practice for using stdlib libraries that support SSL is
to use *note create_default_context(): 155. to obtain an *note
SSLContext: 296. object, modify it if needed, and then pass it as the
'context' argument of the appropriate stdlib API. (Contributed by
Christian Heimes in bpo-19689(2).)
*note SSLContext: 296. method *note load_verify_locations(): ee6.
accepts a new optional argument 'cadata', which can be used to provide
PEM or DER encoded certificates directly via strings or bytes,
respectively. (Contributed by Christian Heimes in bpo-18138(3).)
New function *note get_default_verify_paths(): fa4. returns a named
tuple of the paths and environment variables that the *note
set_default_verify_paths(): fa5. method uses to set OpenSSL’s default
‘cafile’ and ‘capath’. This can be an aid in debugging default
verification issues. (Contributed by Christian Heimes in bpo-18143(4).)
*note SSLContext: 296. has a new method, *note cert_store_stats(): fa6,
that reports the number of loaded ‘X.509’ certs, ‘X.509 CA’ certs, and
certificate revocation lists (‘crl’s), as well as a *note
get_ca_certs(): fa7. method that returns a list of the loaded ‘CA’
certificates. (Contributed by Christian Heimes in bpo-18147(5).)
If OpenSSL 0.9.8 or later is available, *note SSLContext: 296. has a new
attribute *note verify_flags: fa8. that can be used to control the
certificate verification process by setting it to some combination of
the new constants *note VERIFY_DEFAULT: fa9, *note
VERIFY_CRL_CHECK_LEAF: faa, *note VERIFY_CRL_CHECK_CHAIN: fab, or *note
VERIFY_X509_STRICT: 157. OpenSSL does not do any CRL verification by
default. (Contributed by Christien Heimes in bpo-8813(6).)
New *note SSLContext: 296. method *note load_default_certs(): fac. loads
a set of default “certificate authority” (CA) certificates from default
locations, which vary according to the platform. It can be used to load
both TLS web server authentication certificates (‘purpose=’*note
SERVER_AUTH: fad.) for a client to use to verify a server, and
certificates for a server to use in verifying client certificates
(‘purpose=’*note CLIENT_AUTH: fae.). (Contributed by Christian Heimes
in bpo-19292(7).) Two new windows-only functions, *note
enum_certificates(): faf. and *note enum_crls(): fb0. provide the
ability to retrieve certificates, certificate information, and CRLs from
the Windows cert store. (Contributed by Christian Heimes in
bpo-17134(8).) Support for server-side SNI (Server Name Indication)
using the new *note ssl.SSLContext.set_servername_callback(): fb1.
method. (Contributed by Daniel Black in bpo-8109(9).)
The dictionary returned by *note SSLSocket.getpeercert(): fb2. contains
additional ‘X509v3’ extension items: ‘crlDistributionPoints’,
‘calIssuers’, and ‘OCSP’ URIs. (Contributed by Christian Heimes in
bpo-18379(10).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16692
(2) https://bugs.python.org/issue?@action=redirect&bpo=19689
(3) https://bugs.python.org/issue?@action=redirect&bpo=18138
(4) https://bugs.python.org/issue?@action=redirect&bpo=18143
(5) https://bugs.python.org/issue?@action=redirect&bpo=18147
(6) https://bugs.python.org/issue?@action=redirect&bpo=8813
(7) https://bugs.python.org/issue?@action=redirect&bpo=19292
(8) https://bugs.python.org/issue?@action=redirect&bpo=17134
(9) https://bugs.python.org/issue?@action=redirect&bpo=8109
(10) https://bugs.python.org/issue?@action=redirect&bpo=18379
File: python.info, Node: stat, Next: struct<2>, Prev: ssl<10>, Up: Improved Modules<10>
1.10.4.48 stat
..............
The *note stat: d1. module is now backed by a C implementation in
‘_stat’. A C implementation is required as most of the values aren’t
standardized and are platform-dependent. (Contributed by Christian
Heimes in bpo-11016(1).)
The module supports new *note ST_MODE: fb4. flags, *note S_IFDOOR: fb5,
*note S_IFPORT: fb6, and *note S_IFWHT: fb7. (Contributed by Christian
Hiemes in bpo-11016(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11016
(2) https://bugs.python.org/issue?@action=redirect&bpo=11016
File: python.info, Node: struct<2>, Next: subprocess<5>, Prev: stat, Up: Improved Modules<10>
1.10.4.49 struct
................
New function *note iter_unpack: fb9. and a new *note
struct.Struct.iter_unpack(): fba. method on compiled formats provide
streamed unpacking of a buffer containing repeated instances of a given
format of data. (Contributed by Antoine Pitrou in bpo-17804(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17804
File: python.info, Node: subprocess<5>, Next: sunau<2>, Prev: struct<2>, Up: Improved Modules<10>
1.10.4.50 subprocess
....................
*note check_output(): fbc. now accepts an 'input' argument that can be
used to provide the contents of ‘stdin’ for the command that is run.
(Contributed by Zack Weinberg in bpo-16624(1).)
‘getstatus()’ and *note getstatusoutput(): fbd. now work on Windows.
This change was actually inadvertently made in 3.3.4. (Contributed by
Tim Golden in bpo-10197(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16624
(2) https://bugs.python.org/issue?@action=redirect&bpo=10197
File: python.info, Node: sunau<2>, Next: sys<12>, Prev: subprocess<5>, Up: Improved Modules<10>
1.10.4.51 sunau
...............
The ‘getparams()’ method now returns a namedtuple rather than a plain
tuple. (Contributed by Claudiu Popa in bpo-18901(1).)
‘sunau.open()’ now supports the context management protocol: when used
in a *note with: 5ce. block, the ‘close’ method of the returned object
will be called automatically at the end of the block. (Contributed by
Serhiy Storchaka in bpo-18878(2).)
‘AU_write.setsampwidth()’ now supports 24 bit samples, thus adding
support for writing 24 sample using the module. (Contributed by Serhiy
Storchaka in bpo-19261(3).)
The ‘writeframesraw()’ and ‘writeframes()’ methods now accept any *note
bytes-like object: d2d. (Contributed by Serhiy Storchaka in
bpo-8311(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18901
(2) https://bugs.python.org/issue?@action=redirect&bpo=18878
(3) https://bugs.python.org/issue?@action=redirect&bpo=19261
(4) https://bugs.python.org/issue?@action=redirect&bpo=8311
File: python.info, Node: sys<12>, Next: tarfile<8>, Prev: sunau<2>, Up: Improved Modules<10>
1.10.4.52 sys
.............
New function *note sys.getallocatedblocks(): fc0. returns the current
number of blocks allocated by the interpreter. (In CPython with the
default ‘--with-pymalloc’ setting, this is allocations made through the
*note PyObject_Malloc(): c68. API.) This can be useful for tracking
memory leaks, especially if automated via a test suite. (Contributed by
Antoine Pitrou in bpo-13390(1).)
When the Python interpreter starts in *note interactive mode: fc1, it
checks for an *note __interactivehook__: fc2. attribute on the *note
sys: d9. module. If the attribute exists, its value is called with no
arguments just before interactive mode is started. The check is made
after the *note PYTHONSTARTUP: fc3. file is read, so it can be set
there. The *note site: c7. module *note sets it: fc4. to a function
that enables tab completion and history saving (in ‘~/.python-history’)
if the platform supports *note readline: ba. If you do not want this
(new) behavior, you can override it in *note PYTHONSTARTUP: fc3, *note
sitecustomize: c8, or *note usercustomize: 10e. by deleting this
attribute from *note sys: d9. (or setting it to some other callable).
(Contributed by Éric Araujo and Antoine Pitrou in bpo-5845(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13390
(2) https://bugs.python.org/issue?@action=redirect&bpo=5845
File: python.info, Node: tarfile<8>, Next: textwrap, Prev: sys<12>, Up: Improved Modules<10>
1.10.4.53 tarfile
.................
The *note tarfile: de. module now supports a simple *note Command-Line
Interface: fc6. when called as a script directly or via ‘-m’. This can
be used to create and extract tarfile archives. (Contributed by Berker
Peksag in bpo-13477(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13477
File: python.info, Node: textwrap, Next: threading<7>, Prev: tarfile<8>, Up: Improved Modules<10>
1.10.4.54 textwrap
..................
The *note TextWrapper: fc8. class has two new attributes/constructor
arguments: *note max_lines: fc9, which limits the number of lines in the
output, and *note placeholder: fca, which is a string that will appear
at the end of the output if it has been truncated because of
'max_lines'. Building on these capabilities, a new convenience function
*note shorten(): fcb. collapses all of the whitespace in the input to
single spaces and produces a single line of a given 'width' that ends
with the 'placeholder' (by default, ‘[...]’). (Contributed by Antoine
Pitrou and Serhiy Storchaka in bpo-18585(1) and bpo-18725(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18585
(2) https://bugs.python.org/issue?@action=redirect&bpo=18725
File: python.info, Node: threading<7>, Next: traceback<6>, Prev: textwrap, Up: Improved Modules<10>
1.10.4.55 threading
...................
The *note Thread: 94c. object representing the main thread can be
obtained from the new *note main_thread(): fcd. function. In normal
conditions this will be the thread from which the Python interpreter was
started. (Contributed by Andrew Svetlov in bpo-18882(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18882
File: python.info, Node: traceback<6>, Next: types<6>, Prev: threading<7>, Up: Improved Modules<10>
1.10.4.56 traceback
...................
A new *note traceback.clear_frames(): fcf. function takes a traceback
object and clears the local variables in all of the frames it
references, reducing the amount of memory consumed. (Contributed by
Andrew Kuchling in bpo-1565525(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1565525
File: python.info, Node: types<6>, Next: urllib<3>, Prev: traceback<6>, Up: Improved Modules<10>
1.10.4.57 types
...............
A new *note DynamicClassAttribute(): 626. descriptor provides a way to
define an attribute that acts normally when looked up through an
instance object, but which is routed to the 'class' ‘__getattr__’ when
looked up through the class. This allows one to have properties active
on a class, and have virtual attributes on the class with the same name
(see ‘Enum’ for an example). (Contributed by Ethan Furman in
bpo-19030(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19030
File: python.info, Node: urllib<3>, Next: unittest<9>, Prev: types<6>, Up: Improved Modules<10>
1.10.4.58 urllib
................
*note urllib.request: 10b. now supports ‘data:’ URLs via the *note
DataHandler: fd2. class. (Contributed by Mathias Panzenböck in
bpo-16423(1).)
The http method that will be used by a *note Request: fd3. class can now
be specified by setting a *note method: fd4. class attribute on the
subclass. (Contributed by Jason R Coombs in bpo-18978(2).)
*note Request: fd3. objects are now reusable: if the *note full_url:
fd5. or *note data: fd6. attributes are modified, all relevant internal
properties are updated. This means, for example, that it is now
possible to use the same *note Request: fd3. object in more than one
*note OpenerDirector.open(): fd7. call with different 'data' arguments,
or to modify a *note Request: fd3.‘s ‘url’ rather than recomputing it
from scratch. There is also a new *note remove_header(): fd8. method
that can be used to remove headers from a *note Request: fd3.
(Contributed by Alexey Kachayev in bpo-16464(3), Daniel Wozniak in
bpo-17485(4), and Damien Brecht and Senthil Kumaran in bpo-17272(5).)
*note HTTPError: fd9. objects now have a *note headers: fda. attribute
that provides access to the HTTP response headers associated with the
error. (Contributed by Berker Peksag in bpo-15701(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16423
(2) https://bugs.python.org/issue?@action=redirect&bpo=18978
(3) https://bugs.python.org/issue?@action=redirect&bpo=16464
(4) https://bugs.python.org/issue?@action=redirect&bpo=17485
(5) https://bugs.python.org/issue?@action=redirect&bpo=17272
(6) https://bugs.python.org/issue?@action=redirect&bpo=15701
File: python.info, Node: unittest<9>, Next: venv<7>, Prev: urllib<3>, Up: Improved Modules<10>
1.10.4.59 unittest
..................
The *note TestCase: 449. class has a new method, *note subTest(): fdc,
that produces a context manager whose *note with: 5ce. block becomes a
“sub-test”. This context manager allows a test method to dynamically
generate subtests by, say, calling the ‘subTest’ context manager inside
a loop. A single test method can thereby produce an indefinite number
of separately identified and separately counted tests, all of which will
run even if one or more of them fail. For example:
class NumbersTest(unittest.TestCase):
def test_even(self):
for i in range(6):
with self.subTest(i=i):
self.assertEqual(i % 2, 0)
will result in six subtests, each identified in the unittest verbose
output with a label consisting of the variable name ‘i’ and a particular
value for that variable (‘i=0’, ‘i=1’, etc). See *note Distinguishing
test iterations using subtests: fdd. for the full version of this
example. (Contributed by Antoine Pitrou in bpo-16997(1).)
*note unittest.main(): fde. now accepts an iterable of test names for
'defaultTest', where previously it only accepted a single test name as a
string. (Contributed by Jyrki Pulliainen in bpo-15132(2).)
If *note SkipTest: fdf. is raised during test discovery (that is, at the
module level in the test file), it is now reported as a skip instead of
an error. (Contributed by Zach Ware in bpo-16935(3).)
*note discover(): fe0. now sorts the discovered files to provide
consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani
in bpo-16709(4).)
*note TestSuite: df3. now drops references to tests as soon as the test
has been run, if the test is successful. On Python interpreters that do
garbage collection, this allows the tests to be garbage collected if
nothing else is holding a reference to the test. It is possible to
override this behavior by creating a *note TestSuite: df3. subclass that
defines a custom ‘_removeTestAtIndex’ method. (Contributed by Tom
Wardill, Matt McClure, and Andrew Svetlov in bpo-11798(5).)
A new test assertion context-manager, *note assertLogs(): 855, will
ensure that a given block of code emits a log message using the *note
logging: 87. module. By default the message can come from any logger
and have a priority of ‘INFO’ or higher, but both the logger name and an
alternative minimum logging level may be specified. The object returned
by the context manager can be queried for the *note LogRecord: fe1.s
and/or formatted messages that were logged. (Contributed by Antoine
Pitrou in bpo-18937(6).)
Test discovery now works with namespace packages (Contributed by Claudiu
Popa in bpo-17457(7).)
*note unittest.mock: 107. objects now inspect their specification
signatures when matching calls, which means an argument can now be
matched by either position or name, instead of only by position.
(Contributed by Antoine Pitrou in bpo-17015(8).)
‘mock_open()’ objects now have ‘readline’ and ‘readlines’ methods.
(Contributed by Toshio Kuratomi in bpo-17467(9).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16997
(2) https://bugs.python.org/issue?@action=redirect&bpo=15132
(3) https://bugs.python.org/issue?@action=redirect&bpo=16935
(4) https://bugs.python.org/issue?@action=redirect&bpo=16709
(5) https://bugs.python.org/issue?@action=redirect&bpo=11798
(6) https://bugs.python.org/issue?@action=redirect&bpo=18937
(7) https://bugs.python.org/issue?@action=redirect&bpo=17457
(8) https://bugs.python.org/issue?@action=redirect&bpo=17015
(9) https://bugs.python.org/issue?@action=redirect&bpo=17467
File: python.info, Node: venv<7>, Next: wave<2>, Prev: unittest<9>, Up: Improved Modules<10>
1.10.4.60 venv
..............
*note venv: 111. now includes activation scripts for the ‘csh’ and
‘fish’ shells. (Contributed by Andrew Svetlov in bpo-15417(1).)
*note EnvBuilder: 26c. and the *note create(): 26d. convenience function
take a new keyword argument 'with_pip', which defaults to ‘False’, that
controls whether or not *note EnvBuilder: 26c. ensures that ‘pip’ is
installed in the virtual environment. (Contributed by Nick Coghlan in
bpo-19552(2) as part of the PEP 453(3) implementation.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15417
(2) https://bugs.python.org/issue?@action=redirect&bpo=19552
(3) https://peps.python.org/pep-0453/
File: python.info, Node: wave<2>, Next: weakref<2>, Prev: venv<7>, Up: Improved Modules<10>
1.10.4.61 wave
..............
The ‘getparams()’ method now returns a namedtuple rather than a plain
tuple. (Contributed by Claudiu Popa in bpo-17487(1).)
*note wave.open(): 94b. now supports the context management protocol.
(Contributed by Claudiu Popa in bpo-17616(2).)
*note wave: 113. can now *note write output to unseekable files: fe4.
(Contributed by David Jones, Guilherme Polo, and Serhiy Storchaka in
bpo-5202(3).)
The *note writeframesraw(): fe5. and *note writeframes(): fe6. methods
now accept any *note bytes-like object: d2d. (Contributed by Serhiy
Storchaka in bpo-8311(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17487
(2) https://bugs.python.org/issue?@action=redirect&bpo=17616
(3) https://bugs.python.org/issue?@action=redirect&bpo=5202
(4) https://bugs.python.org/issue?@action=redirect&bpo=8311
File: python.info, Node: weakref<2>, Next: xml etree<2>, Prev: wave<2>, Up: Improved Modules<10>
1.10.4.62 weakref
.................
New *note WeakMethod: fe8. class simulates weak references to bound
methods. (Contributed by Antoine Pitrou in bpo-14631(1).)
New *note finalize: a7c. class makes it possible to register a callback
to be invoked when an object is garbage collected, without needing to
carefully manage the lifecycle of the weak reference itself.
(Contributed by Richard Oudkerk in bpo-15528(2).)
The callback, if any, associated with a *note ref: fe9. is now exposed
via the *note __callback__: fea. attribute. (Contributed by Mark
Dickinson in bpo-17643(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14631
(2) https://bugs.python.org/issue?@action=redirect&bpo=15528
(3) https://bugs.python.org/issue?@action=redirect&bpo=17643
File: python.info, Node: xml etree<2>, Next: zipfile<5>, Prev: weakref<2>, Up: Improved Modules<10>
1.10.4.63 xml.etree
...................
A new parser, *note XMLPullParser: fec, allows a non-blocking
applications to parse XML documents. An example can be seen at *note
Pull API for non-blocking parsing: fed. (Contributed by Antoine Pitrou
in bpo-17741(1).)
The *note xml.etree.ElementTree: 125. *note tostring(): fee. and *note
tostringlist(): fef. functions, and the *note ElementTree: 94e. *note
write(): ff0. method, now have a 'short_empty_elements' *note
keyword-only parameter: 2a7. providing control over whether elements
with no content are written in abbreviated (‘ ’) or expanded
(‘ ’) form. (Contributed by Ariel Poliak and Serhiy Storchaka
in bpo-14377(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17741
(2) https://bugs.python.org/issue?@action=redirect&bpo=14377
File: python.info, Node: zipfile<5>, Prev: xml etree<2>, Up: Improved Modules<10>
1.10.4.64 zipfile
.................
The *note writepy(): ff2. method of the *note PyZipFile: ff3. class has
a new 'filterfunc' option that can be used to control which directories
and files are added to the archive. For example, this could be used to
exclude test files from the archive. (Contributed by Christian Tismer
in bpo-19274(1).)
The 'allowZip64' parameter to *note ZipFile: 6c0. and ‘PyZipfile’ is now
‘True’ by default. (Contributed by William Mallard in bpo-17201(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=19274
(2) https://bugs.python.org/issue?@action=redirect&bpo=17201
File: python.info, Node: CPython Implementation Changes, Next: Deprecated<11>, Prev: Improved Modules<10>, Up: What’s New In Python 3 4
1.10.5 CPython Implementation Changes
-------------------------------------
* Menu:
* PEP 445; Customization of CPython Memory Allocators: PEP 445 Customization of CPython Memory Allocators.
* PEP 442; Safe Object Finalization: PEP 442 Safe Object Finalization.
* PEP 456; Secure and Interchangeable Hash Algorithm: PEP 456 Secure and Interchangeable Hash Algorithm.
* PEP 436; Argument Clinic: PEP 436 Argument Clinic.
* Other Build and C API Changes::
* Other Improvements: Other Improvements<2>.
* Significant Optimizations::
File: python.info, Node: PEP 445 Customization of CPython Memory Allocators, Next: PEP 442 Safe Object Finalization, Up: CPython Implementation Changes
1.10.5.1 PEP 445: Customization of CPython Memory Allocators
............................................................
PEP 445(1) adds new C level interfaces to customize memory allocation in
the CPython interpreter.
See also
........
PEP 445(2) – Add new APIs to customize Python memory allocators
PEP written and implemented by Victor Stinner.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0445/
(2) https://peps.python.org/pep-0445/
File: python.info, Node: PEP 442 Safe Object Finalization, Next: PEP 456 Secure and Interchangeable Hash Algorithm, Prev: PEP 445 Customization of CPython Memory Allocators, Up: CPython Implementation Changes
1.10.5.2 PEP 442: Safe Object Finalization
..........................................
PEP 442(1) removes the current limitations and quirks of object
finalization in CPython. With it, objects with ‘__del__()’ methods, as
well as generators with *note finally: 9ca. clauses, can be finalized
when they are part of a reference cycle.
As part of this change, module globals are no longer forcibly set to
*note None: 671. during interpreter shutdown in most cases, instead
relying on the normal operation of the cyclic garbage collector. This
avoids a whole class of interpreter-shutdown-time errors, usually
involving ‘__del__’ methods, that have plagued Python since the cyclic
GC was first introduced.
See also
........
PEP 442(2) – Safe object finalization
PEP written and implemented by Antoine Pitrou.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0442/
(2) https://peps.python.org/pep-0442/
File: python.info, Node: PEP 456 Secure and Interchangeable Hash Algorithm, Next: PEP 436 Argument Clinic, Prev: PEP 442 Safe Object Finalization, Up: CPython Implementation Changes
1.10.5.3 PEP 456: Secure and Interchangeable Hash Algorithm
...........................................................
PEP 456(1) follows up on earlier security fix work done on Python’s hash
algorithm to address certain DOS attacks to which public facing APIs
backed by dictionary lookups may be subject. (See bpo-14621(2) for the
start of the current round of improvements.) The PEP unifies CPython’s
hash code to make it easier for a packager to substitute a different
hash algorithm, and switches Python’s default implementation to a
SipHash implementation on platforms that have a 64 bit data type. Any
performance differences in comparison with the older FNV algorithm are
trivial.
The PEP adds additional fields to the *note sys.hash_info: ff8. named
tuple to describe the hash algorithm in use by the currently executing
binary. Otherwise, the PEP does not alter any existing CPython APIs.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0456/
(2) https://bugs.python.org/issue?@action=redirect&bpo=14621
File: python.info, Node: PEP 436 Argument Clinic, Next: Other Build and C API Changes, Prev: PEP 456 Secure and Interchangeable Hash Algorithm, Up: CPython Implementation Changes
1.10.5.4 PEP 436: Argument Clinic
.................................
“Argument Clinic” ( PEP 436(1)) is now part of the CPython build process
and can be used to simplify the process of defining and maintaining
accurate signatures for builtins and standard library extension modules
implemented in C.
Some standard library extension modules have been converted to use
Argument Clinic in Python 3.4, and *note pydoc: b5. and *note inspect:
7e. have been updated accordingly.
It is expected that signature metadata for programmatic introspection
will be added to additional callables implemented in C as part of Python
3.4 maintenance releases.
Note: The Argument Clinic PEP is not fully up to date with the
state of the implementation. This has been deemed acceptable by
the release manager and core development team in this case, as
Argument Clinic will not be made available as a public API for
third party use in Python 3.4.
See also
........
PEP 436(2) – The Argument Clinic DSL
PEP written and implemented by Larry Hastings.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0436/
(2) https://peps.python.org/pep-0436/
File: python.info, Node: Other Build and C API Changes, Next: Other Improvements<2>, Prev: PEP 436 Argument Clinic, Up: CPython Implementation Changes
1.10.5.5 Other Build and C API Changes
......................................
* The new *note PyType_GetSlot(): 89a. function has been added to the
stable ABI, allowing retrieval of function pointers from named type
slots when using the limited API. (Contributed by Martin von Löwis
in bpo-17162(1).)
* The new ‘Py_SetStandardStreamEncoding()’ pre-initialization API
allows applications embedding the CPython interpreter to reliably
force a particular encoding and error handler for the standard
streams. (Contributed by Bastien Montagne and Nick Coghlan in
bpo-16129(2).)
* Most Python C APIs that don’t mutate string arguments are now
correctly marked as accepting ‘const char *’ rather than ‘char *’.
(Contributed by Serhiy Storchaka in bpo-1772673(3).)
* A new shell version of ‘python-config’ can be used even when a
python interpreter is not available (for example, in cross
compilation scenarios).
* *note PyUnicode_FromFormat(): 385. now supports width and precision
specifications for ‘%s’, ‘%A’, ‘%U’, ‘%V’, ‘%S’, and ‘%R’.
(Contributed by Ysj Ray and Victor Stinner in bpo-7330(4).)
* New function *note PyStructSequence_InitType2(): ffb. supplements
the existing *note PyStructSequence_InitType(): ffc. function. The
difference is that it returns ‘0’ on success and ‘-1’ on failure.
* The CPython source can now be compiled using the address sanity
checking features of recent versions of GCC and clang: the false
alarms in the small object allocator have been silenced.
(Contributed by Dhiru Kholia in bpo-18596(5).)
* The Windows build now uses Address Space Layout Randomization(6)
and Data Execution Prevention(7). (Contributed by Christian Heimes
in bpo-16632(8).)
* New function *note PyObject_LengthHint(): ffd. is the C API
equivalent of *note operator.length_hint(): f68. (Contributed by
Armin Ronacher in bpo-16148(9).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17162
(2) https://bugs.python.org/issue?@action=redirect&bpo=16129
(3) https://bugs.python.org/issue?@action=redirect&bpo=1772673
(4) https://bugs.python.org/issue?@action=redirect&bpo=7330
(5) https://bugs.python.org/issue?@action=redirect&bpo=18596
(6) https://en.wikipedia.org/wiki/Address_space_layout_randomization
(7) https://en.wikipedia.org/wiki/Data_Execution_Prevention
(8) https://bugs.python.org/issue?@action=redirect&bpo=16632
(9) https://bugs.python.org/issue?@action=redirect&bpo=16148
File: python.info, Node: Other Improvements<2>, Next: Significant Optimizations, Prev: Other Build and C API Changes, Up: CPython Implementation Changes
1.10.5.6 Other Improvements
...........................
* The *note python: 1000. command has a new *note option: 1001, ‘-I’,
which causes it to run in “isolated mode”, which means that *note
sys.path: 3b0. contains neither the script’s directory nor the
user’s ‘site-packages’ directory, and all ‘PYTHON*’ environment
variables are ignored (it implies both ‘-s’ and ‘-E’). Other
restrictions may also be applied in the future, with the goal being
to isolate the execution of a script from the user’s environment.
This is appropriate, for example, when Python is used to run a
system script. On most POSIX systems it can and should be used in
the ‘#!’ line of system scripts. (Contributed by Christian Heimes
in bpo-16499(1).)
* Tab-completion is now enabled by default in the interactive
interpreter on systems that support *note readline: ba. History is
also enabled by default, and is written to (and read from) the file
‘~/.python-history’. (Contributed by Antoine Pitrou and Éric
Araujo in bpo-5845(2).)
* Invoking the Python interpreter with ‘--version’ now outputs the
version to standard output instead of standard error
(bpo-18338(3)). Similar changes were made to *note argparse: 6.
(bpo-18920(4)) and other modules that have script-like invocation
capabilities (bpo-18922(5)).
* The CPython Windows installer now adds ‘.py’ to the ‘PATHEXT’
variable when extensions are registered, allowing users to run a
python script at the windows command prompt by just typing its name
without the ‘.py’ extension. (Contributed by Paul Moore in
bpo-18569(6).)
* A new ‘make’ target coverage-report(7) will build python, run the
test suite, and generate an HTML coverage report for the C codebase
using ‘gcov’ and lcov(8).
* The ‘-R’ option to the *note python regression test suite: 1002.
now also checks for memory allocation leaks, using *note
sys.getallocatedblocks(): fc0. (Contributed by Antoine Pitrou in
bpo-13390(9).)
* ‘python -m’ now works with namespace packages.
* The *note stat: d1. module is now implemented in C, which means it
gets the values for its constants from the C header files, instead
of having the values hard-coded in the python module as was
previously the case.
* Loading multiple python modules from a single OS module (‘.so’,
‘.dll’) now works correctly (previously it silently returned the
first python module in the file). (Contributed by Václav Šmilauer
in bpo-16421(10).)
* A new opcode, ‘LOAD_CLASSDEREF’, has been added to fix a bug in the
loading of free variables in class bodies that could be triggered
by certain uses of *note __prepare__: c5e. (Contributed by
Benjamin Peterson in bpo-17853(11).)
* A number of MemoryError-related crashes were identified and fixed
by Victor Stinner using his PEP 445(12)-based ‘pyfailmalloc’ tool
(bpo-18408(13), bpo-18520(14)).
* The ‘pyvenv’ command now accepts a ‘--copies’ option to use copies
rather than symlinks even on systems where symlinks are the
default. (Contributed by Vinay Sajip in bpo-18807(15).)
* The ‘pyvenv’ command also accepts a ‘--without-pip’ option to
suppress the otherwise-automatic bootstrapping of pip into the
virtual environment. (Contributed by Nick Coghlan in bpo-19552(16)
as part of the PEP 453(17) implementation.)
* The encoding name is now optional in the value set for the *note
PYTHONIOENCODING: 1003. environment variable. This makes it
possible to set just the error handler, without changing the
default encoding. (Contributed by Serhiy Storchaka in
bpo-18818(18).)
* The *note bz2: 13, *note lzma: 8a, and *note gzip: 67. module
‘open’ functions now support ‘x’ (exclusive creation) mode.
(Contributed by Tim Heaney and Vajrasky Kok in bpo-19201(19),
bpo-19222(20), and bpo-19223(21).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16499
(2) https://bugs.python.org/issue?@action=redirect&bpo=5845
(3) https://bugs.python.org/issue?@action=redirect&bpo=18338
(4) https://bugs.python.org/issue?@action=redirect&bpo=18920
(5) https://bugs.python.org/issue?@action=redirect&bpo=18922
(6) https://bugs.python.org/issue?@action=redirect&bpo=18569
(7)
https://devguide.python.org/coverage/#measuring-coverage-of-c-code-with-gcov-and-lcov
(8) https://github.com/linux-test-project/lcov
(9) https://bugs.python.org/issue?@action=redirect&bpo=13390
(10) https://bugs.python.org/issue?@action=redirect&bpo=16421
(11) https://bugs.python.org/issue?@action=redirect&bpo=17853
(12) https://peps.python.org/pep-0445/
(13) https://bugs.python.org/issue?@action=redirect&bpo=18408
(14) https://bugs.python.org/issue?@action=redirect&bpo=18520
(15) https://bugs.python.org/issue?@action=redirect&bpo=18807
(16) https://bugs.python.org/issue?@action=redirect&bpo=19552
(17) https://peps.python.org/pep-0453/
(18) https://bugs.python.org/issue?@action=redirect&bpo=18818
(19) https://bugs.python.org/issue?@action=redirect&bpo=19201
(20) https://bugs.python.org/issue?@action=redirect&bpo=19222
(21) https://bugs.python.org/issue?@action=redirect&bpo=19223
File: python.info, Node: Significant Optimizations, Prev: Other Improvements<2>, Up: CPython Implementation Changes
1.10.5.7 Significant Optimizations
..................................
* The UTF-32 decoder is now 3x to 4x faster. (Contributed by Serhiy
Storchaka in bpo-14625(1).)
* The cost of hash collisions for sets is now reduced. Each hash
table probe now checks a series of consecutive, adjacent key/hash
pairs before continuing to make random probes through the hash
table. This exploits cache locality to make collision resolution
less expensive. The collision resolution scheme can be described
as a hybrid of linear probing and open addressing. The number of
additional linear probes defaults to nine. This can be changed at
compile-time by defining LINEAR_PROBES to be any value. Set
LINEAR_PROBES=0 to turn-off linear probing entirely. (Contributed
by Raymond Hettinger in bpo-18771(2).)
* The interpreter starts about 30% faster. A couple of measures lead
to the speedup. The interpreter loads fewer modules on startup,
e.g. the *note re: b9, *note collections: 1d. and *note locale:
86. modules and their dependencies are no longer imported by
default. The marshal module has been improved to load compiled
Python code faster. (Contributed by Antoine Pitrou, Christian
Heimes and Victor Stinner in bpo-19219(3), bpo-19218(4),
bpo-19209(5), bpo-19205(6) and bpo-9548(7).)
* *note bz2.BZ2File: 863. is now as fast or faster than the Python2
version for most cases. *note lzma.LZMAFile: 1005. has also been
optimized. (Contributed by Serhiy Storchaka and Nadeem Vawda in
bpo-16034(8).)
* *note random.getrandbits(): 1006. is 20%-40% faster for small
integers (the most common use case). (Contributed by Serhiy
Storchaka in bpo-16674(9).)
* By taking advantage of the new storage format for strings, pickling
of strings is now significantly faster. (Contributed by Victor
Stinner and Antoine Pitrou in bpo-15596(10).)
* A performance issue in ‘io.FileIO.readall()’ has been solved. This
particularly affects Windows, and significantly speeds up the case
of piping significant amounts of data through *note subprocess: d6.
(Contributed by Richard Oudkerk in bpo-15758(11).)
* *note html.escape(): 1007. is now 10x faster. (Contributed by Matt
Bryant in bpo-18020(12).)
* On Windows, the native ‘VirtualAlloc’ is now used instead of the
CRT ‘malloc’ in ‘obmalloc’. Artificial benchmarks show about a 3%
memory savings.
* *note os.urandom(): 51e. now uses a lazily opened persistent file
descriptor so as to avoid using many file descriptors when run in
parallel from multiple threads. (Contributed by Antoine Pitrou in
bpo-18756(13).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14625
(2) https://bugs.python.org/issue?@action=redirect&bpo=18771
(3) https://bugs.python.org/issue?@action=redirect&bpo=19219
(4) https://bugs.python.org/issue?@action=redirect&bpo=19218
(5) https://bugs.python.org/issue?@action=redirect&bpo=19209
(6) https://bugs.python.org/issue?@action=redirect&bpo=19205
(7) https://bugs.python.org/issue?@action=redirect&bpo=9548
(8) https://bugs.python.org/issue?@action=redirect&bpo=16034
(9) https://bugs.python.org/issue?@action=redirect&bpo=16674
(10) https://bugs.python.org/issue?@action=redirect&bpo=15596
(11) https://bugs.python.org/issue?@action=redirect&bpo=15758
(12) https://bugs.python.org/issue?@action=redirect&bpo=18020
(13) https://bugs.python.org/issue?@action=redirect&bpo=18756
File: python.info, Node: Deprecated<11>, Next: Removed<11>, Prev: CPython Implementation Changes, Up: What’s New In Python 3 4
1.10.6 Deprecated
-----------------
This section covers various APIs and other features that have been
deprecated in Python 3.4, and will be removed in Python 3.5 or later.
In most (but not all) cases, using the deprecated APIs will produce a
*note DeprecationWarning: 1a5. when the interpreter is run with
deprecation warnings enabled (for example, by using ‘-Wd’).
* Menu:
* Deprecations in the Python API::
* Deprecated Features::
File: python.info, Node: Deprecations in the Python API, Next: Deprecated Features, Up: Deprecated<11>
1.10.6.1 Deprecations in the Python API
.......................................
* As mentioned in *note PEP 451; A ModuleSpec Type for the Import
System: ed4, a number of *note importlib: 77. methods and functions
are deprecated: ‘importlib.find_loader()’ is replaced by *note
importlib.util.find_spec(): 2d0.;
‘importlib.machinery.PathFinder.find_module()’ is replaced by *note
importlib.machinery.PathFinder.find_spec(): 100a.;
‘importlib.abc.MetaPathFinder.find_module()’ is replaced by *note
importlib.abc.MetaPathFinder.find_spec(): 865.;
‘importlib.abc.PathEntryFinder.find_loader()’ and ‘find_module()’
are replaced by *note importlib.abc.PathEntryFinder.find_spec():
869.; all of the ‘XXXLoader’ ABC ‘load_module’ methods
(‘importlib.abc.Loader.load_module()’,
‘importlib.abc.InspectLoader.load_module()’,
‘importlib.abc.FileLoader.load_module()’,
‘importlib.abc.SourceLoader.load_module()’) should no longer be
implemented, instead loaders should implement an ‘exec_module’
method (*note importlib.abc.Loader.exec_module(): 867, *note
importlib.abc.InspectLoader.exec_module(): 100b. *note
importlib.abc.SourceLoader.exec_module(): 100c.) and let the import
system take care of the rest; and
‘importlib.abc.Loader.module_repr()’,
‘importlib.util.module_for_loader()’,
‘importlib.util.set_loader()’, and ‘importlib.util.set_package()’
are no longer needed because their functions are now handled
automatically by the import system.
* The ‘imp’ module is pending deprecation. To keep compatibility
with Python 2/3 code bases, the module’s removal is currently not
scheduled.
* The ‘formatter’ module is pending deprecation and is slated for
removal in Python 3.6.
* ‘MD5’ as the default 'digestmod' for the *note hmac.new(): a9f.
function is deprecated. Python 3.6 will require an explicit digest
name or constructor as 'digestmod' argument.
* The internal ‘Netrc’ class in the *note ftplib: 5e. module has been
documented as deprecated in its docstring for quite some time. It
now emits a *note DeprecationWarning: 1a5. and will be removed
completely in Python 3.5.
* The undocumented 'endtime' argument to *note
subprocess.Popen.wait(): ce4. should not have been exposed and is
hopefully not in use; it is deprecated and will mostly likely be
removed in Python 3.5.
* The 'strict' argument of *note HTMLParser: 874. is deprecated.
* The *note plistlib: ab. ‘readPlist()’, ‘writePlist()’,
‘readPlistFromBytes()’, and ‘writePlistToBytes()’ functions are
deprecated in favor of the corresponding new functions *note
load(): 94f, *note dump(): 951, *note loads(): 950, and *note
dumps(): 952. ‘Data()’ is deprecated in favor of just using the
*note bytes: 1c2. constructor.
* The *note sysconfig: db. key ‘SO’ is deprecated, it has been
replaced by ‘EXT_SUFFIX’.
* The ‘U’ mode accepted by various ‘open’ functions is deprecated.
In Python3 it does not do anything useful, and should be replaced
by appropriate uses of *note io.TextIOWrapper: 7b6. (if needed) and
its 'newline' argument.
* The 'parser' argument of *note xml.etree.ElementTree.iterparse():
275. has been deprecated, as has the 'html' argument of *note
XMLParser(): a53. To prepare for the removal of the latter, all
arguments to ‘XMLParser’ should be passed by keyword.
File: python.info, Node: Deprecated Features, Prev: Deprecations in the Python API, Up: Deprecated<11>
1.10.6.2 Deprecated Features
............................
* Running *note IDLE — Python editor and shell: 100e. with the ‘-n’
flag (no subprocess) is deprecated. However, the feature will not
be removed until bpo-18823(1) is resolved.
* The site module adding a “site-python” directory to sys.path, if it
exists, is deprecated (bpo-19375(2)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18823
(2) https://bugs.python.org/issue?@action=redirect&bpo=19375
File: python.info, Node: Removed<11>, Next: Porting to Python 3 4, Prev: Deprecated<11>, Up: What’s New In Python 3 4
1.10.7 Removed
--------------
* Menu:
* Operating Systems No Longer Supported::
* API and Feature Removals: API and Feature Removals<5>.
* Code Cleanups::
File: python.info, Node: Operating Systems No Longer Supported, Next: API and Feature Removals<5>, Up: Removed<11>
1.10.7.1 Operating Systems No Longer Supported
..............................................
Support for the following operating systems has been removed from the
source and build tools:
* OS/2 (bpo-16135(1)).
* Windows 2000 (changeset e52df05b496a).
* Windows systems where ‘COMSPEC’ points to ‘command.com’
(bpo-14470(2)).
* VMS (bpo-16136(3)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16135
(2) https://bugs.python.org/issue?@action=redirect&bpo=14470
(3) https://bugs.python.org/issue?@action=redirect&bpo=16136
File: python.info, Node: API and Feature Removals<5>, Next: Code Cleanups, Prev: Operating Systems No Longer Supported, Up: Removed<11>
1.10.7.2 API and Feature Removals
.................................
The following obsolete and previously deprecated APIs and features have
been removed:
* The unmaintained ‘Misc/TextMate’ and ‘Misc/vim’ directories have
been removed (see the devguide(1) for suggestions on what to use
instead).
* The ‘SO’ makefile macro is removed (it was replaced by the
‘SHLIB_SUFFIX’ and ‘EXT_SUFFIX’ macros) (bpo-16754(2)).
* The ‘PyThreadState.tick_counter’ field has been removed; its value
has been meaningless since Python 3.2, when the “new GIL” was
introduced (bpo-19199(3)).
* ‘PyLoader’ and ‘PyPycLoader’ have been removed from *note
importlib: 77. (Contributed by Taras Lyapun in bpo-15641(4).)
* The 'strict' argument to *note HTTPConnection: b40. and *note
HTTPSConnection: b41. has been removed. HTTP 0.9-style “Simple
Responses” are no longer supported.
* The deprecated *note urllib.request.Request: fd3. getter and setter
methods ‘add_data’, ‘has_data’, ‘get_data’, ‘get_type’, ‘get_host’,
‘get_selector’, ‘set_proxy’, ‘get_origin_req_host’, and
‘is_unverifiable’ have been removed (use direct attribute access
instead).
* Support for loading the deprecated ‘TYPE_INT64’ has been removed
from *note marshal: 8d. (Contributed by Dan Riti in bpo-15480(5).)
* *note inspect.Signature: 1cf.: positional-only parameters are now
required to have a valid name.
* *note object.__format__(): 61a. no longer accepts non-empty format
strings, it now raises a *note TypeError: 534. instead. Using a
non-empty string has been deprecated since Python 3.2. This change
has been made to prevent a situation where previously working (but
incorrect) code would start failing if an object gained a
__format__ method, which means that your code may now raise a *note
TypeError: 534. if you are using an ‘'s'’ format code with objects
that do not have a __format__ method that handles it. See
bpo-7994(6) for background.
* ‘difflib.SequenceMatcher.isbjunk()’ and
‘difflib.SequenceMatcher.isbpopular()’ were deprecated in 3.2, and
have now been removed: use ‘x in sm.bjunk’ and ‘x in sm.bpopular’,
where 'sm' is a *note SequenceMatcher: 1012. object (bpo-13248(7)).
---------- Footnotes ----------
(1) https://devguide.python.org
(2) https://bugs.python.org/issue?@action=redirect&bpo=16754
(3) https://bugs.python.org/issue?@action=redirect&bpo=19199
(4) https://bugs.python.org/issue?@action=redirect&bpo=15641
(5) https://bugs.python.org/issue?@action=redirect&bpo=15480
(6) https://bugs.python.org/issue?@action=redirect&bpo=7994
(7) https://bugs.python.org/issue?@action=redirect&bpo=13248
File: python.info, Node: Code Cleanups, Prev: API and Feature Removals<5>, Up: Removed<11>
1.10.7.3 Code Cleanups
......................
* The unused and undocumented internal ‘Scanner’ class has been
removed from the *note pydoc: b5. module.
* The private and effectively unused ‘_gestalt’ module has been
removed, along with the private *note platform: aa. functions
‘_mac_ver_lookup’, ‘_mac_ver_gstalt’, and ‘_bcd2str’, which would
only have ever been called on badly broken OSX systems (see
bpo-18393(1)).
* The hardcoded copies of certain *note stat: d1. constants that were
included in the *note tarfile: de. module namespace have been
removed.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18393
File: python.info, Node: Porting to Python 3 4, Next: Changed in 3 4 3, Prev: Removed<11>, Up: What’s New In Python 3 4
1.10.8 Porting to Python 3.4
----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Changes in ‘python’ Command Behavior: Changes in ‘python’ Command Behavior<2>.
* Changes in the Python API: Changes in the Python API<10>.
* Changes in the C API: Changes in the C API<8>.
File: python.info, Node: Changes in ‘python’ Command Behavior<2>, Next: Changes in the Python API<10>, Up: Porting to Python 3 4
1.10.8.1 Changes in ‘python’ Command Behavior
.............................................
* In a posix shell, setting the ‘PATH’ environment variable to an
empty value is equivalent to not setting it at all. However,
setting *note PYTHONPATH: 1016. to an empty value was 'not'
equivalent to not setting it at all: setting *note PYTHONPATH:
1016. to an empty value was equivalent to setting it to ‘.’, which
leads to confusion when reasoning by analogy to how ‘PATH’ works.
The behavior now conforms to the posix convention for ‘PATH’.
* The [X refs, Y blocks] output of a debug (‘--with-pydebug’) build
of the CPython interpreter is now off by default. It can be
re-enabled using the ‘-X showrefcount’ option. (Contributed by
Ezio Melotti in bpo-17323(1).)
* The python command and most stdlib scripts (as well as *note
argparse: 6.) now output ‘--version’ information to ‘stdout’
instead of ‘stderr’ (for issue list see *note Other Improvements:
fff. above).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17323
File: python.info, Node: Changes in the Python API<10>, Next: Changes in the C API<8>, Prev: Changes in ‘python’ Command Behavior<2>, Up: Porting to Python 3 4
1.10.8.2 Changes in the Python API
..................................
* The ABCs defined in *note importlib.abc: 78. now either raise the
appropriate exception or return a default value instead of raising
*note NotImplementedError: 22a. blindly. This will only affect
code calling *note super(): 4d7. and falling through all the way to
the ABCs. For compatibility, catch both *note NotImplementedError:
22a. or the appropriate exception as needed.
* The module type now initializes the *note __package__: 2db. and
*note __loader__: 2e6. attributes to ‘None’ by default. To
determine if these attributes were set in a backwards-compatible
fashion, use e.g. ‘getattr(module, '__loader__', None) is not
None’. (bpo-17115(1).)
* ‘importlib.util.module_for_loader()’ now sets ‘__loader__’ and
‘__package__’ unconditionally to properly support reloading. If
this is not desired then you will need to set these attributes
manually. You can use ‘importlib.util.module_to_load()’ for module
management.
* Import now resets relevant attributes (e.g. ‘__name__’,
‘__loader__’, ‘__package__’, ‘__file__’, ‘__cached__’)
unconditionally when reloading. Note that this restores a pre-3.3
behavior in that it means a module is re-found when re-loaded
(bpo-19413(2)).
* Frozen packages no longer set ‘__path__’ to a list containing the
package name, they now set it to an empty list. The previous
behavior could cause the import system to do the wrong thing on
submodule imports if there was also a directory with the same name
as the frozen package. The correct way to determine if a module is
a package or not is to use ‘hasattr(module, '__path__')’
(bpo-18065(3)).
* Frozen modules no longer define a ‘__file__’ attribute. It’s
semantically incorrect for frozen modules to set the attribute as
they are not loaded from any explicit location. If you must know
that a module comes from frozen code then you can see if the
module’s ‘__spec__.location’ is set to ‘'frozen'’, check if the
loader is a subclass of *note importlib.machinery.FrozenImporter:
1018, or if Python 2 compatibility is necessary you can use
‘imp.is_frozen()’.
* *note py_compile.compile(): a24. now raises *note FileExistsError:
1019. if the file path it would write to is a symlink or a
non-regular file. This is to act as a warning that import will
overwrite those files with a regular file regardless of what type
of file path they were originally.
* *note importlib.abc.SourceLoader.get_source(): 101a. no longer
raises *note ImportError: 415. when the source code being loaded
triggers a *note SyntaxError: 18d. or *note UnicodeDecodeError:
a12. As *note ImportError: 415. is meant to be raised only when
source code cannot be found but it should, it was felt to be
over-reaching/overloading of that meaning when the source code is
found but improperly structured. If you were catching ImportError
before and wish to continue to ignore syntax or decoding issues,
catch all three exceptions now.
* *note functools.update_wrapper(): 101b. and *note
functools.wraps(): f58. now correctly set the ‘__wrapped__’
attribute to the function being wrapped, even if that function also
had its ‘__wrapped__’ attribute set. This means ‘__wrapped__’
attributes now correctly link a stack of decorated functions rather
than every ‘__wrapped__’ attribute in the chain referring to the
innermost function. Introspection libraries that assumed the
previous behaviour was intentional can use *note inspect.unwrap():
f57. to access the first function in the chain that has no
‘__wrapped__’ attribute.
* *note inspect.getfullargspec(): 734. has been reimplemented on top
of *note inspect.signature(): 733. and hence handles a much wider
variety of callable objects than it did in the past. It is
expected that additional builtin and extension module callables
will gain signature metadata over the course of the Python 3.4
series. Code that assumes that *note inspect.getfullargspec():
734. will fail on non-Python callables may need to be adjusted
accordingly.
* *note importlib.machinery.PathFinder: 101c. now passes on the
current working directory to objects in *note sys.path_hooks: 101d.
for the empty string. This results in *note
sys.path_importer_cache: 5e0. never containing ‘''’, thus iterating
through *note sys.path_importer_cache: 5e0. based on *note
sys.path: 3b0. will not find all keys. A module’s ‘__file__’ when
imported in the current working directory will also now have an
absolute path, including when using ‘-m’ with the interpreter
(except for ‘__main__.__file__’ when a script has been executed
directly using a relative path) (Contributed by Brett Cannon in
bpo-18416(4)). is specified on the command-line) (bpo-18416(5)).
* The removal of the 'strict' argument to *note HTTPConnection: b40.
and *note HTTPSConnection: b41. changes the meaning of the
remaining arguments if you are specifying them positionally rather
than by keyword. If you’ve been paying attention to deprecation
warnings your code should already be specifying any additional
arguments via keywords.
* Strings between ‘from __future__ import ...’ statements now
'always' raise a *note SyntaxError: 18d. Previously if there was
no leading docstring, an interstitial string would sometimes be
ignored. This brings CPython into compliance with the language
spec; Jython and PyPy already were. (bpo-17434(6)).
* *note ssl.SSLSocket.getpeercert(): fb2. and *note
ssl.SSLSocket.do_handshake(): e68. now raise an *note OSError: 413.
with ‘ENOTCONN’ when the ‘SSLSocket’ is not connected, instead of
the previous behavior of raising an *note AttributeError: 348. In
addition, *note getpeercert(): fb2. will raise a *note ValueError:
204. if the handshake has not yet been done.
* *note base64.b32decode(): 101e. now raises a *note binascii.Error:
101f. when the input string contains non-b32-alphabet characters,
instead of a *note TypeError: 534. This particular *note
TypeError: 534. was missed when the other *note TypeError: 534.s
were converted. (Contributed by Serhiy Storchaka in bpo-18011(7).)
Note: this change was also inadvertently applied in Python 3.3.3.
* The ‘file’ attribute is now automatically closed when the creating
‘cgi.FieldStorage’ instance is garbage collected. If you were
pulling the file object out separately from the ‘cgi.FieldStorage’
instance and not keeping the instance alive, then you should either
store the entire ‘cgi.FieldStorage’ instance or read the contents
of the file before the ‘cgi.FieldStorage’ instance is garbage
collected.
* Calling ‘read’ or ‘write’ on a closed SSL socket now raises an
informative *note ValueError: 204. rather than the previous more
mysterious *note AttributeError: 348. (bpo-9177(8)).
* *note slice.indices(): 1020. no longer produces an *note
OverflowError: 87f. for huge values. As a consequence of this fix,
*note slice.indices(): 1020. now raises a *note ValueError: 204. if
given a negative length; previously it returned nonsense values
(bpo-14794(9)).
* The *note complex: 2f2. constructor, unlike the *note cmath: 18.
functions, was incorrectly accepting *note float: 2f1. values if an
object’s ‘__complex__’ special method returned one. This now
raises a *note TypeError: 534. (bpo-16290(10).)
* The *note int: 259. constructor in 3.2 and 3.3 erroneously accepts
*note float: 2f1. values for the 'base' parameter. It is unlikely
anyone was doing this, but if so, it will now raise a *note
TypeError: 534. (bpo-16772(11)).
* Defaults for keyword-only arguments are now evaluated 'after'
defaults for regular keyword arguments, instead of before.
Hopefully no one wrote any code that depends on the previous buggy
behavior (bpo-16967(12)).
* Stale thread states are now cleared after *note fork(): 197. This
may cause some system resources to be released that previously were
incorrectly kept perpetually alive (for example, database
connections kept in thread-local storage). (bpo-17094(13).)
* Parameter names in ‘__annotations__’ dicts are now mangled
properly, similarly to *note __kwdefaults__: 1021. (Contributed by
Yury Selivanov in bpo-20625(14).)
* *note hashlib.hash.name: f43. now always returns the identifier in
lower case. Previously some builtin hashes had uppercase names,
but now that it is a formal public interface the naming has been
made consistent (bpo-18532(15)).
* Because *note unittest.TestSuite: df3. now drops references to
tests after they are run, test harnesses that reuse a *note
TestSuite: df3. to re-run a set of tests may fail. Test suites
should not be re-used in this fashion since it means state is
retained between test runs, breaking the test isolation that *note
unittest: 106. is designed to provide. However, if the lack of
isolation is considered acceptable, the old behavior can be
restored by creating a *note TestSuite: df3. subclass that defines
a ‘_removeTestAtIndex’ method that does nothing (see *note
TestSuite.__iter__(): 1022.) (bpo-11798(16)).
* *note unittest: 106. now uses *note argparse: 6. for command line
parsing. There are certain invalid command forms that used to work
that are no longer allowed; in theory this should not cause
backward compatibility issues since the disallowed command forms
didn’t make any sense and are unlikely to be in use.
* The *note re.split(): 2a4, *note re.findall(): 1023, and *note
re.sub(): 2a5. functions, and the ‘group()’ and ‘groups()’ methods
of ‘match’ objects now always return a 'bytes' object when the
string to be matched is a *note bytes-like object: d2d. Previously
the return type matched the input type, so if your code was
depending on the return value being, say, a ‘bytearray’, you will
need to change your code.
* ‘audioop’ functions now raise an error immediately if passed string
input, instead of failing randomly later on (bpo-16685(17)).
* The new 'convert_charrefs' argument to *note HTMLParser: 874.
currently defaults to ‘False’ for backward compatibility, but will
eventually be changed to default to ‘True’. It is recommended that
you add this keyword, with the appropriate value, to any *note
HTMLParser: 874. calls in your code (bpo-13633(18)).
* Since the 'digestmod' argument to the *note hmac.new(): a9f.
function will in the future have no default, all calls to *note
hmac.new(): a9f. should be changed to explicitly specify a
'digestmod' (bpo-17276(19)).
* Calling *note sysconfig.get_config_var(): 1024. with the ‘SO’ key,
or looking ‘SO’ up in the results of a call to *note
sysconfig.get_config_vars(): 1025. is deprecated. This key should
be replaced by ‘EXT_SUFFIX’ or ‘SHLIB_SUFFIX’, depending on the
context (bpo-19555(20)).
* Any calls to ‘open’ functions that specify ‘U’ should be modified.
‘U’ is ineffective in Python3 and will eventually raise an error if
used. Depending on the function, the equivalent of its old Python2
behavior can be achieved using either a 'newline' argument, or if
necessary by wrapping the stream in *note TextIOWrapper: 7b6. to
use its 'newline' argument (bpo-15204(21)).
* If you use ‘pyvenv’ in a script and desire that pip 'not' be
installed, you must add ‘--without-pip’ to your command invocation.
* The default behavior of *note json.dump(): d44. and *note
json.dumps(): d45. when an indent is specified has changed: it no
longer produces trailing spaces after the item separating commas at
the ends of lines. This will matter only if you have tests that
are doing white-space-sensitive comparisons of such output
(bpo-16333(22)).
* *note doctest: 3a. now looks for doctests in extension module
‘__doc__’ strings, so if your doctest test discovery includes
extension modules that have things that look like doctests in them
you may see test failures you’ve never seen before when running
your tests (bpo-3158(23)).
* The *note collections.abc: 1e. module has been slightly refactored
as part of the Python startup improvements. As a consequence of
this, it is no longer the case that importing *note collections:
1d. automatically imports *note collections.abc: 1e. If your
program depended on the (undocumented) implicit import, you will
need to add an explicit ‘import collections.abc’ (bpo-20784(24)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=17115
(2) https://bugs.python.org/issue?@action=redirect&bpo=19413
(3) https://bugs.python.org/issue?@action=redirect&bpo=18065
(4) https://bugs.python.org/issue?@action=redirect&bpo=18416
(5) https://bugs.python.org/issue?@action=redirect&bpo=18416
(6) https://bugs.python.org/issue?@action=redirect&bpo=17434
(7) https://bugs.python.org/issue?@action=redirect&bpo=18011
(8) https://bugs.python.org/issue?@action=redirect&bpo=9177
(9) https://bugs.python.org/issue?@action=redirect&bpo=14794
(10) https://bugs.python.org/issue?@action=redirect&bpo=16290
(11) https://bugs.python.org/issue?@action=redirect&bpo=16772
(12) https://bugs.python.org/issue?@action=redirect&bpo=16967
(13) https://bugs.python.org/issue?@action=redirect&bpo=17094
(14) https://bugs.python.org/issue?@action=redirect&bpo=20625
(15) https://bugs.python.org/issue?@action=redirect&bpo=18532
(16) https://bugs.python.org/issue?@action=redirect&bpo=11798
(17) https://bugs.python.org/issue?@action=redirect&bpo=16685
(18) https://bugs.python.org/issue?@action=redirect&bpo=13633
(19) https://bugs.python.org/issue?@action=redirect&bpo=17276
(20) https://bugs.python.org/issue?@action=redirect&bpo=19555
(21) https://bugs.python.org/issue?@action=redirect&bpo=15204
(22) https://bugs.python.org/issue?@action=redirect&bpo=16333
(23) https://bugs.python.org/issue?@action=redirect&bpo=3158
(24) https://bugs.python.org/issue?@action=redirect&bpo=20784
File: python.info, Node: Changes in the C API<8>, Prev: Changes in the Python API<10>, Up: Porting to Python 3 4
1.10.8.3 Changes in the C API
.............................
* *note PyEval_EvalFrameEx(): 1027, *note PyObject_Repr(): 1028, and
*note PyObject_Str(): 1029, along with some other internal C APIs,
now include a debugging assertion that ensures they are not used in
situations where they may silently discard a currently active
exception. In cases where discarding the active exception is
expected and desired (for example, because it has already been
saved locally with *note PyErr_Fetch(): 3ef. or is being
deliberately replaced with a different exception), an explicit
*note PyErr_Clear(): 102a. call will be needed to avoid triggering
the assertion when invoking these operations (directly or
indirectly) and running against a version of Python that is
compiled with assertions enabled.
* *note PyErr_SetImportError(): d3e. now sets *note TypeError: 534.
when its 'msg' argument is not set. Previously only ‘NULL’ was
returned with no exception set.
* The result of the *note PyOS_ReadlineFunctionPointer: 583. callback
must now be a string allocated by *note PyMem_RawMalloc(): 38b. or
*note PyMem_RawRealloc(): 38d, or ‘NULL’ if an error occurred,
instead of a string allocated by *note PyMem_Malloc(): c66. or
*note PyMem_Realloc(): 102b. (bpo-16742(1))
* *note PyThread_set_key_value(): 404. now always set the value. In
Python 3.3, the function did nothing if the key already exists (if
the current value is a non-‘NULL’ pointer).
* The ‘f_tstate’ (thread state) field of the *note PyFrameObject:
784. structure has been removed to fix a bug: see bpo-14432(2) for
the rationale.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=16742
(2) https://bugs.python.org/issue?@action=redirect&bpo=14432
File: python.info, Node: Changed in 3 4 3, Prev: Porting to Python 3 4, Up: What’s New In Python 3 4
1.10.9 Changed in 3.4.3
-----------------------
* Menu:
* PEP 476; Enabling certificate verification by default for stdlib http clients: PEP 476 Enabling certificate verification by default for stdlib http clients.
File: python.info, Node: PEP 476 Enabling certificate verification by default for stdlib http clients, Up: Changed in 3 4 3
1.10.9.1 PEP 476: Enabling certificate verification by default for stdlib http clients
......................................................................................
*note http.client: 6f. and modules which use it, such as *note
urllib.request: 10b. and *note xmlrpc.client: 12e, will now verify that
the server presents a certificate which is signed by a CA in the
platform trust store and whose hostname matches the hostname being
requested by default, significantly improving security for many
applications.
For applications which require the old previous behavior, they can pass
an alternate context:
import urllib.request
import ssl
# This disables all verification
context = ssl._create_unverified_context()
# This allows using a specific certificate for the host, which doesn't need
# to be in the trust store
context = ssl.create_default_context(cafile="/path/to/file.crt")
urllib.request.urlopen("https://invalid-cert", context=context)
File: python.info, Node: What’s New In Python 3 3, Next: What’s New In Python 3 2, Prev: What’s New In Python 3 4, Up: What’s New in Python
1.11 What’s New In Python 3.3
=============================
This article explains the new features in Python 3.3, compared to 3.2.
Python 3.3 was released on September 29, 2012. For full details, see
the changelog(1).
See also
........
PEP 398(2) - Python 3.3 Release Schedule
* Menu:
* Summary – Release highlights: Summary – Release highlights<8>.
* PEP 405; Virtual Environments: PEP 405 Virtual Environments.
* PEP 420; Implicit Namespace Packages: PEP 420 Implicit Namespace Packages.
* PEP 3118; New memoryview implementation and buffer protocol documentation: PEP 3118 New memoryview implementation and buffer protocol documentation.
* PEP 393; Flexible String Representation: PEP 393 Flexible String Representation.
* PEP 397; Python Launcher for Windows: PEP 397 Python Launcher for Windows.
* PEP 3151; Reworking the OS and IO exception hierarchy: PEP 3151 Reworking the OS and IO exception hierarchy.
* PEP 380; Syntax for Delegating to a Subgenerator: PEP 380 Syntax for Delegating to a Subgenerator.
* PEP 409; Suppressing exception context: PEP 409 Suppressing exception context.
* PEP 414; Explicit Unicode literals: PEP 414 Explicit Unicode literals.
* PEP 3155; Qualified name for classes and functions: PEP 3155 Qualified name for classes and functions.
* PEP 412; Key-Sharing Dictionary: PEP 412 Key-Sharing Dictionary.
* PEP 362; Function Signature Object: PEP 362 Function Signature Object.
* PEP 421; Adding sys.implementation: PEP 421 Adding sys implementation.
* Using importlib as the Implementation of Import::
* Other Language Changes: Other Language Changes<11>.
* A Finer-Grained Import Lock::
* Builtin functions and types::
* New Modules: New Modules<11>.
* Improved Modules: Improved Modules<11>.
* Optimizations: Optimizations<10>.
* Build and C API Changes: Build and C API Changes<4>.
* Deprecated: Deprecated<12>.
* Porting to Python 3.3: Porting to Python 3 3.
---------- Footnotes ----------
(1) https://docs.python.org/3.3/whatsnew/changelog.html
(2) https://peps.python.org/pep-0398/
File: python.info, Node: Summary – Release highlights<8>, Next: PEP 405 Virtual Environments, Up: What’s New In Python 3 3
1.11.1 Summary – Release highlights
-----------------------------------
New syntax features:
* New ‘yield from’ expression for *note generator delegation: 1032.
* The ‘u'unicode'’ syntax is accepted again for *note str: 447.
objects.
New library modules:
* *note faulthandler: 58. (helps debugging low-level crashes)
* *note ipaddress: 80. (high-level objects representing IP addresses
and masks)
* *note lzma: 8a. (compress data using the XZ / LZMA algorithm)
* *note unittest.mock: 107. (replace parts of your system under test
with mock objects)
* *note venv: 111. (Python *note virtual environments: 1033, as in
the popular ‘virtualenv’ package)
New built-in features:
* Reworked *note I/O exception hierarchy: 1034.
Implementation improvements:
* Rewritten *note import machinery: 1035. based on *note importlib:
77.
* More compact *note unicode strings: 1036.
* More compact *note attribute dictionaries: 1037.
Significantly Improved Library Modules:
* C Accelerator for the *note decimal: 1038. module.
* Better unicode handling in the *note email: 1039. module (*note
provisional: 103a.).
Security improvements:
* Hash randomization is switched on by default.
Please read on for a comprehensive list of user-facing changes.
File: python.info, Node: PEP 405 Virtual Environments, Next: PEP 420 Implicit Namespace Packages, Prev: Summary – Release highlights<8>, Up: What’s New In Python 3 3
1.11.2 PEP 405: Virtual Environments
------------------------------------
Virtual environments help create separate Python setups while sharing a
system-wide base install, for ease of maintenance. Virtual environments
have their own set of private site packages (i.e. locally installed
libraries), and are optionally segregated from the system-wide site
packages. Their concept and implementation are inspired by the popular
‘virtualenv’ third-party package, but benefit from tighter integration
with the interpreter core.
This PEP adds the *note venv: 111. module for programmatic access, and
the ‘pyvenv’ script for command-line access and administration. The
Python interpreter checks for a ‘pyvenv.cfg’, file whose existence
signals the base of a virtual environment’s directory tree.
See also
........
PEP 405(1) - Python Virtual Environments
PEP written by Carl Meyer; implementation by Carl Meyer and Vinay
Sajip
---------- Footnotes ----------
(1) https://peps.python.org/pep-0405/
File: python.info, Node: PEP 420 Implicit Namespace Packages, Next: PEP 3118 New memoryview implementation and buffer protocol documentation, Prev: PEP 405 Virtual Environments, Up: What’s New In Python 3 3
1.11.3 PEP 420: Implicit Namespace Packages
-------------------------------------------
Native support for package directories that don’t require ‘__init__.py’
marker files and can automatically span multiple path segments (inspired
by various third party approaches to namespace packages, as described in
PEP 420(1))
See also
........
PEP 420(2) - Implicit Namespace Packages
PEP written by Eric V. Smith; implementation by Eric V. Smith and
Barry Warsaw
---------- Footnotes ----------
(1) https://peps.python.org/pep-0420/
(2) https://peps.python.org/pep-0420/
File: python.info, Node: PEP 3118 New memoryview implementation and buffer protocol documentation, Next: PEP 393 Flexible String Representation, Prev: PEP 420 Implicit Namespace Packages, Up: What’s New In Python 3 3
1.11.4 PEP 3118: New memoryview implementation and buffer protocol documentation
--------------------------------------------------------------------------------
The implementation of PEP 3118(1) has been significantly improved.
The new memoryview implementation comprehensively fixes all ownership
and lifetime issues of dynamically allocated fields in the Py_buffer
struct that led to multiple crash reports. Additionally, several
functions that crashed or returned incorrect results for non-contiguous
or multi-dimensional input have been fixed.
The memoryview object now has a PEP-3118 compliant getbufferproc() that
checks the consumer’s request type. Many new features have been added,
most of them work in full generality for non-contiguous arrays and
arrays with suboffsets.
The documentation has been updated, clearly spelling out
responsibilities for both exporters and consumers. Buffer request flags
are grouped into basic and compound flags. The memory layout of
non-contiguous and multi-dimensional NumPy-style arrays is explained.
* Menu:
* Features::
* API changes::
---------- Footnotes ----------
(1) https://peps.python.org/pep-3118/
File: python.info, Node: Features, Next: API changes, Up: PEP 3118 New memoryview implementation and buffer protocol documentation
1.11.4.1 Features
.................
* All native single character format specifiers in struct module
syntax (optionally prefixed with ‘@’) are now supported.
* With some restrictions, the cast() method allows changing of format
and shape of C-contiguous arrays.
* Multi-dimensional list representations are supported for any array
type.
* Multi-dimensional comparisons are supported for any array type.
* One-dimensional memoryviews of hashable (read-only) types with
formats B, b or c are now hashable. (Contributed by Antoine Pitrou
in bpo-13411(1).)
* Arbitrary slicing of any 1-D arrays type is supported. For
example, it is now possible to reverse a memoryview in 'O'(1) by
using a negative step.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13411
File: python.info, Node: API changes, Prev: Features, Up: PEP 3118 New memoryview implementation and buffer protocol documentation
1.11.4.2 API changes
....................
* The maximum number of dimensions is officially limited to 64.
* The representation of empty shape, strides and suboffsets is now an
empty tuple instead of ‘None’.
* Accessing a memoryview element with format ‘B’ (unsigned bytes) now
returns an integer (in accordance with the struct module syntax).
For returning a bytes object the view must be cast to ‘c’ first.
* memoryview comparisons now use the logical structure of the
operands and compare all array elements by value. All format
strings in struct module syntax are supported. Views with
unrecognised format strings are still permitted, but will always
compare as unequal, regardless of view contents.
* For further changes see *note Build and C API Changes: 1041. and
*note Porting C code: 1042.
(Contributed by Stefan Krah in bpo-10181(1).)
See also
........
PEP 3118(2) - Revising the Buffer Protocol
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10181
(2) https://peps.python.org/pep-3118/
File: python.info, Node: PEP 393 Flexible String Representation, Next: PEP 397 Python Launcher for Windows, Prev: PEP 3118 New memoryview implementation and buffer protocol documentation, Up: What’s New In Python 3 3
1.11.5 PEP 393: Flexible String Representation
----------------------------------------------
The Unicode string type is changed to support multiple internal
representations, depending on the character with the largest Unicode
ordinal (1, 2, or 4 bytes) in the represented string. This allows a
space-efficient representation in common cases, but gives access to full
UCS-4 on all systems. For compatibility with existing APIs, several
representations may exist in parallel; over time, this compatibility
should be phased out.
On the Python side, there should be no downside to this change.
On the C API side, PEP 393(1) is fully backward compatible. The legacy
API should remain available at least five years. Applications using the
legacy API will not fully benefit of the memory reduction, or - worse -
may use a bit more memory, because Python may have to maintain two
versions of each string (in the legacy format and in the new efficient
storage).
* Menu:
* Functionality::
* Performance and resource usage::
---------- Footnotes ----------
(1) https://peps.python.org/pep-0393/
File: python.info, Node: Functionality, Next: Performance and resource usage, Up: PEP 393 Flexible String Representation
1.11.5.1 Functionality
......................
Changes introduced by PEP 393(1) are the following:
* Python now always supports the full range of Unicode code points,
including non-BMP ones (i.e. from ‘U+0000’ to ‘U+10FFFF’). The
distinction between narrow and wide builds no longer exists and
Python now behaves like a wide build, even under Windows.
* With the death of narrow builds, the problems specific to narrow
builds have also been fixed, for example:
* *note len(): 62a. now always returns 1 for non-BMP characters,
so ‘len('\U0010FFFF') == 1’;
* surrogate pairs are not recombined in string literals, so
‘'\uDBFF\uDFFF' != '\U0010FFFF'’;
* indexing or slicing non-BMP characters returns the expected
value, so ‘'\U0010FFFF'[0]’ now returns ‘'\U0010FFFF'’ and not
‘'\uDBFF'’;
* all other functions in the standard library now correctly
handle non-BMP code points.
* The value of *note sys.maxunicode: 1045. is now always ‘1114111’
(‘0x10FFFF’ in hexadecimal). The ‘PyUnicode_GetMax()’ function
still returns either ‘0xFFFF’ or ‘0x10FFFF’ for backward
compatibility, and it should not be used with the new Unicode API
(see bpo-13054(2)).
* The ‘./configure’ flag ‘--with-wide-unicode’ has been removed.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0393/
(2) https://bugs.python.org/issue?@action=redirect&bpo=13054
File: python.info, Node: Performance and resource usage, Prev: Functionality, Up: PEP 393 Flexible String Representation
1.11.5.2 Performance and resource usage
.......................................
The storage of Unicode strings now depends on the highest code point in
the string:
* pure ASCII and Latin1 strings (‘U+0000-U+00FF’) use 1 byte per code
point;
* BMP strings (‘U+0000-U+FFFF’) use 2 bytes per code point;
* non-BMP strings (‘U+10000-U+10FFFF’) use 4 bytes per code point.
The net effect is that for most applications, memory usage of string
storage should decrease significantly - especially compared to former
wide unicode builds - as, in many cases, strings will be pure ASCII even
in international contexts (because many strings store non-human language
data, such as XML fragments, HTTP headers, JSON-encoded data, etc.). We
also hope that it will, for the same reasons, increase CPU cache
efficiency on non-trivial applications. The memory usage of Python 3.3
is two to three times smaller than Python 3.2, and a little bit better
than Python 2.7, on a Django benchmark (see the PEP for details).
See also
........
PEP 393(1) - Flexible String Representation
PEP written by Martin von Löwis; implementation by Torsten Becker
and Martin von Löwis.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0393/
File: python.info, Node: PEP 397 Python Launcher for Windows, Next: PEP 3151 Reworking the OS and IO exception hierarchy, Prev: PEP 393 Flexible String Representation, Up: What’s New In Python 3 3
1.11.6 PEP 397: Python Launcher for Windows
-------------------------------------------
The Python 3.3 Windows installer now includes a ‘py’ launcher
application that can be used to launch Python applications in a version
independent fashion.
This launcher is invoked implicitly when double-clicking ‘*.py’ files.
If only a single Python version is installed on the system, that version
will be used to run the file. If multiple versions are installed, the
most recent version is used by default, but this can be overridden by
including a Unix-style “shebang line” in the Python script.
The launcher can also be used explicitly from the command line as the
‘py’ application. Running ‘py’ follows the same version selection rules
as implicitly launching scripts, but a more specific version can be
selected by passing appropriate arguments (such as ‘-3’ to request
Python 3 when Python 2 is also installed, or ‘-2.6’ to specifically
request an earlier Python version when a more recent version is
installed).
In addition to the launcher, the Windows installer now includes an
option to add the newly installed Python to the system PATH.
(Contributed by Brian Curtin in bpo-3561(1).)
See also
........
PEP 397(2) - Python Launcher for Windows
PEP written by Mark Hammond and Martin v. Löwis; implementation by
Vinay Sajip.
Launcher documentation: *note Python Launcher for Windows: 5ba.
Installer PATH modification: *note Finding the Python executable: 1049.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=3561
(2) https://peps.python.org/pep-0397/
File: python.info, Node: PEP 3151 Reworking the OS and IO exception hierarchy, Next: PEP 380 Syntax for Delegating to a Subgenerator, Prev: PEP 397 Python Launcher for Windows, Up: What’s New In Python 3 3
1.11.7 PEP 3151: Reworking the OS and IO exception hierarchy
------------------------------------------------------------
The hierarchy of exceptions raised by operating system errors is now
both simplified and finer-grained.
You don’t have to worry anymore about choosing the appropriate exception
type between *note OSError: 413, *note IOError: 104b, *note
EnvironmentError: 104c, *note WindowsError: 104d, ‘mmap.error’, *note
socket.error: 104e. or *note select.error: 104f. All these exception
types are now only one: *note OSError: 413. The other names are kept as
aliases for compatibility reasons.
Also, it is now easier to catch a specific error condition. Instead of
inspecting the ‘errno’ attribute (or ‘args[0]’) for a particular
constant from the *note errno: 57. module, you can catch the adequate
*note OSError: 413. subclass. The available subclasses are the
following:
* *note BlockingIOError: 1050.
* *note ChildProcessError: 1051.
* *note ConnectionError: e09.
* *note FileExistsError: 1019.
* *note FileNotFoundError: 427.
* *note InterruptedError: d87.
* *note IsADirectoryError: 1052.
* *note NotADirectoryError: 1053.
* *note PermissionError: d42.
* *note ProcessLookupError: 1054.
* *note TimeoutError: 831.
And the *note ConnectionError: e09. itself has finer-grained subclasses:
* *note BrokenPipeError: 1055.
* *note ConnectionAbortedError: 1056.
* *note ConnectionRefusedError: 1057.
* *note ConnectionResetError: 1058.
Thanks to the new exceptions, common usages of the *note errno: 57. can
now be avoided. For example, the following code written for Python 3.2:
from errno import ENOENT, EACCES, EPERM
try:
with open("document.txt") as f:
content = f.read()
except IOError as err:
if err.errno == ENOENT:
print("document.txt file is missing")
elif err.errno in (EACCES, EPERM):
print("You are not allowed to read document.txt")
else:
raise
can now be written without the *note errno: 57. import and without
manual inspection of exception attributes:
try:
with open("document.txt") as f:
content = f.read()
except FileNotFoundError:
print("document.txt file is missing")
except PermissionError:
print("You are not allowed to read document.txt")
See also
........
PEP 3151(1) - Reworking the OS and IO Exception Hierarchy
PEP written and implemented by Antoine Pitrou
---------- Footnotes ----------
(1) https://peps.python.org/pep-3151/
File: python.info, Node: PEP 380 Syntax for Delegating to a Subgenerator, Next: PEP 409 Suppressing exception context, Prev: PEP 3151 Reworking the OS and IO exception hierarchy, Up: What’s New In Python 3 3
1.11.8 PEP 380: Syntax for Delegating to a Subgenerator
-------------------------------------------------------
PEP 380 adds the ‘yield from’ expression, allowing a *note generator:
105a. to delegate part of its operations to another generator. This
allows a section of code containing *note yield: 9cd. to be factored out
and placed in another generator. Additionally, the subgenerator is
allowed to return with a value, and the value is made available to the
delegating generator.
While designed primarily for use in delegating to a subgenerator, the
‘yield from’ expression actually allows delegation to arbitrary
subiterators.
For simple iterators, ‘yield from iterable’ is essentially just a
shortened form of ‘for item in iterable: yield item’:
>>> def g(x):
... yield from range(x, 0, -1)
... yield from range(x)
...
>>> list(g(5))
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
However, unlike an ordinary loop, ‘yield from’ allows subgenerators to
receive sent and thrown values directly from the calling scope, and
return a final value to the outer generator:
>>> def accumulate():
... tally = 0
... while 1:
... next = yield
... if next is None:
... return tally
... tally += next
...
>>> def gather_tallies(tallies):
... while 1:
... tally = yield from accumulate()
... tallies.append(tally)
...
>>> tallies = []
>>> acc = gather_tallies(tallies)
>>> next(acc) # Ensure the accumulator is ready to accept values
>>> for i in range(4):
... acc.send(i)
...
>>> acc.send(None) # Finish the first tally
>>> for i in range(5):
... acc.send(i)
...
>>> acc.send(None) # Finish the second tally
>>> tallies
[6, 10]
The main principle driving this change is to allow even generators that
are designed to be used with the ‘send’ and ‘throw’ methods to be split
into multiple subgenerators as easily as a single large function can be
split into multiple subfunctions.
See also
........
PEP 380(1) - Syntax for Delegating to a Subgenerator
PEP written by Greg Ewing; implementation by Greg Ewing, integrated
into 3.3 by Renaud Blanch, Ryan Kelly and Nick Coghlan;
documentation by Zbigniew Jędrzejewski-Szmek and Nick Coghlan
---------- Footnotes ----------
(1) https://peps.python.org/pep-0380/
File: python.info, Node: PEP 409 Suppressing exception context, Next: PEP 414 Explicit Unicode literals, Prev: PEP 380 Syntax for Delegating to a Subgenerator, Up: What’s New In Python 3 3
1.11.9 PEP 409: Suppressing exception context
---------------------------------------------
PEP 409 introduces new syntax that allows the display of the chained
exception context to be disabled. This allows cleaner error messages in
applications that convert between exception types:
>>> class D:
... def __init__(self, extra):
... self._extra_attributes = extra
... def __getattr__(self, attr):
... try:
... return self._extra_attributes[attr]
... except KeyError:
... raise AttributeError(attr) from None
...
>>> D({}).x
Traceback (most recent call last):
File "", line 1, in
File "", line 8, in __getattr__
AttributeError: x
Without the ‘from None’ suffix to suppress the cause, the original
exception would be displayed by default:
>>> class C:
... def __init__(self, extra):
... self._extra_attributes = extra
... def __getattr__(self, attr):
... try:
... return self._extra_attributes[attr]
... except KeyError:
... raise AttributeError(attr)
...
>>> C({}).x
Traceback (most recent call last):
File "", line 6, in __getattr__
KeyError: 'x'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "", line 8, in __getattr__
AttributeError: x
No debugging capability is lost, as the original exception context
remains available if needed (for example, if an intervening library has
incorrectly suppressed valuable underlying details):
>>> try:
... D({}).x
... except AttributeError as exc:
... print(repr(exc.__context__))
...
KeyError('x',)
See also
........
PEP 409(1) - Suppressing exception context
PEP written by Ethan Furman; implemented by Ethan Furman and Nick
Coghlan.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0409/
File: python.info, Node: PEP 414 Explicit Unicode literals, Next: PEP 3155 Qualified name for classes and functions, Prev: PEP 409 Suppressing exception context, Up: What’s New In Python 3 3
1.11.10 PEP 414: Explicit Unicode literals
------------------------------------------
To ease the transition from Python 2 for Unicode aware Python
applications that make heavy use of Unicode literals, Python 3.3 once
again supports the “‘u’” prefix for string literals. This prefix has no
semantic significance in Python 3, it is provided solely to reduce the
number of purely mechanical changes in migrating to Python 3, making it
easier for developers to focus on the more significant semantic changes
(such as the stricter default separation of binary and text data).
See also
........
PEP 414(1) - Explicit Unicode literals
PEP written by Armin Ronacher.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0414/
File: python.info, Node: PEP 3155 Qualified name for classes and functions, Next: PEP 412 Key-Sharing Dictionary, Prev: PEP 414 Explicit Unicode literals, Up: What’s New In Python 3 3
1.11.11 PEP 3155: Qualified name for classes and functions
----------------------------------------------------------
Functions and class objects have a new *note __qualname__: 105e.
attribute representing the “path” from the module top-level to their
definition. For global functions and classes, this is the same as *note
__name__: 105f. For other functions and classes, it provides better
information about where they were actually defined, and how they might
be accessible from the global scope.
Example with (non-bound) methods:
>>> class C:
... def meth(self):
... pass
...
>>> C.meth.__name__
'meth'
>>> C.meth.__qualname__
'C.meth'
Example with nested classes:
>>> class C:
... class D:
... def meth(self):
... pass
...
>>> C.D.__name__
'D'
>>> C.D.__qualname__
'C.D'
>>> C.D.meth.__name__
'meth'
>>> C.D.meth.__qualname__
'C.D.meth'
Example with nested functions:
>>> def outer():
... def inner():
... pass
... return inner
...
>>> outer().__name__
'inner'
>>> outer().__qualname__
'outer..inner'
The string representation of those objects is also changed to include
the new, more precise information:
>>> str(C.D)
""
>>> str(C.D.meth)
''
See also
........
PEP 3155(1) - Qualified name for classes and functions
PEP written and implemented by Antoine Pitrou.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3155/
File: python.info, Node: PEP 412 Key-Sharing Dictionary, Next: PEP 362 Function Signature Object, Prev: PEP 3155 Qualified name for classes and functions, Up: What’s New In Python 3 3
1.11.12 PEP 412: Key-Sharing Dictionary
---------------------------------------
Dictionaries used for the storage of objects’ attributes are now able to
share part of their internal storage between each other (namely, the
part which stores the keys and their respective hashes). This reduces
the memory consumption of programs creating many instances of
non-builtin types.
See also
........
PEP 412(1) - Key-Sharing Dictionary
PEP written and implemented by Mark Shannon.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0412/
File: python.info, Node: PEP 362 Function Signature Object, Next: PEP 421 Adding sys implementation, Prev: PEP 412 Key-Sharing Dictionary, Up: What’s New In Python 3 3
1.11.13 PEP 362: Function Signature Object
------------------------------------------
A new function *note inspect.signature(): 733. makes introspection of
python callables easy and straightforward. A broad range of callables
is supported: python functions, decorated or not, classes, and *note
functools.partial(): 410. objects. New classes *note inspect.Signature:
1cf, *note inspect.Parameter: 1d0. and *note inspect.BoundArguments:
1062. hold information about the call signatures, such as, annotations,
default values, parameters kinds, and bound arguments, which
considerably simplifies writing decorators and any code that validates
or amends calling signatures or arguments.
See also
........
PEP 362(1): - Function Signature Object
PEP written by Brett Cannon, Yury Selivanov, Larry Hastings, Jiwon
Seo; implemented by Yury Selivanov.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0362/
File: python.info, Node: PEP 421 Adding sys implementation, Next: Using importlib as the Implementation of Import, Prev: PEP 362 Function Signature Object, Up: What’s New In Python 3 3
1.11.14 PEP 421: Adding sys.implementation
------------------------------------------
A new attribute on the *note sys: d9. module exposes details specific to
the implementation of the currently running interpreter. The initial
set of attributes on *note sys.implementation: 512. are ‘name’,
‘version’, ‘hexversion’, and ‘cache_tag’.
The intention of ‘sys.implementation’ is to consolidate into one
namespace the implementation-specific data used by the standard library.
This allows different Python implementations to share a single standard
library code base much more easily. In its initial state,
‘sys.implementation’ holds only a small portion of the
implementation-specific data. Over time that ratio will shift in order
to make the standard library more portable.
One example of improved standard library portability is ‘cache_tag’. As
of Python 3.3, ‘sys.implementation.cache_tag’ is used by *note
importlib: 77. to support PEP 3147(1) compliance. Any Python
implementation that uses ‘importlib’ for its built-in import system may
use ‘cache_tag’ to control the caching behavior for modules.
* Menu:
* SimpleNamespace::
---------- Footnotes ----------
(1) https://peps.python.org/pep-3147/
File: python.info, Node: SimpleNamespace, Up: PEP 421 Adding sys implementation
1.11.14.1 SimpleNamespace
.........................
The implementation of ‘sys.implementation’ also introduces a new type to
Python: *note types.SimpleNamespace: 1d1. In contrast to a
mapping-based namespace, like *note dict: 258, ‘SimpleNamespace’ is
attribute-based, like *note object: a8c. However, unlike ‘object’,
‘SimpleNamespace’ instances are writable. This means that you can add,
remove, and modify the namespace through normal attribute access.
See also
........
PEP 421(1) - Adding sys.implementation
PEP written and implemented by Eric Snow.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0421/
File: python.info, Node: Using importlib as the Implementation of Import, Next: Other Language Changes<11>, Prev: PEP 421 Adding sys implementation, Up: What’s New In Python 3 3
1.11.15 Using importlib as the Implementation of Import
-------------------------------------------------------
bpo-2377(1) - Replace __import__ w/ importlib.__import__ bpo-13959(2) -
Re-implement parts of ‘imp’ in pure Python bpo-14605(3) - Make import
machinery explicit bpo-14646(4) - Require loaders set __loader__ and
__package__
The *note __import__(): 8d3. function is now powered by *note
importlib.__import__(): 1066. This work leads to the completion of
“phase 2” of PEP 302(5). There are multiple benefits to this change.
First, it has allowed for more of the machinery powering import to be
exposed instead of being implicit and hidden within the C code. It also
provides a single implementation for all Python VMs supporting Python
3.3 to use, helping to end any VM-specific deviations in import
semantics. And finally it eases the maintenance of import, allowing for
future growth to occur.
For the common user, there should be no visible change in semantics.
For those whose code currently manipulates import or calls import
programmatically, the code changes that might possibly be required are
covered in the *note Porting Python code: 1067. section of this
document.
* Menu:
* New APIs::
* Visible Changes::
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2377
(2) https://bugs.python.org/issue?@action=redirect&bpo=13959
(3) https://bugs.python.org/issue?@action=redirect&bpo=14605
(4) https://bugs.python.org/issue?@action=redirect&bpo=14646
(5) https://peps.python.org/pep-0302/
File: python.info, Node: New APIs, Next: Visible Changes, Up: Using importlib as the Implementation of Import
1.11.15.1 New APIs
..................
One of the large benefits of this work is the exposure of what goes into
making the import statement work. That means the various importers that
were once implicit are now fully exposed as part of the *note importlib:
77. package.
The abstract base classes defined in *note importlib.abc: 78. have been
expanded to properly delineate between *note meta path finders: 1069.
and *note path entry finders: 106a. by introducing *note
importlib.abc.MetaPathFinder: 86b. and *note
importlib.abc.PathEntryFinder: 86c, respectively. The old ABC of
‘importlib.abc.Finder’ is now only provided for backwards-compatibility
and does not enforce any method requirements.
In terms of finders, *note importlib.machinery.FileFinder: 106b. exposes
the mechanism used to search for source and bytecode files of a module.
Previously this class was an implicit member of *note sys.path_hooks:
101d.
For loaders, the new abstract base class *note importlib.abc.FileLoader:
106c. helps write a loader that uses the file system as the storage
mechanism for a module’s code. The loader for source files (*note
importlib.machinery.SourceFileLoader: 106d.), sourceless bytecode files
(*note importlib.machinery.SourcelessFileLoader: 106e.), and extension
modules (*note importlib.machinery.ExtensionFileLoader: cb0.) are now
available for direct use.
*note ImportError: 415. now has ‘name’ and ‘path’ attributes which are
set when there is relevant data to provide. The message for failed
imports will also provide the full name of the module now instead of
just the tail end of the module’s name.
The *note importlib.invalidate_caches(): c10. function will now call the
method with the same name on all finders cached in *note
sys.path_importer_cache: 5e0. to help clean up any stored state as
necessary.
File: python.info, Node: Visible Changes, Prev: New APIs, Up: Using importlib as the Implementation of Import
1.11.15.2 Visible Changes
.........................
For potential required changes to code, see the *note Porting Python
code: 1067. section.
Beyond the expanse of what *note importlib: 77. now exposes, there are
other visible changes to import. The biggest is that *note
sys.meta_path: d2b. and *note sys.path_hooks: 101d. now store all of the
meta path finders and path entry hooks used by import. Previously the
finders were implicit and hidden within the C code of import instead of
being directly exposed. This means that one can now easily remove or
change the order of the various finders to fit one’s needs.
Another change is that all modules have a ‘__loader__’ attribute,
storing the loader used to create the module. PEP 302(1) has been
updated to make this attribute mandatory for loaders to implement, so in
the future once 3rd-party loaders have been updated people will be able
to rely on the existence of the attribute. Until such time, though,
import is setting the module post-load.
Loaders are also now expected to set the ‘__package__’ attribute from
PEP 366(2). Once again, import itself is already setting this on all
loaders from *note importlib: 77. and import itself is setting the
attribute post-load.
‘None’ is now inserted into *note sys.path_importer_cache: 5e0. when no
finder can be found on *note sys.path_hooks: 101d. Since
‘imp.NullImporter’ is not directly exposed on *note sys.path_hooks:
101d. it could no longer be relied upon to always be available to use as
a value representing no finder found.
All other changes relate to semantic changes which should be taken into
consideration when updating code for Python 3.3, and thus should be read
about in the *note Porting Python code: 1067. section of this document.
(Implementation by Brett Cannon)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0302/
(2) https://peps.python.org/pep-0366/
File: python.info, Node: Other Language Changes<11>, Next: A Finer-Grained Import Lock, Prev: Using importlib as the Implementation of Import, Up: What’s New In Python 3 3
1.11.16 Other Language Changes
------------------------------
Some smaller changes made to the core Python language are:
* Added support for Unicode name aliases and named sequences. Both
*note unicodedata.lookup(): 1071. and ‘'\N{...}'’ now resolve name
aliases, and *note unicodedata.lookup(): 1071. resolves named
sequences too.
(Contributed by Ezio Melotti in bpo-12753(1).)
* Unicode database updated to UCD version 6.1.0
* Equality comparisons on *note range(): 943. objects now return a
result reflecting the equality of the underlying sequences
generated by those range objects. (bpo-13201(2))
* The ‘count()’, ‘find()’, ‘rfind()’, ‘index()’ and ‘rindex()’
methods of *note bytes: 1c2. and *note bytearray: 53a. objects now
accept an integer between 0 and 255 as their first argument.
(Contributed by Petri Lehtinen in bpo-12170(3).)
* The ‘rjust()’, ‘ljust()’, and ‘center()’ methods of *note bytes:
1c2. and *note bytearray: 53a. now accept a *note bytearray: 53a.
for the ‘fill’ argument. (Contributed by Petri Lehtinen in
bpo-12380(4).)
* New methods have been added to *note list: 60d. and *note
bytearray: 53a.: ‘copy()’ and ‘clear()’ (bpo-10516(5)).
Consequently, *note MutableSequence: 1a1. now also defines a
‘clear()’ method (bpo-11388(6)).
* Raw bytes literals can now be written ‘rb"..."’ as well as
‘br"..."’.
(Contributed by Antoine Pitrou in bpo-13748(7).)
* *note dict.setdefault(): 1072. now does only one lookup for the
given key, making it atomic when used with built-in types.
(Contributed by Filip Gruszczyński in bpo-13521(8).)
* The error messages produced when a function call does not match the
function signature have been significantly improved.
(Contributed by Benjamin Peterson.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12753
(2) https://bugs.python.org/issue?@action=redirect&bpo=13201
(3) https://bugs.python.org/issue?@action=redirect&bpo=12170
(4) https://bugs.python.org/issue?@action=redirect&bpo=12380
(5) https://bugs.python.org/issue?@action=redirect&bpo=10516
(6) https://bugs.python.org/issue?@action=redirect&bpo=11388
(7) https://bugs.python.org/issue?@action=redirect&bpo=13748
(8) https://bugs.python.org/issue?@action=redirect&bpo=13521
File: python.info, Node: A Finer-Grained Import Lock, Next: Builtin functions and types, Prev: Other Language Changes<11>, Up: What’s New In Python 3 3
1.11.17 A Finer-Grained Import Lock
-----------------------------------
Previous versions of CPython have always relied on a global import lock.
This led to unexpected annoyances, such as deadlocks when importing a
module would trigger code execution in a different thread as a
side-effect. Clumsy workarounds were sometimes employed, such as the
*note PyImport_ImportModuleNoBlock(): 3b9. C API function.
In Python 3.3, importing a module takes a per-module lock. This
correctly serializes importation of a given module from multiple threads
(preventing the exposure of incompletely initialized modules), while
eliminating the aforementioned annoyances.
(Contributed by Antoine Pitrou in bpo-9260(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9260
File: python.info, Node: Builtin functions and types, Next: New Modules<11>, Prev: A Finer-Grained Import Lock, Up: What’s New In Python 3 3
1.11.18 Builtin functions and types
-----------------------------------
* *note open(): 517. gets a new 'opener' parameter: the underlying
file descriptor for the file object is then obtained by calling
'opener' with ('file', 'flags'). It can be used to use custom
flags like *note os.O_CLOEXEC: 1075. for example. The ‘'x'’ mode
was added: open for exclusive creation, failing if the file already
exists.
* *note print(): f70.: added the 'flush' keyword argument. If the
'flush' keyword argument is true, the stream is forcibly flushed.
* *note hash(): 5e7.: hash randomization is enabled by default, see
*note object.__hash__(): afb. and *note PYTHONHASHSEED: 1076.
* The *note str: 447. type gets a new *note casefold(): dd5. method:
return a casefolded copy of the string, casefolded strings may be
used for caseless matching. For example, ‘'ß'.casefold()’ returns
‘'ss'’.
* The sequence documentation has been substantially rewritten to
better explain the binary/text sequence distinction and to provide
specific documentation sections for the individual builtin sequence
types (bpo-4966(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4966
File: python.info, Node: New Modules<11>, Next: Improved Modules<11>, Prev: Builtin functions and types, Up: What’s New In Python 3 3
1.11.19 New Modules
-------------------
* Menu:
* faulthandler: faulthandler<4>.
* ipaddress: ipaddress<6>.
* lzma: lzma<2>.
File: python.info, Node: faulthandler<4>, Next: ipaddress<6>, Up: New Modules<11>
1.11.19.1 faulthandler
......................
This new debug module *note faulthandler: 58. contains functions to dump
Python tracebacks explicitly, on a fault (a crash like a segmentation
fault), after a timeout, or on a user signal. Call *note
faulthandler.enable(): c9e. to install fault handlers for the ‘SIGSEGV’,
‘SIGFPE’, ‘SIGABRT’, ‘SIGBUS’, and ‘SIGILL’ signals. You can also
enable them at startup by setting the *note PYTHONFAULTHANDLER: 1079.
environment variable or by using *note -X: 176. ‘faulthandler’ command
line option.
Example of a segmentation fault on Linux:
$ python -q -X faulthandler
>>> import ctypes
>>> ctypes.string_at(0)
Fatal Python error: Segmentation fault
Current thread 0x00007fb899f39700:
File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at
File "", line 1 in
Segmentation fault
File: python.info, Node: ipaddress<6>, Next: lzma<2>, Prev: faulthandler<4>, Up: New Modules<11>
1.11.19.2 ipaddress
...................
The new *note ipaddress: 80. module provides tools for creating and
manipulating objects representing IPv4 and IPv6 addresses, networks and
interfaces (i.e. an IP address associated with a specific IP subnet).
(Contributed by Google and Peter Moody in PEP 3144(1).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-3144/
File: python.info, Node: lzma<2>, Prev: ipaddress<6>, Up: New Modules<11>
1.11.19.3 lzma
..............
The newly added *note lzma: 8a. module provides data compression and
decompression using the LZMA algorithm, including support for the ‘.xz’
and ‘.lzma’ file formats.
(Contributed by Nadeem Vawda and Per Øyvind Karlsen in bpo-6715(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6715
File: python.info, Node: Improved Modules<11>, Next: Optimizations<10>, Prev: New Modules<11>, Up: What’s New In Python 3 3
1.11.20 Improved Modules
------------------------
* Menu:
* abc: abc<2>.
* array: array<5>.
* base64: base64<4>.
* binascii: binascii<3>.
* bz2: bz2<2>.
* codecs: codecs<2>.
* collections: collections<7>.
* contextlib: contextlib<7>.
* crypt: crypt<2>.
* curses: curses<5>.
* datetime: datetime<6>.
* decimal: decimal<3>.
* email: email<5>.
* ftplib: ftplib<3>.
* functools: functools<6>.
* gc: gc<6>.
* hmac: hmac<4>.
* http: http<4>.
* html: html<2>.
* imaplib: imaplib<3>.
* inspect: inspect<9>.
* io: io<6>.
* itertools: itertools<6>.
* logging: logging<7>.
* math: math<9>.
* mmap: mmap<4>.
* multiprocessing: multiprocessing<8>.
* nntplib: nntplib<2>.
* os: os<12>.
* pdb: pdb<7>.
* pickle: pickle<5>.
* pydoc: pydoc<5>.
* re: re<9>.
* sched::
* select: select<2>.
* shlex: shlex<3>.
* shutil: shutil<7>.
* signal: signal<4>.
* smtpd: smtpd<4>.
* smtplib: smtplib<4>.
* socket: socket<10>.
* socketserver: socketserver<3>.
* sqlite3: sqlite3<10>.
* ssl: ssl<11>.
* stat: stat<2>.
* struct: struct<3>.
* subprocess: subprocess<6>.
* sys: sys<13>.
* tarfile: tarfile<9>.
* tempfile: tempfile<4>.
* textwrap: textwrap<2>.
* threading: threading<8>.
* time: time<8>.
* types: types<7>.
* unittest: unittest<10>.
* urllib: urllib<4>.
* webbrowser: webbrowser<3>.
* xml.etree.ElementTree: xml etree ElementTree<2>.
* zlib: zlib<2>.
File: python.info, Node: abc<2>, Next: array<5>, Up: Improved Modules<11>
1.11.20.1 abc
.............
Improved support for abstract base classes containing descriptors
composed with abstract methods. The recommended approach to declaring
abstract descriptors is now to provide ‘__isabstractmethod__’ as a
dynamically updated property. The built-in descriptors have been
updated accordingly.
* *note abc.abstractproperty: 107e. has been deprecated, use *note
property: 194. with *note abc.abstractmethod(): 107f. instead.
* *note abc.abstractclassmethod: 1080. has been deprecated, use *note
classmethod: 166. with *note abc.abstractmethod(): 107f. instead.
* *note abc.abstractstaticmethod: 1081. has been deprecated, use
*note staticmethod: 412. with *note abc.abstractmethod(): 107f.
instead.
(Contributed by Darren Dale in bpo-11610(1).)
*note abc.ABCMeta.register(): 1082. now returns the registered subclass,
which means it can now be used as a class decorator (bpo-10868(2)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11610
(2) https://bugs.python.org/issue?@action=redirect&bpo=10868
File: python.info, Node: array<5>, Next: base64<4>, Prev: abc<2>, Up: Improved Modules<11>
1.11.20.2 array
...............
The *note array: 7. module supports the long long type using ‘q’ and ‘Q’
type codes.
(Contributed by Oren Tirosh and Hirokazu Yamamoto in bpo-1172711(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1172711
File: python.info, Node: base64<4>, Next: binascii<3>, Prev: array<5>, Up: Improved Modules<11>
1.11.20.3 base64
................
ASCII-only Unicode strings are now accepted by the decoding functions of
the *note base64: e. modern interface. For example,
‘base64.b64decode('YWJj')’ returns ‘b'abc'’. (Contributed by Catalin
Iacob in bpo-13641(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13641
File: python.info, Node: binascii<3>, Next: bz2<2>, Prev: base64<4>, Up: Improved Modules<11>
1.11.20.4 binascii
..................
In addition to the binary objects they normally accept, the ‘a2b_’
functions now all also accept ASCII-only strings as input. (Contributed
by Antoine Pitrou in bpo-13637(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13637
File: python.info, Node: bz2<2>, Next: codecs<2>, Prev: binascii<3>, Up: Improved Modules<11>
1.11.20.5 bz2
.............
The *note bz2: 13. module has been rewritten from scratch. In the
process, several new features have been added:
* New *note bz2.open(): 1087. function: open a bzip2-compressed file
in binary or text mode.
* *note bz2.BZ2File: 863. can now read from and write to arbitrary
file-like objects, by means of its constructor’s 'fileobj'
argument.
(Contributed by Nadeem Vawda in bpo-5863(1).)
* *note bz2.BZ2File: 863. and *note bz2.decompress(): 1088. can now
decompress multi-stream inputs (such as those produced by the
‘pbzip2’ tool). *note bz2.BZ2File: 863. can now also be used to
create this type of file, using the ‘'a'’ (append) mode.
(Contributed by Nir Aides in bpo-1625(2).)
* *note bz2.BZ2File: 863. now implements all of the *note
io.BufferedIOBase: 690. API, except for the ‘detach()’ and
‘truncate()’ methods.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5863
(2) https://bugs.python.org/issue?@action=redirect&bpo=1625
File: python.info, Node: codecs<2>, Next: collections<7>, Prev: bz2<2>, Up: Improved Modules<11>
1.11.20.6 codecs
................
The *note mbcs: 53. codec has been rewritten to handle correctly
‘replace’ and ‘ignore’ error handlers on all Windows versions. The
*note mbcs: 53. codec now supports all error handlers, instead of only
‘replace’ to encode and ‘ignore’ to decode.
A new Windows-only codec has been added: ‘cp65001’ (bpo-13216(1)). It
is the Windows code page 65001 (Windows UTF-8, ‘CP_UTF8’). For example,
it is used by ‘sys.stdout’ if the console output code page is set to
cp65001 (e.g., using ‘chcp 65001’ command).
Multibyte CJK decoders now resynchronize faster. They only ignore the
first byte of an invalid byte sequence. For example,
‘b'\xff\n'.decode('gb2312', 'replace')’ now returns a ‘\n’ after the
replacement character.
(bpo-12016(2))
Incremental CJK codec encoders are no longer reset at each call to their
encode() methods. For example:
>>> import codecs
>>> encoder = codecs.getincrementalencoder('hz')('strict')
>>> b''.join(encoder.encode(x) for x in '\u52ff\u65bd\u65bc\u4eba\u3002 Bye.')
b'~{NpJ)l6HK!#~} Bye.'
This example gives ‘b'~{Np~}~{J)~}~{l6~}~{HK~}~{!#~} Bye.'’ with older
Python versions.
(bpo-12100(3))
The ‘unicode_internal’ codec has been deprecated.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13216
(2) https://bugs.python.org/issue?@action=redirect&bpo=12016
(3) https://bugs.python.org/issue?@action=redirect&bpo=12100
File: python.info, Node: collections<7>, Next: contextlib<7>, Prev: codecs<2>, Up: Improved Modules<11>
1.11.20.7 collections
.....................
Addition of a new *note ChainMap: c1c. class to allow treating a number
of mappings as a single unit. (Written by Raymond Hettinger for
bpo-11089(1), made public in bpo-11297(2).)
The abstract base classes have been moved in a new *note
collections.abc: 1e. module, to better differentiate between the
abstract and the concrete collections classes. Aliases for ABCs are
still present in the *note collections: 1d. module to preserve existing
imports. (bpo-11085(3))
The *note Counter: 108b. class now supports the unary ‘+’ and ‘-’
operators, as well as the in-place operators ‘+=’, ‘-=’, ‘|=’, and ‘&=’.
(Contributed by Raymond Hettinger in bpo-13121(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11089
(2) https://bugs.python.org/issue?@action=redirect&bpo=11297
(3) https://bugs.python.org/issue?@action=redirect&bpo=11085
(4) https://bugs.python.org/issue?@action=redirect&bpo=13121
File: python.info, Node: contextlib<7>, Next: crypt<2>, Prev: collections<7>, Up: Improved Modules<11>
1.11.20.8 contextlib
....................
*note ExitStack: b29. now provides a solid foundation for programmatic
manipulation of context managers and similar cleanup functionality.
Unlike the previous ‘contextlib.nested’ API (which was deprecated and
removed), the new API is designed to work correctly regardless of
whether context managers acquire their resources in their ‘__init__’
method (for example, file objects) or in their ‘__enter__’ method (for
example, synchronisation objects from the *note threading: ed. module).
(bpo-13585(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13585
File: python.info, Node: crypt<2>, Next: curses<5>, Prev: contextlib<7>, Up: Improved Modules<11>
1.11.20.9 crypt
...............
Addition of salt and modular crypt format (hashing method) and the
‘mksalt()’ function to the ‘crypt’ module.
(bpo-10924(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10924
File: python.info, Node: curses<5>, Next: datetime<6>, Prev: crypt<2>, Up: Improved Modules<11>
1.11.20.10 curses
.................
* If the *note curses: 2b. module is linked to the ncursesw library,
use Unicode functions when Unicode strings or characters are passed
(e.g. ‘waddwstr()’), and bytes functions otherwise (e.g.
‘waddstr()’).
* Use the locale encoding instead of ‘utf-8’ to encode Unicode
strings.
* ‘curses.window’ has a new *note curses.window.encoding: 108f.
attribute.
* The ‘curses.window’ class has a new *note get_wch(): 1090. method
to get a wide character
* The *note curses: 2b. module has a new *note unget_wch(): 1091.
function to push a wide character so the next *note get_wch():
1090. will return it
(Contributed by Iñigo Serna in bpo-6755(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6755
File: python.info, Node: datetime<6>, Next: decimal<3>, Prev: curses<5>, Up: Improved Modules<11>
1.11.20.11 datetime
...................
* Equality comparisons between naive and aware *note datetime: 1cc.
instances now return *note False: b37. instead of raising *note
TypeError: 534. (bpo-15006(1)).
* New *note datetime.datetime.timestamp(): 1093. method: Return POSIX
timestamp corresponding to the *note datetime: 1cc. instance.
* The *note datetime.datetime.strftime(): c90. method supports
formatting years older than 1000.
* The *note datetime.datetime.astimezone(): 9d0. method can now be
called without arguments to convert datetime instance to the system
timezone.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15006
File: python.info, Node: decimal<3>, Next: email<5>, Prev: datetime<6>, Up: Improved Modules<11>
1.11.20.12 decimal
..................
bpo-7652(1) - integrate fast native decimal arithmetic.
C-module and libmpdec written by Stefan Krah.
The new C version of the decimal module integrates the high speed
libmpdec library for arbitrary precision correctly rounded decimal
floating-point arithmetic. libmpdec conforms to IBM’s General Decimal
Arithmetic Specification.
Performance gains range from 10x for database applications to 100x for
numerically intensive applications. These numbers are expected gains
for standard precisions used in decimal floating-point arithmetic.
Since the precision is user configurable, the exact figures may vary.
For example, in integer bignum arithmetic the differences can be
significantly higher.
The following table is meant as an illustration. Benchmarks are
available at ‘https://www.bytereef.org/mpdecimal/quickstart.html’.
decimal.py _decimal speedup
---------------------------------------------------------------------
pi 42.02s 0.345s 120x
telco 172.19s 5.68s 30x
psycopg 3.57s 0.29s 12x
* Menu:
* Features: Features<2>.
* API changes: API changes<2>.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=7652
File: python.info, Node: Features<2>, Next: API changes<2>, Up: decimal<3>
1.11.20.13 Features
...................
* The *note FloatOperation: 1096. signal optionally enables stricter
semantics for mixing floats and Decimals.
* If Python is compiled without threads, the C version automatically
disables the expensive thread local context machinery. In this
case, the variable *note HAVE_THREADS: 1097. is set to ‘False’.
File: python.info, Node: API changes<2>, Prev: Features<2>, Up: decimal<3>
1.11.20.14 API changes
......................
* The C module has the following context limits, depending on the
machine architecture:
32-bit 64-bit
---------------------------------------------------------------------------
‘MAX_PREC’ ‘425000000’ ‘999999999999999999’
‘MAX_EMAX’ ‘425000000’ ‘999999999999999999’
‘MIN_EMIN’ ‘-425000000’ ‘-999999999999999999’
* In the context templates (*note DefaultContext: 1099, *note
BasicContext: 109a. and *note ExtendedContext: 109b.) the magnitude
of *note Emax: 109c. and *note Emin: 109d. has changed to ‘999999’.
* The *note Decimal: 29f. constructor in decimal.py does not observe
the context limits and converts values with arbitrary exponents or
precision exactly. Since the C version has internal limits, the
following scheme is used: If possible, values are converted
exactly, otherwise *note InvalidOperation: 109e. is raised and the
result is NaN. In the latter case it is always possible to use
*note create_decimal(): 109f. in order to obtain a rounded or
inexact value.
* The power function in decimal.py is always correctly rounded. In
the C version, it is defined in terms of the correctly rounded
*note exp(): 10a0. and *note ln(): 10a1. functions, but the final
result is only “almost always correctly rounded”.
* In the C version, the context dictionary containing the signals is
a *note MutableMapping: 10a2. For speed reasons, *note flags:
10a3. and *note traps: 10a4. always refer to the same *note
MutableMapping: 10a2. that the context was initialized with. If a
new signal dictionary is assigned, *note flags: 10a3. and *note
traps: 10a4. are updated with the new values, but they do not
reference the RHS dictionary.
* Pickling a *note Context: 10a5. produces a different output in
order to have a common interchange format for the Python and C
versions.
* The order of arguments in the *note Context: 10a5. constructor has
been changed to match the order displayed by *note repr(): 7f9.
* The ‘watchexp’ parameter in the *note quantize(): 10a6. method is
deprecated.
File: python.info, Node: email<5>, Next: ftplib<3>, Prev: decimal<3>, Up: Improved Modules<11>
1.11.20.15 email
................
* Menu:
* Policy Framework::
* Provisional Policy with New Header API::
* Other API Changes::
File: python.info, Node: Policy Framework, Next: Provisional Policy with New Header API, Up: email<5>
1.11.20.16 Policy Framework
...........................
The email package now has a *note policy: 4f. framework. A *note
Policy: 10a9. is an object with several methods and properties that
control how the email package behaves. The primary policy for Python
3.3 is the *note Compat32: 10aa. policy, which provides backward
compatibility with the email package in Python 3.2. A ‘policy’ can be
specified when an email message is parsed by a *note parser: 4e, or when
a *note Message: 27e. object is created, or when an email is serialized
using a *note generator: 40. Unless overridden, a policy passed to a
‘parser’ is inherited by all the ‘Message’ object and sub-objects
created by the ‘parser’. By default a ‘generator’ will use the policy
of the ‘Message’ object it is serializing. The default policy is *note
compat32: c99.
The minimum set of controls implemented by all ‘policy’ objects are:
max_line_length The maximum length, excluding the linesep character(s),
individual lines may have when a ‘Message’ is serialized.
Defaults to 78.
linesep The character used to separate individual lines when a
‘Message’ is serialized. Defaults to ‘\n’.
cte_type ‘7bit’ or ‘8bit’. ‘8bit’ applies only to a ‘Bytes’
‘generator’, and means that non-ASCII may be used where
allowed by the protocol (or where it exists in the
original input).
raise_on_defect Causes a ‘parser’ to raise error when defects are
encountered instead of adding them to the ‘Message’
object’s ‘defects’ list.
A new policy instance, with new settings, is created using the *note
clone(): 10ab. method of policy objects. ‘clone’ takes any of the above
controls as keyword arguments. Any control not specified in the call
retains its default value. Thus you can create a policy that uses
‘\r\n’ linesep characters like this:
mypolicy = compat32.clone(linesep='\r\n')
Policies can be used to make the generation of messages in the format
needed by your application simpler. Instead of having to remember to
specify ‘linesep='\r\n'’ in all the places you call a ‘generator’, you
can specify it once, when you set the policy used by the ‘parser’ or the
‘Message’, whichever your program uses to create ‘Message’ objects. On
the other hand, if you need to generate messages in multiple forms, you
can still specify the parameters in the appropriate ‘generator’ call.
Or you can have custom policy instances for your different cases, and
pass those in when you create the ‘generator’.
File: python.info, Node: Provisional Policy with New Header API, Next: Other API Changes, Prev: Policy Framework, Up: email<5>
1.11.20.17 Provisional Policy with New Header API
.................................................
While the policy framework is worthwhile all by itself, the main
motivation for introducing it is to allow the creation of new policies
that implement new features for the email package in a way that
maintains backward compatibility for those who do not use the new
policies. Because the new policies introduce a new API, we are
releasing them in Python 3.3 as a *note provisional policy: 103a.
Backwards incompatible changes (up to and including removal of the code)
may occur if deemed necessary by the core developers.
The new policies are instances of *note EmailPolicy: 10ad, and add the
following additional controls:
refold_source Controls whether or not headers parsed by a
*note parser: 4e. are refolded by the
*note generator: 40. It can be ‘none’, ‘long’, or ‘all’.
The default is ‘long’, which means that source headers
with a line longer than ‘max_line_length’ get refolded.
‘none’ means no line get refolded, and ‘all’ means that
all lines get refolded.
header_factory A callable that take a ‘name’ and ‘value’ and produces a
custom header object.
The ‘header_factory’ is the key to the new features provided by the new
policies. When one of the new policies is used, any header retrieved
from a ‘Message’ object is an object produced by the ‘header_factory’,
and any time you set a header on a ‘Message’ it becomes an object
produced by ‘header_factory’. All such header objects have a ‘name’
attribute equal to the header name. Address and Date headers have
additional attributes that give you access to the parsed data of the
header. This means you can now do things like this:
>>> m = Message(policy=SMTP)
>>> m['To'] = 'Éric '
>>> m['to']
'Éric '
>>> m['to'].addresses
(Address(display_name='Éric', username='foo', domain='example.com'),)
>>> m['to'].addresses[0].username
'foo'
>>> m['to'].addresses[0].display_name
'Éric'
>>> m['Date'] = email.utils.localtime()
>>> m['Date'].datetime
datetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT'))
>>> m['Date']
'Fri, 25 May 2012 21:44:27 -0400'
>>> print(m)
To: =?utf-8?q?=C3=89ric?=
Date: Fri, 25 May 2012 21:44:27 -0400
You will note that the unicode display name is automatically encoded as
‘utf-8’ when the message is serialized, but that when the header is
accessed directly, you get the unicode version. This eliminates any
need to deal with the *note email.header: 41. *note decode_header():
10ae. or *note make_header(): 10af. functions.
You can also create addresses from parts:
>>> m['cc'] = [Group('pals', [Address('Bob', 'bob', 'example.com'),
... Address('Sally', 'sally', 'example.com')]),
... Address('Bonzo', addr_spec='bonz@laugh.com')]
>>> print(m)
To: =?utf-8?q?=C3=89ric?=
Date: Fri, 25 May 2012 21:44:27 -0400
cc: pals: Bob , Sally ;, Bonzo
Decoding to unicode is done automatically:
>>> m2 = message_from_string(str(m))
>>> m2['to']
'Éric '
When you parse a message, you can use the ‘addresses’ and ‘groups’
attributes of the header objects to access the groups and individual
addresses:
>>> m2['cc'].addresses
(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com'))
>>> m2['cc'].groups
(Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),))
In summary, if you use one of the new policies, header manipulation
works the way it ought to: your application works with unicode strings,
and the email package transparently encodes and decodes the unicode to
and from the RFC standard Content Transfer Encodings.
File: python.info, Node: Other API Changes, Prev: Provisional Policy with New Header API, Up: email<5>
1.11.20.18 Other API Changes
............................
New *note BytesHeaderParser: 10b1, added to the *note parser: 4e. module
to complement *note HeaderParser: 10b2. and complete the Bytes API.
New utility functions:
* *note format_datetime(): 10b3.: given a *note datetime: 1cc,
produce a string formatted for use in an email header.
* *note parsedate_to_datetime(): 10b4.: given a date string from an
email header, convert it into an aware *note datetime: 1cc, or a
naive *note datetime: 1cc. if the offset is ‘-0000’.
* *note localtime(): 2c7.: With no argument, returns the current
local time as an aware *note datetime: 1cc. using the local *note
timezone: 10b5. Given an aware *note datetime: 1cc, converts it
into an aware *note datetime: 1cc. using the local *note timezone:
10b5.
File: python.info, Node: ftplib<3>, Next: functools<6>, Prev: email<5>, Up: Improved Modules<11>
1.11.20.19 ftplib
.................
* *note ftplib.FTP: 8f9. now accepts a ‘source_address’ keyword
argument to specify the ‘(host, port)’ to use as the source address
in the bind call when creating the outgoing socket. (Contributed
by Giampaolo Rodolà in bpo-8594(1).)
* The *note FTP_TLS: 8fa. class now provides a new *note ccc(): 10b7.
function to revert control channel back to plaintext. This can be
useful to take advantage of firewalls that know how to handle NAT
with non-secure FTP without opening fixed ports. (Contributed by
Giampaolo Rodolà in bpo-12139(2).)
* Added *note ftplib.FTP.mlsd(): 10b8. method which provides a
parsable directory listing format and deprecates *note
ftplib.FTP.nlst(): 10b9. and *note ftplib.FTP.dir(): 10ba.
(Contributed by Giampaolo Rodolà in bpo-11072(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8594
(2) https://bugs.python.org/issue?@action=redirect&bpo=12139
(3) https://bugs.python.org/issue?@action=redirect&bpo=11072
File: python.info, Node: functools<6>, Next: gc<6>, Prev: ftplib<3>, Up: Improved Modules<11>
1.11.20.20 functools
....................
The *note functools.lru_cache(): 9ee. decorator now accepts a ‘typed’
keyword argument (that defaults to ‘False’ to ensure that it caches
values of different types that compare equal in separate cache slots.
(Contributed by Raymond Hettinger in bpo-13227(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13227
File: python.info, Node: gc<6>, Next: hmac<4>, Prev: functools<6>, Up: Improved Modules<11>
1.11.20.21 gc
.............
It is now possible to register callbacks invoked by the garbage
collector before and after collection using the new *note callbacks:
10bd. list.
File: python.info, Node: hmac<4>, Next: http<4>, Prev: gc<6>, Up: Improved Modules<11>
1.11.20.22 hmac
...............
A new *note compare_digest(): 10bf. function has been added to prevent
side channel attacks on digests through timing analysis. (Contributed
by Nick Coghlan and Christian Heimes in bpo-15061(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15061
File: python.info, Node: http<4>, Next: html<2>, Prev: hmac<4>, Up: Improved Modules<11>
1.11.20.23 http
...............
*note http.server.BaseHTTPRequestHandler: 10c1. now buffers the headers
and writes them all at once when *note end_headers(): 10c2. is called.
A new method *note flush_headers(): 10c3. can be used to directly manage
when the accumulated headers are sent. (Contributed by Andrew Schaaf in
bpo-3709(1).)
*note http.server: 72. now produces valid ‘HTML 4.01 strict’ output.
(Contributed by Ezio Melotti in bpo-13295(2).)
*note http.client.HTTPResponse: 10c4. now has a *note readinto(): 10c5.
method, which means it can be used as an *note io.RawIOBase: 10c6.
class. (Contributed by John Kuhn in bpo-13464(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=3709
(2) https://bugs.python.org/issue?@action=redirect&bpo=13295
(3) https://bugs.python.org/issue?@action=redirect&bpo=13464
File: python.info, Node: html<2>, Next: imaplib<3>, Prev: http<4>, Up: Improved Modules<11>
1.11.20.24 html
...............
*note html.parser.HTMLParser: 874. is now able to parse broken markup
without raising errors, therefore the 'strict' argument of the
constructor and the ‘HTMLParseError’ exception are now deprecated. The
ability to parse broken markup is the result of a number of bug fixes
that are also available on the latest bug fix releases of Python
2.7/3.2. (Contributed by Ezio Melotti in bpo-15114(1), and
bpo-14538(2), bpo-13993(3), bpo-13960(4), bpo-13358(5), bpo-1745761(6),
bpo-755670(7), bpo-13357(8), bpo-12629(9), bpo-1200313(10),
bpo-670664(11), bpo-13273(12), bpo-12888(13), bpo-7311(14).)
A new *note html5: 10c8. dictionary that maps HTML5 named character
references to the equivalent Unicode character(s) (e.g. ‘html5['gt;']
== '>'’) has been added to the *note html.entities: 6c. module. The
dictionary is now also used by *note HTMLParser: 874. (Contributed by
Ezio Melotti in bpo-11113(15) and bpo-15156(16).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=15114
(2) https://bugs.python.org/issue?@action=redirect&bpo=14538
(3) https://bugs.python.org/issue?@action=redirect&bpo=13993
(4) https://bugs.python.org/issue?@action=redirect&bpo=13960
(5) https://bugs.python.org/issue?@action=redirect&bpo=13358
(6) https://bugs.python.org/issue?@action=redirect&bpo=1745761
(7) https://bugs.python.org/issue?@action=redirect&bpo=755670
(8) https://bugs.python.org/issue?@action=redirect&bpo=13357
(9) https://bugs.python.org/issue?@action=redirect&bpo=12629
(10) https://bugs.python.org/issue?@action=redirect&bpo=1200313
(11) https://bugs.python.org/issue?@action=redirect&bpo=670664
(12) https://bugs.python.org/issue?@action=redirect&bpo=13273
(13) https://bugs.python.org/issue?@action=redirect&bpo=12888
(14) https://bugs.python.org/issue?@action=redirect&bpo=7311
(15) https://bugs.python.org/issue?@action=redirect&bpo=11113
(16) https://bugs.python.org/issue?@action=redirect&bpo=15156
File: python.info, Node: imaplib<3>, Next: inspect<9>, Prev: html<2>, Up: Improved Modules<11>
1.11.20.25 imaplib
..................
The *note IMAP4_SSL: 903. constructor now accepts an SSLContext
parameter to control parameters of the secure channel.
(Contributed by Sijin Joseph in bpo-8808(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8808
File: python.info, Node: inspect<9>, Next: io<6>, Prev: imaplib<3>, Up: Improved Modules<11>
1.11.20.26 inspect
..................
A new *note getclosurevars(): 10cb. function has been added. This
function reports the current binding of all names referenced from the
function body and where those names were resolved, making it easier to
verify correct internal state when testing code that relies on stateful
closures.
(Contributed by Meador Inge and Nick Coghlan in bpo-13062(1).)
A new *note getgeneratorlocals(): 10cc. function has been added. This
function reports the current binding of local variables in the
generator’s stack frame, making it easier to verify correct internal
state when testing generators.
(Contributed by Meador Inge in bpo-15153(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13062
(2) https://bugs.python.org/issue?@action=redirect&bpo=15153
File: python.info, Node: io<6>, Next: itertools<6>, Prev: inspect<9>, Up: Improved Modules<11>
1.11.20.27 io
.............
The *note open(): 518. function has a new ‘'x'’ mode that can be used to
exclusively create a new file, and raise a *note FileExistsError: 1019.
if the file already exists. It is based on the C11 ‘x’ mode to fopen().
(Contributed by David Townshend in bpo-12760(1).)
The constructor of the *note TextIOWrapper: 7b6. class has a new
'write_through' optional argument. If 'write_through' is ‘True’, calls
to ‘write()’ are guaranteed not to be buffered: any data written on the
*note TextIOWrapper: 7b6. object is immediately handled to its
underlying binary buffer.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12760
File: python.info, Node: itertools<6>, Next: logging<7>, Prev: io<6>, Up: Improved Modules<11>
1.11.20.28 itertools
....................
*note accumulate(): 9fc. now takes an optional ‘func’ argument for
providing a user-supplied binary function.
File: python.info, Node: logging<7>, Next: math<9>, Prev: itertools<6>, Up: Improved Modules<11>
1.11.20.29 logging
..................
The *note basicConfig(): 9ff. function now supports an optional
‘handlers’ argument taking an iterable of handlers to be added to the
root logger.
A class level attribute ‘append_nul’ has been added to *note
SysLogHandler: 658. to allow control of the appending of the ‘NUL’
(‘\000’) byte to syslog records, since for some daemons it is required
while for others it is passed through to the log.
File: python.info, Node: math<9>, Next: mmap<4>, Prev: logging<7>, Up: Improved Modules<11>
1.11.20.30 math
...............
The *note math: 8e. module has a new function, *note log2(): 10d1, which
returns the base-2 logarithm of 'x'.
(Written by Mark Dickinson in bpo-11888(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11888
File: python.info, Node: mmap<4>, Next: multiprocessing<8>, Prev: math<9>, Up: Improved Modules<11>
1.11.20.31 mmap
...............
The *note read(): 10d3. method is now more compatible with other
file-like objects: if the argument is omitted or specified as ‘None’, it
returns the bytes from the current file position to the end of the
mapping. (Contributed by Petri Lehtinen in bpo-12021(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12021
File: python.info, Node: multiprocessing<8>, Next: nntplib<2>, Prev: mmap<4>, Up: Improved Modules<11>
1.11.20.32 multiprocessing
..........................
The new *note multiprocessing.connection.wait(): 10d5. function allows
polling multiple objects (such as connections, sockets and pipes) with a
timeout. (Contributed by Richard Oudkerk in bpo-12328(1).)
‘multiprocessing.Connection’ objects can now be transferred over
multiprocessing connections. (Contributed by Richard Oudkerk in
bpo-4892(2).)
*note multiprocessing.Process: b59. now accepts a ‘daemon’ keyword
argument to override the default behavior of inheriting the ‘daemon’
flag from the parent process (bpo-6064(3)).
New attribute *note multiprocessing.Process.sentinel: 10d6. allows a
program to wait on multiple *note Process: b59. objects at one time
using the appropriate OS primitives (for example, *note select: c1. on
posix systems).
New methods *note multiprocessing.pool.Pool.starmap(): 10d7. and *note
starmap_async(): 10d8. provide *note itertools.starmap(): 10d9.
equivalents to the existing *note multiprocessing.pool.Pool.map(): 10da.
and *note map_async(): 10db. functions. (Contributed by Hynek Schlawack
in bpo-12708(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12328
(2) https://bugs.python.org/issue?@action=redirect&bpo=4892
(3) https://bugs.python.org/issue?@action=redirect&bpo=6064
(4) https://bugs.python.org/issue?@action=redirect&bpo=12708
File: python.info, Node: nntplib<2>, Next: os<12>, Prev: multiprocessing<8>, Up: Improved Modules<11>
1.11.20.33 nntplib
..................
The ‘nntplib.NNTP’ class now supports the context management protocol to
unconditionally consume *note socket.error: 104e. exceptions and to
close the NNTP connection when done:
>>> from nntplib import NNTP
>>> with NNTP('news.gmane.org') as n:
... n.group('gmane.comp.python.committers')
...
('211 1755 1 1755 gmane.comp.python.committers', 1755, 1, 1755, 'gmane.comp.python.committers')
>>>
(Contributed by Giampaolo Rodolà in bpo-9795(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9795
File: python.info, Node: os<12>, Next: pdb<7>, Prev: nntplib<2>, Up: Improved Modules<11>
1.11.20.34 os
.............
* The *note os: a1. module has a new *note pipe2(): 10de. function
that makes it possible to create a pipe with *note O_CLOEXEC: 1075.
or *note O_NONBLOCK: e3a. flags set atomically. This is especially
useful to avoid race conditions in multi-threaded programs.
* The *note os: a1. module has a new *note sendfile(): b0f. function
which provides an efficient “zero-copy” way for copying data from
one file (or socket) descriptor to another. The phrase “zero-copy”
refers to the fact that all of the copying of data between the two
descriptors is done entirely by the kernel, with no copying of data
into userspace buffers. *note sendfile(): b0f. can be used to
efficiently copy data from a file on disk to a network socket, e.g.
for downloading a file.
(Patch submitted by Ross Lagerwall and Giampaolo Rodolà in
bpo-10882(1).)
* To avoid race conditions like symlink attacks and issues with
temporary files and directories, it is more reliable (and also
faster) to manipulate file descriptors instead of file names.
Python 3.3 enhances existing functions and introduces new functions
to work on file descriptors (bpo-4761(2), bpo-10755(3) and
bpo-14626(4)).
- The *note os: a1. module has a new *note fwalk(): b5b.
function similar to *note walk(): 4a5. except that it also
yields file descriptors referring to the directories visited.
This is especially useful to avoid symlink races.
- The following functions get new optional 'dir_fd' (*note paths
relative to directory descriptors: 10df.) and/or
'follow_symlinks' (*note not following symlinks: 10e0.): *note
access(): 10e1, *note chflags(): 10e2, *note chmod(): 21d,
*note chown(): 10e3, *note link(): 10e4, *note lstat(): 49d,
*note mkdir(): 21f, *note mkfifo(): d8f, *note mknod(): d90,
*note open(): d91, *note readlink(): 91f, *note remove():
10e5, *note rename(): 10e6, *note replace(): 10e7, *note
rmdir(): 10e8, *note stat(): 49c, *note symlink(): 10e9, *note
unlink(): 10ea, *note utime(): 10eb. Platform support for
using these parameters can be checked via the sets *note
os.supports_dir_fd: 10ec. and ‘os.supports_follows_symlinks’.
- The following functions now support a file descriptor for
their path argument: *note chdir(): 609, *note chmod(): 21d,
*note chown(): 10e3, *note execve(): 10ed, *note listdir():
10ee, *note pathconf(): 10ef, *note exists(): a0d, *note
stat(): 49c, *note statvfs(): 10f0, *note utime(): 10eb.
Platform support for this can be checked via the *note
os.supports_fd: 10f1. set.
* *note access(): 10e1. accepts an ‘effective_ids’ keyword argument
to turn on using the effective uid/gid rather than the real uid/gid
in the access check. Platform support for this can be checked via
the *note supports_effective_ids: 10f2. set.
* The *note os: a1. module has two new functions: *note
getpriority(): 10f3. and *note setpriority(): 10f4. They can be
used to get or set process niceness/priority in a fashion similar
to *note os.nice(): 10f5. but extended to all processes instead of
just the current one.
(Patch submitted by Giampaolo Rodolà in bpo-10784(5).)
* The new *note os.replace(): 10e7. function allows cross-platform
renaming of a file with overwriting the destination. With *note
os.rename(): 10e6, an existing destination file is overwritten
under POSIX, but raises an error under Windows. (Contributed by
Antoine Pitrou in bpo-8828(6).)
* The stat family of functions (*note stat(): 49c, *note fstat():
d8b, and *note lstat(): 49d.) now support reading a file’s
timestamps with nanosecond precision. Symmetrically, *note
utime(): 10eb. can now write file timestamps with nanosecond
precision. (Contributed by Larry Hastings in bpo-14127(7).)
* The new *note os.get_terminal_size(): 10f6. function queries the
size of the terminal attached to a file descriptor. See also *note
shutil.get_terminal_size(): 10f7. (Contributed by Zbigniew
Jędrzejewski-Szmek in bpo-13609(8).)
* New functions to support Linux extended attributes (bpo-12720(9)):
*note getxattr(): 10f8, *note listxattr(): 10f9, *note
removexattr(): 10fa, *note setxattr(): 10fb.
* New interface to the scheduler. These functions control how a
process is allocated CPU time by the operating system. New
functions: *note sched_get_priority_max(): 10fc, *note
sched_get_priority_min(): 10fd, *note sched_getaffinity(): 10fe,
*note sched_getparam(): 10ff, *note sched_getscheduler(): 1100,
*note sched_rr_get_interval(): 1101, *note sched_setaffinity():
1102, *note sched_setparam(): 1103, *note sched_setscheduler():
1104, *note sched_yield(): 1105,
* New functions to control the file system:
* *note posix_fadvise(): d92.: Announces an intention to access
data in a specific pattern thus allowing the kernel to make
optimizations.
* *note posix_fallocate(): d93.: Ensures that enough disk space
is allocated for a file.
* *note sync(): 1106.: Force write of everything to disk.
* Additional new posix functions:
* *note lockf(): 1107.: Apply, test or remove a POSIX lock on an
open file descriptor.
* *note pread(): b5f.: Read from a file descriptor at an offset,
the file offset remains unchanged.
* *note pwrite(): b62.: Write to a file descriptor from an
offset, leaving the file offset unchanged.
* *note readv(): b5e.: Read from a file descriptor into a number
of writable buffers.
* *note truncate(): e3b.: Truncate the file corresponding to
'path', so that it is at most 'length' bytes in size.
* *note waitid(): d98.: Wait for the completion of one or more
child processes.
* *note writev(): b61.: Write the contents of 'buffers' to a
file descriptor, where 'buffers' is an arbitrary sequence of
buffers.
* *note getgrouplist(): 1108. (bpo-9344(10)): Return list of
group ids that specified user belongs to.
* *note times(): 1109. and *note uname(): 110a.: Return type changed
from a tuple to a tuple-like object with named attributes.
* Some platforms now support additional constants for the *note
lseek(): 110b. function, such as ‘os.SEEK_HOLE’ and ‘os.SEEK_DATA’.
* New constants *note RTLD_LAZY: 110c, *note RTLD_NOW: 110d, *note
RTLD_GLOBAL: 110e, *note RTLD_LOCAL: 110f, *note RTLD_NODELETE:
1110, *note RTLD_NOLOAD: 1111, and *note RTLD_DEEPBIND: 1112. are
available on platforms that support them. These are for use with
the *note sys.setdlopenflags(): 1113. function, and supersede the
similar constants defined in *note ctypes: 2a. and ‘DLFCN’.
(Contributed by Victor Stinner in bpo-13226(11).)
* *note os.symlink(): 10e9. now accepts (and ignores) the
‘target_is_directory’ keyword argument on non-Windows platforms, to
ease cross-platform support.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10882
(2) https://bugs.python.org/issue?@action=redirect&bpo=4761
(3) https://bugs.python.org/issue?@action=redirect&bpo=10755
(4) https://bugs.python.org/issue?@action=redirect&bpo=14626
(5) https://bugs.python.org/issue?@action=redirect&bpo=10784
(6) https://bugs.python.org/issue?@action=redirect&bpo=8828
(7) https://bugs.python.org/issue?@action=redirect&bpo=14127
(8) https://bugs.python.org/issue?@action=redirect&bpo=13609
(9) https://bugs.python.org/issue?@action=redirect&bpo=12720
(10) https://bugs.python.org/issue?@action=redirect&bpo=9344
(11) https://bugs.python.org/issue?@action=redirect&bpo=13226
File: python.info, Node: pdb<7>, Next: pickle<5>, Prev: os<12>, Up: Improved Modules<11>
1.11.20.35 pdb
..............
Tab-completion is now available not only for command names, but also
their arguments. For example, for the ‘break’ command, function and
file names are completed.
(Contributed by Georg Brandl in bpo-14210(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14210
File: python.info, Node: pickle<5>, Next: pydoc<5>, Prev: pdb<7>, Up: Improved Modules<11>
1.11.20.36 pickle
.................
*note pickle.Pickler: a1d. objects now have an optional *note
dispatch_table: 1116. attribute allowing per-pickler reduction functions
to be set.
(Contributed by Richard Oudkerk in bpo-14166(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14166
File: python.info, Node: pydoc<5>, Next: re<9>, Prev: pickle<5>, Up: Improved Modules<11>
1.11.20.37 pydoc
................
The Tk GUI and the ‘serve()’ function have been removed from the *note
pydoc: b5. module: ‘pydoc -g’ and ‘serve()’ have been deprecated in
Python 3.2.
File: python.info, Node: re<9>, Next: sched, Prev: pydoc<5>, Up: Improved Modules<11>
1.11.20.38 re
.............
*note str: 447. regular expressions now support ‘\u’ and ‘\U’ escapes.
(Contributed by Serhiy Storchaka in bpo-3665(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=3665
File: python.info, Node: sched, Next: select<2>, Prev: re<9>, Up: Improved Modules<11>
1.11.20.39 sched
................
* *note run(): 111a. now accepts a 'blocking' parameter which when
set to false makes the method execute the scheduled events due to
expire soonest (if any) and then return immediately. This is
useful in case you want to use the *note scheduler: 111b. in
non-blocking applications. (Contributed by Giampaolo Rodolà in
bpo-13449(1).)
* *note scheduler: 111b. class can now be safely used in
multi-threaded environments. (Contributed by Josiah Carlson and
Giampaolo Rodolà in bpo-8684(2).)
* 'timefunc' and 'delayfunct' parameters of *note scheduler: 111b.
class constructor are now optional and defaults to *note
time.time(): 256. and *note time.sleep(): 699. respectively.
(Contributed by Chris Clark in bpo-13245(3).)
* *note enter(): 111c. and *note enterabs(): 111d. 'argument'
parameter is now optional. (Contributed by Chris Clark in
bpo-13245(4).)
* *note enter(): 111c. and *note enterabs(): 111d. now accept a
'kwargs' parameter. (Contributed by Chris Clark in bpo-13245(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13449
(2) https://bugs.python.org/issue?@action=redirect&bpo=8684
(3) https://bugs.python.org/issue?@action=redirect&bpo=13245
(4) https://bugs.python.org/issue?@action=redirect&bpo=13245
(5) https://bugs.python.org/issue?@action=redirect&bpo=13245
File: python.info, Node: select<2>, Next: shlex<3>, Prev: sched, Up: Improved Modules<11>
1.11.20.40 select
.................
Solaris and derivative platforms have a new class *note select.devpoll:
f90. for high performance asynchronous sockets via ‘/dev/poll’.
(Contributed by Jesús Cea Avión in bpo-6397(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6397
File: python.info, Node: shlex<3>, Next: shutil<7>, Prev: select<2>, Up: Improved Modules<11>
1.11.20.41 shlex
................
The previously undocumented helper function ‘quote’ from the ‘pipes’
modules has been moved to the *note shlex: c4. module and documented.
*note quote(): 27f. properly escapes all characters in a string that
might be otherwise given special meaning by the shell.
File: python.info, Node: shutil<7>, Next: signal<4>, Prev: shlex<3>, Up: Improved Modules<11>
1.11.20.42 shutil
.................
* New functions:
* *note disk_usage(): 1121.: provides total, used and free disk
space statistics. (Contributed by Giampaolo Rodolà in
bpo-12442(1).)
* *note chown(): 242.: allows one to change user and/or group of
the given path also specifying the user/group names and not
only their numeric ids. (Contributed by Sandro Tosi in
bpo-12191(2).)
* *note shutil.get_terminal_size(): 10f7.: returns the size of
the terminal window to which the interpreter is attached.
(Contributed by Zbigniew Jędrzejewski-Szmek in bpo-13609(3).)
* *note copy2(): a5a. and *note copystat(): 1122. now preserve file
timestamps with nanosecond precision on platforms that support it.
They also preserve file “extended attributes” on Linux.
(Contributed by Larry Hastings in bpo-14127(4) and bpo-15238(5).)
* Several functions now take an optional ‘symlinks’ argument: when
that parameter is true, symlinks aren’t dereferenced and the
operation instead acts on the symlink itself (or creates one, if
relevant). (Contributed by Hynek Schlawack in bpo-12715(6).)
* When copying files to a different file system, *note move(): a5b.
now handles symlinks the way the posix ‘mv’ command does,
recreating the symlink rather than copying the target file
contents. (Contributed by Jonathan Niehof in bpo-9993(7).) *note
move(): a5b. now also returns the ‘dst’ argument as its result.
* *note rmtree(): 2fb. is now resistant to symlink attacks on
platforms which support the new ‘dir_fd’ parameter in *note
os.open(): d91. and *note os.unlink(): 10ea. (Contributed by
Martin von Löwis and Hynek Schlawack in bpo-4489(8).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12442
(2) https://bugs.python.org/issue?@action=redirect&bpo=12191
(3) https://bugs.python.org/issue?@action=redirect&bpo=13609
(4) https://bugs.python.org/issue?@action=redirect&bpo=14127
(5) https://bugs.python.org/issue?@action=redirect&bpo=15238
(6) https://bugs.python.org/issue?@action=redirect&bpo=12715
(7) https://bugs.python.org/issue?@action=redirect&bpo=9993
(8) https://bugs.python.org/issue?@action=redirect&bpo=4489
File: python.info, Node: signal<4>, Next: smtpd<4>, Prev: shutil<7>, Up: Improved Modules<11>
1.11.20.43 signal
.................
* The *note signal: c6. module has new functions:
* *note pthread_sigmask(): 1124.: fetch and/or change the signal
mask of the calling thread (Contributed by Jean-Paul Calderone
in bpo-8407(1));
* *note pthread_kill(): 1125.: send a signal to a thread;
* *note sigpending(): 1126.: examine pending functions;
* *note sigwait(): 1127.: wait a signal;
* *note sigwaitinfo(): daa.: wait for a signal, returning
detailed information about it;
* *note sigtimedwait(): da9.: like *note sigwaitinfo(): daa. but
with a timeout.
* The signal handler writes the signal number as a single byte
instead of a nul byte into the wakeup file descriptor. So it is
possible to wait more than one signal and know which signals were
raised.
* *note signal.signal(): 1128. and *note signal.siginterrupt(): 1129.
raise an OSError, instead of a RuntimeError: OSError has an errno
attribute.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8407
File: python.info, Node: smtpd<4>, Next: smtplib<4>, Prev: signal<4>, Up: Improved Modules<11>
1.11.20.44 smtpd
................
The ‘smtpd’ module now supports RFC 5321(1) (extended SMTP) and RFC
1870(2) (size extension). Per the standard, these extensions are
enabled if and only if the client initiates the session with an ‘EHLO’
command.
(Initial ‘ELHO’ support by Alberto Trevino. Size extension by Juhana
Jauhiainen. Substantial additional work on the patch contributed by
Michele Orrù and Dan Boswell. bpo-8739(3))
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc5321.html
(2) https://datatracker.ietf.org/doc/html/rfc1870.html
(3) https://bugs.python.org/issue?@action=redirect&bpo=8739
File: python.info, Node: smtplib<4>, Next: socket<10>, Prev: smtpd<4>, Up: Improved Modules<11>
1.11.20.45 smtplib
..................
The *note SMTP: 92d, *note SMTP_SSL: 92e, and *note LMTP: 92f. classes
now accept a ‘source_address’ keyword argument to specify the ‘(host,
port)’ to use as the source address in the bind call when creating the
outgoing socket. (Contributed by Paulo Scardine in bpo-11281(1).)
*note SMTP: 92d. now supports the context management protocol, allowing
an ‘SMTP’ instance to be used in a ‘with’ statement. (Contributed by
Giampaolo Rodolà in bpo-11289(2).)
The *note SMTP_SSL: 92e. constructor and the *note starttls(): 112c.
method now accept an SSLContext parameter to control parameters of the
secure channel. (Contributed by Kasun Herath in bpo-8809(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11281
(2) https://bugs.python.org/issue?@action=redirect&bpo=11289
(3) https://bugs.python.org/issue?@action=redirect&bpo=8809
File: python.info, Node: socket<10>, Next: socketserver<3>, Prev: smtplib<4>, Up: Improved Modules<11>
1.11.20.46 socket
.................
* The *note socket: da0. class now exposes additional methods to
process ancillary data when supported by the underlying platform:
* *note sendmsg(): 472.
* *note recvmsg(): da5.
* *note recvmsg_into(): 112e.
(Contributed by David Watson in bpo-6560(1), based on an earlier
patch by Heiko Wundram)
* The *note socket: da0. class now supports the PF_CAN protocol
family (‘https://en.wikipedia.org/wiki/Socketcan’), on Linux
(‘https://lwn.net/Articles/253425’).
(Contributed by Matthias Fuchs, updated by Tiago Gonçalves in
bpo-10141(2).)
* The *note socket: da0. class now supports the PF_RDS protocol
family (‘https://en.wikipedia.org/wiki/Reliable_Datagram_Sockets’
and https://oss.oracle.com/projects/rds(3)).
* The *note socket: da0. class now supports the ‘PF_SYSTEM’ protocol
family on OS X. (Contributed by Michael Goderbauer in
bpo-13777(4).)
* New function *note sethostname(): 112f. allows the hostname to be
set on Unix systems if the calling process has sufficient
privileges. (Contributed by Ross Lagerwall in bpo-10866(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6560
(2) https://bugs.python.org/issue?@action=redirect&bpo=10141
(3)
https://web.archive.org/web/20130115155505/https://oss.oracle.com/projects/rds/
(4) https://bugs.python.org/issue?@action=redirect&bpo=13777
(5) https://bugs.python.org/issue?@action=redirect&bpo=10866
File: python.info, Node: socketserver<3>, Next: sqlite3<10>, Prev: socket<10>, Up: Improved Modules<11>
1.11.20.47 socketserver
.......................
*note BaseServer: 1131. now has an overridable method *note
service_actions(): 1132. that is called by the *note serve_forever():
1133. method in the service loop. *note ForkingMixIn: b77. now uses
this to clean up zombie child processes. (Contributed by Justin
Warkentin in bpo-11109(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11109
File: python.info, Node: sqlite3<10>, Next: ssl<11>, Prev: socketserver<3>, Up: Improved Modules<11>
1.11.20.48 sqlite3
..................
New *note sqlite3.Connection: 247. method *note set_trace_callback():
2af. can be used to capture a trace of all sql commands processed by
sqlite. (Contributed by Torsten Landschoff in bpo-11688(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11688
File: python.info, Node: ssl<11>, Next: stat<2>, Prev: sqlite3<10>, Up: Improved Modules<11>
1.11.20.49 ssl
..............
* The *note ssl: d0. module has two new random generation functions:
* *note RAND_bytes(): 51f.: generate cryptographically strong
pseudo-random bytes.
* ‘RAND_pseudo_bytes()’: generate pseudo-random bytes.
(Contributed by Victor Stinner in bpo-12049(1).)
* The *note ssl: d0. module now exposes a finer-grained exception
hierarchy in order to make it easier to inspect the various kinds
of errors. (Contributed by Antoine Pitrou in bpo-11183(2).)
* *note load_cert_chain(): 297. now accepts a 'password' argument to
be used if the private key is encrypted. (Contributed by Adam
Simpkins in bpo-12803(3).)
* Diffie-Hellman key exchange, both regular and Elliptic Curve-based,
is now supported through the *note load_dh_params(): 1136. and
*note set_ecdh_curve(): 1137. methods. (Contributed by Antoine
Pitrou in bpo-13626(4) and bpo-13627(5).)
* SSL sockets have a new *note get_channel_binding(): 1138. method
allowing the implementation of certain authentication mechanisms
such as SCRAM-SHA-1-PLUS. (Contributed by Jacek Konieczny in
bpo-12551(6).)
* You can query the SSL compression algorithm used by an SSL socket,
thanks to its new *note compression(): 1139. method. The new
attribute *note OP_NO_COMPRESSION: 113a. can be used to disable
compression. (Contributed by Antoine Pitrou in bpo-13634(7).)
* Support has been added for the Next Protocol Negotiation extension
using the *note ssl.SSLContext.set_npn_protocols(): 2fc. method.
(Contributed by Colin Marc in bpo-14204(8).)
* SSL errors can now be introspected more easily thanks to *note
library: 113b. and *note reason: 113c. attributes. (Contributed by
Antoine Pitrou in bpo-14837(9).)
* The *note get_server_certificate(): 838. function now supports
IPv6. (Contributed by Charles-François Natali in bpo-11811(10).)
* New attribute *note OP_CIPHER_SERVER_PREFERENCE: 113d. allows
setting SSLv3 server sockets to use the server’s cipher ordering
preference rather than the client’s (bpo-13635(11)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12049
(2) https://bugs.python.org/issue?@action=redirect&bpo=11183
(3) https://bugs.python.org/issue?@action=redirect&bpo=12803
(4) https://bugs.python.org/issue?@action=redirect&bpo=13626
(5) https://bugs.python.org/issue?@action=redirect&bpo=13627
(6) https://bugs.python.org/issue?@action=redirect&bpo=12551
(7) https://bugs.python.org/issue?@action=redirect&bpo=13634
(8) https://bugs.python.org/issue?@action=redirect&bpo=14204
(9) https://bugs.python.org/issue?@action=redirect&bpo=14837
(10) https://bugs.python.org/issue?@action=redirect&bpo=11811
(11) https://bugs.python.org/issue?@action=redirect&bpo=13635
File: python.info, Node: stat<2>, Next: struct<3>, Prev: ssl<11>, Up: Improved Modules<11>
1.11.20.50 stat
...............
The undocumented tarfile.filemode function has been moved to *note
stat.filemode(): 113f. It can be used to convert a file’s mode to a
string of the form ‘-rwxrwxrwx’.
(Contributed by Giampaolo Rodolà in bpo-14807(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14807
File: python.info, Node: struct<3>, Next: subprocess<6>, Prev: stat<2>, Up: Improved Modules<11>
1.11.20.51 struct
.................
The *note struct: d5. module now supports ‘ssize_t’ and ‘size_t’ via the
new codes ‘n’ and ‘N’, respectively. (Contributed by Antoine Pitrou in
bpo-3163(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=3163
File: python.info, Node: subprocess<6>, Next: sys<13>, Prev: struct<3>, Up: Improved Modules<11>
1.11.20.52 subprocess
.....................
Command strings can now be bytes objects on posix platforms.
(Contributed by Victor Stinner in bpo-8513(1).)
A new constant *note DEVNULL: 1142. allows suppressing output in a
platform-independent fashion. (Contributed by Ross Lagerwall in
bpo-5870(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8513
(2) https://bugs.python.org/issue?@action=redirect&bpo=5870
File: python.info, Node: sys<13>, Next: tarfile<9>, Prev: subprocess<6>, Up: Improved Modules<11>
1.11.20.53 sys
..............
The *note sys: d9. module has a new *note thread_info: 1144. *note named
tuple: 64a. holding information about the thread implementation
(bpo-11223(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=11223
File: python.info, Node: tarfile<9>, Next: tempfile<4>, Prev: sys<13>, Up: Improved Modules<11>
1.11.20.54 tarfile
..................
*note tarfile: de. now supports ‘lzma’ encoding via the *note lzma: 8a.
module. (Contributed by Lars Gustäbel in bpo-5689(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5689
File: python.info, Node: tempfile<4>, Next: textwrap<2>, Prev: tarfile<9>, Up: Improved Modules<11>
1.11.20.55 tempfile
...................
*note tempfile.SpooledTemporaryFile: 68f.'s ‘truncate()’ method now
accepts a ‘size’ parameter. (Contributed by Ryan Kelly in bpo-9957(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9957
File: python.info, Node: textwrap<2>, Next: threading<8>, Prev: tempfile<4>, Up: Improved Modules<11>
1.11.20.56 textwrap
...................
The *note textwrap: ec. module has a new *note indent(): 278. that makes
it straightforward to add a common prefix to selected lines in a block
of text (bpo-13857(1)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13857
File: python.info, Node: threading<8>, Next: time<8>, Prev: textwrap<2>, Up: Improved Modules<11>
1.11.20.57 threading
....................
*note threading.Condition: 1149, *note threading.Semaphore: 114a, *note
threading.BoundedSemaphore: 114b, *note threading.Event: 114c, and *note
threading.Timer: 114d, all of which used to be factory functions
returning a class instance, are now classes and may be subclassed.
(Contributed by Éric Araujo in bpo-10968(1).)
The *note threading.Thread: 94c. constructor now accepts a ‘daemon’
keyword argument to override the default behavior of inheriting the
‘daemon’ flag value from the parent thread (bpo-6064(2)).
The formerly private function ‘_thread.get_ident’ is now available as
the public function *note threading.get_ident(): 114e. This eliminates
several cases of direct access to the ‘_thread’ module in the stdlib.
Third party code that used ‘_thread.get_ident’ should likewise be
changed to use the new public interface.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10968
(2) https://bugs.python.org/issue?@action=redirect&bpo=6064
File: python.info, Node: time<8>, Next: types<7>, Prev: threading<8>, Up: Improved Modules<11>
1.11.20.58 time
...............
The PEP 418(1) added new functions to the *note time: ee. module:
* *note get_clock_info(): 1150.: Get information on a clock.
* *note monotonic(): 255.: Monotonic clock (cannot go backward), not
affected by system clock updates.
* *note perf_counter(): a88.: Performance counter with the highest
available resolution to measure a short duration.
* *note process_time(): a89.: Sum of the system and user CPU time of
the current process.
Other new functions:
* *note clock_getres(): 1151, *note clock_gettime(): 1152. and *note
clock_settime(): 1153. functions with ‘CLOCK_XXX’ constants.
(Contributed by Victor Stinner in bpo-10278(2).)
To improve cross platform consistency, *note sleep(): 699. now raises a
*note ValueError: 204. when passed a negative sleep value. Previously
this was an error on posix, but produced an infinite sleep on Windows.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0418/
(2) https://bugs.python.org/issue?@action=redirect&bpo=10278
File: python.info, Node: types<7>, Next: unittest<10>, Prev: time<8>, Up: Improved Modules<11>
1.11.20.59 types
................
Add a new *note types.MappingProxyType: 469. class: Read-only proxy of a
mapping. (bpo-14386(1))
The new functions *note types.new_class(): 1155. and *note
types.prepare_class(): 1156. provide support for PEP 3115(2) compliant
dynamic type creation. (bpo-14588(3))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14386
(2) https://peps.python.org/pep-3115/
(3) https://bugs.python.org/issue?@action=redirect&bpo=14588
File: python.info, Node: unittest<10>, Next: urllib<4>, Prev: types<7>, Up: Improved Modules<11>
1.11.20.60 unittest
...................
*note assertRaises(): 528, *note assertRaisesRegex(): 52a, *note
assertWarns(): 1158, and *note assertWarnsRegex(): 1159. now accept a
keyword argument 'msg' when used as context managers. (Contributed by
Ezio Melotti and Winston Ewert in bpo-10775(1).)
*note unittest.TestCase.run(): 115a. now returns the *note TestResult:
115b. object.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10775
File: python.info, Node: urllib<4>, Next: webbrowser<3>, Prev: unittest<10>, Up: Improved Modules<11>
1.11.20.61 urllib
.................
The *note Request: fd3. class, now accepts a 'method' argument used by
*note get_method(): 115d. to determine what HTTP method should be used.
For example, this will send a ‘'HEAD'’ request:
>>> urlopen(Request('https://www.python.org', method='HEAD'))
(bpo-1673007(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1673007
File: python.info, Node: webbrowser<3>, Next: xml etree ElementTree<2>, Prev: urllib<4>, Up: Improved Modules<11>
1.11.20.62 webbrowser
.....................
The *note webbrowser: 115. module supports more “browsers”: Google
Chrome (named ‘chrome’, ‘chromium’, ‘chrome-browser’ or
‘chromium-browser’ depending on the version and operating system), and
the generic launchers ‘xdg-open’, from the FreeDesktop.org project, and
‘gvfs-open’, which is the default URI handler for GNOME 3. (The former
contributed by Arnaud Calmettes in bpo-13620(1), the latter by Matthias
Klose in bpo-14493(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=13620
(2) https://bugs.python.org/issue?@action=redirect&bpo=14493
File: python.info, Node: xml etree ElementTree<2>, Next: zlib<2>, Prev: webbrowser<3>, Up: Improved Modules<11>
1.11.20.63 xml.etree.ElementTree
................................
The *note xml.etree.ElementTree: 125. module now imports its C
accelerator by default; there is no longer a need to explicitly import
‘xml.etree.cElementTree’ (this module stays for backwards compatibility,
but is now deprecated). In addition, the ‘iter’ family of methods of
*note Element: 30a. has been optimized (rewritten in C). The module’s
documentation has also been greatly improved with added examples and a
more detailed reference.
File: python.info, Node: zlib<2>, Prev: xml etree ElementTree<2>, Up: Improved Modules<11>
1.11.20.64 zlib
...............
New attribute *note zlib.Decompress.eof: 1161. makes it possible to
distinguish between a properly formed compressed stream and an
incomplete or truncated one. (Contributed by Nadeem Vawda in
bpo-12646(1).)
New attribute *note zlib.ZLIB_RUNTIME_VERSION: 1162. reports the version
string of the underlying ‘zlib’ library that is loaded at runtime.
(Contributed by Torsten Landschoff in bpo-12306(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12646
(2) https://bugs.python.org/issue?@action=redirect&bpo=12306
File: python.info, Node: Optimizations<10>, Next: Build and C API Changes<4>, Prev: Improved Modules<11>, Up: What’s New In Python 3 3
1.11.21 Optimizations
---------------------
Major performance enhancements have been added:
* Thanks to PEP 393(1), some operations on Unicode strings have been
optimized:
* the memory footprint is divided by 2 to 4 depending on the
text
* encode an ASCII string to UTF-8 doesn’t need to encode
characters anymore, the UTF-8 representation is shared with
the ASCII representation
* the UTF-8 encoder has been optimized
* repeating a single ASCII letter and getting a substring of an
ASCII string is 4 times faster
* UTF-8 is now 2x to 4x faster. UTF-16 encoding is now up to 10x
faster.
(Contributed by Serhiy Storchaka, bpo-14624(2), bpo-14738(3) and
bpo-15026(4).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0393/
(2) https://bugs.python.org/issue?@action=redirect&bpo=14624
(3) https://bugs.python.org/issue?@action=redirect&bpo=14738
(4) https://bugs.python.org/issue?@action=redirect&bpo=15026
File: python.info, Node: Build and C API Changes<4>, Next: Deprecated<12>, Prev: Optimizations<10>, Up: What’s New In Python 3 3
1.11.22 Build and C API Changes
-------------------------------
Changes to Python’s build process and to the C API include:
* New PEP 3118(1) related function:
* *note PyMemoryView_FromMemory(): 1164.
* PEP 393(2) added new Unicode types, macros and functions:
* High-level API:
* *note PyUnicode_CopyCharacters(): 8b4.
* *note PyUnicode_FindChar(): 8b8.
* *note PyUnicode_GetLength(): 8b2, *note
PyUnicode_GET_LENGTH: 8b3.
* *note PyUnicode_New(): 8aa.
* *note PyUnicode_Substring(): 8b5.
* *note PyUnicode_ReadChar(): 1165, *note
PyUnicode_WriteChar(): 1166.
* Low-level API:
* *note Py_UCS1: 1167, *note Py_UCS2: 1168, *note Py_UCS4:
29d. types
* *note PyASCIIObject: 1169. and *note
PyCompactUnicodeObject: 116a. structures
* *note PyUnicode_READY: 3fd.
* *note PyUnicode_FromKindAndData(): 116b.
* *note PyUnicode_AsUCS4(): 116c, *note
PyUnicode_AsUCS4Copy(): 8ba.
* *note PyUnicode_DATA: 116d, *note PyUnicode_1BYTE_DATA:
116e, *note PyUnicode_2BYTE_DATA: 116f, *note
PyUnicode_4BYTE_DATA: 1170.
* *note PyUnicode_KIND: 1171. with ‘PyUnicode_Kind’ enum:
‘PyUnicode_WCHAR_KIND’, *note PyUnicode_1BYTE_KIND: 1172,
*note PyUnicode_2BYTE_KIND: 1173, *note
PyUnicode_4BYTE_KIND: 1174.
* *note PyUnicode_READ: 1175, *note PyUnicode_READ_CHAR:
1176, *note PyUnicode_WRITE: 1177.
* *note PyUnicode_MAX_CHAR_VALUE: 1178.
* *note PyArg_ParseTuple: 56e. now accepts a *note bytearray: 53a.
for the ‘c’ format (bpo-12380(3)).
---------- Footnotes ----------
(1) https://peps.python.org/pep-3118/
(2) https://peps.python.org/pep-0393/
(3) https://bugs.python.org/issue?@action=redirect&bpo=12380
File: python.info, Node: Deprecated<12>, Next: Porting to Python 3 3, Prev: Build and C API Changes<4>, Up: What’s New In Python 3 3
1.11.23 Deprecated
------------------
* Menu:
* Unsupported Operating Systems: Unsupported Operating Systems<2>.
* Deprecated Python modules, functions and methods: Deprecated Python modules functions and methods<4>.
* Deprecated functions and types of the C API: Deprecated functions and types of the C API<3>.
* Deprecated features::
File: python.info, Node: Unsupported Operating Systems<2>, Next: Deprecated Python modules functions and methods<4>, Up: Deprecated<12>
1.11.23.1 Unsupported Operating Systems
.......................................
OS/2 and VMS are no longer supported due to the lack of a maintainer.
Windows 2000 and Windows platforms which set ‘COMSPEC’ to ‘command.com’
are no longer supported due to maintenance burden.
OSF support, which was deprecated in 3.2, has been completely removed.
File: python.info, Node: Deprecated Python modules functions and methods<4>, Next: Deprecated functions and types of the C API<3>, Prev: Unsupported Operating Systems<2>, Up: Deprecated<12>
1.11.23.2 Deprecated Python modules, functions and methods
..........................................................
* Passing a non-empty string to ‘object.__format__()’ is deprecated,
and will produce a *note TypeError: 534. in Python 3.4
(bpo-9856(1)).
* The ‘unicode_internal’ codec has been deprecated because of the PEP
393(2), use UTF-8, UTF-16 (‘utf-16-le’ or ‘utf-16-be’), or UTF-32
(‘utf-32-le’ or ‘utf-32-be’)
* *note ftplib.FTP.nlst(): 10b9. and *note ftplib.FTP.dir(): 10ba.:
use *note ftplib.FTP.mlsd(): 10b8.
* ‘platform.popen()’: use the *note subprocess: d6. module. Check
especially the *note Replacing Older Functions with the subprocess
Module: 117c. section (bpo-11377(3)).
* bpo-13374(4): The Windows bytes API has been deprecated in the
*note os: a1. module. Use Unicode filenames, instead of bytes
filenames, to not depend on the ANSI code page anymore and to
support any filename.
* bpo-13988(5): The ‘xml.etree.cElementTree’ module is deprecated.
The accelerator is used automatically whenever available.
* The behaviour of ‘time.clock()’ depends on the platform: use the
new *note time.perf_counter(): a88. or *note time.process_time():
a89. function instead, depending on your requirements, to have a
well defined behaviour.
* The ‘os.stat_float_times()’ function is deprecated.
* *note abc: 4. module:
* *note abc.abstractproperty: 107e. has been deprecated, use
*note property: 194. with *note abc.abstractmethod(): 107f.
instead.
* *note abc.abstractclassmethod: 1080. has been deprecated, use
*note classmethod: 166. with *note abc.abstractmethod(): 107f.
instead.
* *note abc.abstractstaticmethod: 1081. has been deprecated, use
*note staticmethod: 412. with *note abc.abstractmethod():
107f. instead.
* *note importlib: 77. package:
* *note importlib.abc.SourceLoader.path_mtime(): 117d. is now
deprecated in favour of *note
importlib.abc.SourceLoader.path_stats(): 117e. as bytecode
files now store both the modification time and size of the
source file the bytecode file was compiled from.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9856
(2) https://peps.python.org/pep-0393/
(3) https://bugs.python.org/issue?@action=redirect&bpo=11377
(4) https://bugs.python.org/issue?@action=redirect&bpo=13374
(5) https://bugs.python.org/issue?@action=redirect&bpo=13988
File: python.info, Node: Deprecated functions and types of the C API<3>, Next: Deprecated features, Prev: Deprecated Python modules functions and methods<4>, Up: Deprecated<12>
1.11.23.3 Deprecated functions and types of the C API
.....................................................
The *note Py_UNICODE: 3e9. has been deprecated by PEP 393(1) and will be
removed in Python 4. All functions using this type are deprecated:
Unicode functions and methods using *note Py_UNICODE: 3e9. and *note
Py_UNICODE: 3e9.* types:
* ‘PyUnicode_FromUnicode’: use *note PyUnicode_FromWideChar(): 1180.
or *note PyUnicode_FromKindAndData(): 116b.
* ‘PyUnicode_AS_UNICODE’, ‘PyUnicode_AsUnicode()’,
‘PyUnicode_AsUnicodeAndSize()’: use *note
PyUnicode_AsWideCharString(): 8bb.
* ‘PyUnicode_AS_DATA’: use *note PyUnicode_DATA: 116d. with *note
PyUnicode_READ: 1175. and *note PyUnicode_WRITE: 1177.
* ‘PyUnicode_GET_SIZE’, ‘PyUnicode_GetSize()’: use *note
PyUnicode_GET_LENGTH: 8b3. or *note PyUnicode_GetLength(): 8b2.
* ‘PyUnicode_GET_DATA_SIZE’: use ‘PyUnicode_GET_LENGTH(str) *
PyUnicode_KIND(str)’ (only work on ready strings)
* ‘PyUnicode_AsUnicodeCopy()’: use *note PyUnicode_AsUCS4Copy(): 8ba.
or *note PyUnicode_AsWideCharString(): 8bb.
* ‘PyUnicode_GetMax()’
Functions and macros manipulating Py_UNICODE* strings:
* ‘Py_UNICODE_strlen()’: use *note PyUnicode_GetLength(): 8b2. or
*note PyUnicode_GET_LENGTH: 8b3.
* ‘Py_UNICODE_strcat()’: use *note PyUnicode_CopyCharacters(): 8b4.
or *note PyUnicode_FromFormat(): 385.
* ‘Py_UNICODE_strcpy()’, ‘Py_UNICODE_strncpy()’, ‘Py_UNICODE_COPY()’:
use *note PyUnicode_CopyCharacters(): 8b4. or *note
PyUnicode_Substring(): 8b5.
* ‘Py_UNICODE_strcmp()’: use *note PyUnicode_Compare(): 8b6.
* ‘Py_UNICODE_strncmp()’: use *note PyUnicode_Tailmatch(): 8b7.
* ‘Py_UNICODE_strchr()’, ‘Py_UNICODE_strrchr()’: use *note
PyUnicode_FindChar(): 8b8.
* ‘Py_UNICODE_FILL()’: use *note PyUnicode_Fill(): 1181.
* ‘Py_UNICODE_MATCH’
Encoders:
* ‘PyUnicode_Encode()’: use ‘PyUnicode_AsEncodedObject()’
* ‘PyUnicode_EncodeUTF7()’
* ‘PyUnicode_EncodeUTF8()’: use *note PyUnicode_AsUTF8(): bbb. or
*note PyUnicode_AsUTF8String(): 1182.
* ‘PyUnicode_EncodeUTF32()’
* ‘PyUnicode_EncodeUTF16()’
* ‘PyUnicode_EncodeUnicodeEscape()’ use *note
PyUnicode_AsUnicodeEscapeString(): 1183.
* ‘PyUnicode_EncodeRawUnicodeEscape()’ use *note
PyUnicode_AsRawUnicodeEscapeString(): 1184.
* ‘PyUnicode_EncodeLatin1()’: use *note PyUnicode_AsLatin1String():
1185.
* ‘PyUnicode_EncodeASCII()’: use *note PyUnicode_AsASCIIString():
1186.
* ‘PyUnicode_EncodeCharmap()’
* ‘PyUnicode_TranslateCharmap()’
* ‘PyUnicode_EncodeMBCS()’: use *note PyUnicode_AsMBCSString(): 1187.
or *note PyUnicode_EncodeCodePage(): 1188. (with ‘CP_ACP’
code_page)
* ‘PyUnicode_EncodeDecimal()’, ‘PyUnicode_TransformDecimalToASCII()’
---------- Footnotes ----------
(1) https://peps.python.org/pep-0393/
File: python.info, Node: Deprecated features, Prev: Deprecated functions and types of the C API<3>, Up: Deprecated<12>
1.11.23.4 Deprecated features
.............................
The *note array: 7. module’s ‘'u'’ format code is now deprecated and
will be removed in Python 4 together with the rest of the (*note
Py_UNICODE: 3e9.) API.
File: python.info, Node: Porting to Python 3 3, Prev: Deprecated<12>, Up: What’s New In Python 3 3
1.11.24 Porting to Python 3.3
-----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code.
* Menu:
* Porting Python code::
* Porting C code::
* Building C extensions::
* Command Line Switch Changes::
File: python.info, Node: Porting Python code, Next: Porting C code, Up: Porting to Python 3 3
1.11.24.1 Porting Python code
.............................
* Hash randomization is enabled by default. Set the *note
PYTHONHASHSEED: 1076. environment variable to ‘0’ to disable hash
randomization. See also the *note object.__hash__(): afb. method.
* bpo-12326(1): On Linux, sys.platform doesn’t contain the major
version anymore. It is now always ‘linux’, instead of ‘linux2’ or
‘linux3’ depending on the Linux version used to build Python.
Replace sys.platform == ‘linux2’ with
sys.platform.startswith(‘linux’), or directly sys.platform ==
‘linux’ if you don’t need to support older Python versions.
* bpo-13847(2), bpo-14180(3): *note time: ee. and *note datetime:
30.: *note OverflowError: 87f. is now raised instead of *note
ValueError: 204. if a timestamp is out of range. *note OSError:
413. is now raised if C functions ‘gmtime()’ or ‘localtime()’
failed.
* The default finders used by import now utilize a cache of what is
contained within a specific directory. If you create a Python
source file or sourceless bytecode file, make sure to call *note
importlib.invalidate_caches(): c10. to clear out the cache for the
finders to notice the new file.
* *note ImportError: 415. now uses the full name of the module that
was attempted to be imported. Doctests that check ImportErrors’
message will need to be updated to use the full name of the module
instead of just the tail of the name.
* The 'index' argument to *note __import__(): 8d3. now defaults to 0
instead of -1 and no longer support negative values. It was an
oversight when PEP 328(4) was implemented that the default value
remained -1. If you need to continue to perform a relative import
followed by an absolute import, then perform the relative import
using an index of 1, followed by another import using an index of
0. It is preferred, though, that you use *note
importlib.import_module(): 513. rather than call *note
__import__(): 8d3. directly.
* *note __import__(): 8d3. no longer allows one to use an index value
other than 0 for top-level modules. E.g. ‘__import__('sys',
level=1)’ is now an error.
* Because *note sys.meta_path: d2b. and *note sys.path_hooks: 101d.
now have finders on them by default, you will most likely want to
use ‘list.insert()’ instead of ‘list.append()’ to add to those
lists.
* Because ‘None’ is now inserted into *note sys.path_importer_cache:
5e0, if you are clearing out entries in the dictionary of paths
that do not have a finder, you will need to remove keys paired with
values of ‘None’ 'and' ‘imp.NullImporter’ to be
backwards-compatible. This will lead to extra overhead on older
versions of Python that re-insert ‘None’ into *note
sys.path_importer_cache: 5e0. where it represents the use of
implicit finders, but semantically it should not change anything.
* ‘importlib.abc.Finder’ no longer specifies a ‘find_module()’
abstract method that must be implemented. If you were relying on
subclasses to implement that method, make sure to check for the
method’s existence first. You will probably want to check for
‘find_loader()’ first, though, in the case of working with *note
path entry finders: 106a.
* *note pkgutil: a9. has been converted to use *note importlib: 77.
internally. This eliminates many edge cases where the old
behaviour of the PEP 302(5) import emulation failed to match the
behaviour of the real import system. The import emulation itself
is still present, but is now deprecated. The *note
pkgutil.iter_importers(): 118c. and *note pkgutil.walk_packages():
bff. functions special case the standard import hooks so they are
still supported even though they do not provide the non-standard
‘iter_modules()’ method.
* A longstanding RFC-compliance bug (bpo-1079(6)) in the parsing done
by *note email.header.decode_header(): 10ae. has been fixed. Code
that uses the standard idiom to convert encoded headers into
unicode (‘str(make_header(decode_header(h))’) will see no change,
but code that looks at the individual tuples returned by
decode_header will see that whitespace that precedes or follows
‘ASCII’ sections is now included in the ‘ASCII’ section. Code that
builds headers using ‘make_header’ should also continue to work
without change, since ‘make_header’ continues to add whitespace
between ‘ASCII’ and non-‘ASCII’ sections if it is not already
present in the input strings.
* *note email.utils.formataddr(): 118d. now does the correct content
transfer encoding when passed non-‘ASCII’ display names. Any code
that depended on the previous buggy behavior that preserved the
non-‘ASCII’ unicode in the formatted output string will need to be
changed (bpo-1690608(7)).
* *note poplib.POP3.quit(): 118e. may now raise protocol errors like
all other ‘poplib’ methods. Code that assumes ‘quit’ does not
raise *note poplib.error_proto: 118f. errors may need to be changed
if errors on ‘quit’ are encountered by a particular application
(bpo-11291(8)).
* The ‘strict’ argument to *note email.parser.Parser: 1190,
deprecated since Python 2.4, has finally been removed.
* The deprecated method ‘unittest.TestCase.assertSameElements’ has
been removed.
* The deprecated variable ‘time.accept2dyear’ has been removed.
* The deprecated ‘Context._clamp’ attribute has been removed from the
*note decimal: 36. module. It was previously replaced by the
public attribute *note clamp: 1191. (See bpo-8540(9).)
* The undocumented internal helper class ‘SSLFakeFile’ has been
removed from *note smtplib: ca, since its functionality has long
been provided directly by *note socket.socket.makefile(): 1192.
* Passing a negative value to *note time.sleep(): 699. on Windows now
raises an error instead of sleeping forever. It has always raised
an error on posix.
* The ‘ast.__version__’ constant has been removed. If you need to
make decisions affected by the AST version, use *note
sys.version_info: 69c. to make the decision.
* Code that used to work around the fact that the *note threading:
ed. module used factory functions by subclassing the private
classes will need to change to subclass the now-public classes.
* The undocumented debugging machinery in the threading module has
been removed, simplifying the code. This should have no effect on
production code, but is mentioned here in case any application
debug frameworks were interacting with it (bpo-13550(10)).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=12326
(2) https://bugs.python.org/issue?@action=redirect&bpo=13847
(3) https://bugs.python.org/issue?@action=redirect&bpo=14180
(4) https://peps.python.org/pep-0328/
(5) https://peps.python.org/pep-0302/
(6) https://bugs.python.org/issue?@action=redirect&bpo=1079
(7) https://bugs.python.org/issue?@action=redirect&bpo=1690608
(8) https://bugs.python.org/issue?@action=redirect&bpo=11291
(9) https://bugs.python.org/issue?@action=redirect&bpo=8540
(10) https://bugs.python.org/issue?@action=redirect&bpo=13550
File: python.info, Node: Porting C code, Next: Building C extensions, Prev: Porting Python code, Up: Porting to Python 3 3
1.11.24.2 Porting C code
........................
* In the course of changes to the buffer API the undocumented
‘smalltable’ member of the *note Py_buffer: 753. structure has been
removed and the layout of the ‘PyMemoryViewObject’ has changed.
All extensions relying on the relevant parts in ‘memoryobject.h’ or
‘object.h’ must be rebuilt.
* Due to *note PEP 393: 1036, the *note Py_UNICODE: 3e9. type and all
functions using this type are deprecated (but will stay available
for at least five years). If you were using low-level Unicode APIs
to construct and access unicode objects and you want to benefit of
the memory footprint reduction provided by PEP 393(1), you have to
convert your code to the new *note Unicode API: 1193.
However, if you only have been using high-level functions such as
*note PyUnicode_Concat(): 1194, *note PyUnicode_Join(): 1195. or
*note PyUnicode_FromFormat(): 385, your code will automatically
take advantage of the new unicode representations.
* *note PyImport_GetMagicNumber(): 1196. now returns ‘-1’ upon
failure.
* As a negative value for the 'level' argument to *note __import__():
8d3. is no longer valid, the same now holds for *note
PyImport_ImportModuleLevel(): 1197. This also means that the value
of 'level' used by *note PyImport_ImportModuleEx(): 1198. is now
‘0’ instead of ‘-1’.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0393/
File: python.info, Node: Building C extensions, Next: Command Line Switch Changes, Prev: Porting C code, Up: Porting to Python 3 3
1.11.24.3 Building C extensions
...............................
* The range of possible file names for C extensions has been
narrowed. Very rarely used spellings have been suppressed: under
POSIX, files named ‘xxxmodule.so’, ‘xxxmodule.abi3.so’ and
‘xxxmodule.cpython-*.so’ are no longer recognized as implementing
the ‘xxx’ module. If you had been generating such files, you have
to switch to the other spellings (i.e., remove the ‘module’ string
from the file names).
(implemented in bpo-14040(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=14040
File: python.info, Node: Command Line Switch Changes, Prev: Building C extensions, Up: Porting to Python 3 3
1.11.24.4 Command Line Switch Changes
.....................................
* The -Q command-line flag and related artifacts have been removed.
Code checking sys.flags.division_warning will need updating.
(bpo-10998(1), contributed by Éric Araujo.)
* When ‘python’ is started with *note -S: 119b, ‘import site’ will no
longer add site-specific paths to the module search paths. In
previous versions, it did.
(bpo-11591(2), contributed by Carl Meyer with editions by Éric
Araujo.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10998
(2) https://bugs.python.org/issue?@action=redirect&bpo=11591
File: python.info, Node: What’s New In Python 3 2, Next: What’s New In Python 3 1, Prev: What’s New In Python 3 3, Up: What’s New in Python
1.12 What’s New In Python 3.2
=============================
Author: Raymond Hettinger
This article explains the new features in Python 3.2 as compared to 3.1.
Python 3.2 was released on February 20, 2011. It focuses on a few
highlights and gives a few examples. For full details, see the
Misc/NEWS(1) file.
See also
........
PEP 392(2) - Python 3.2 Release Schedule
* Menu:
* PEP 384; Defining a Stable ABI: PEP 384 Defining a Stable ABI.
* PEP 389; Argparse Command Line Parsing Module: PEP 389 Argparse Command Line Parsing Module.
* PEP 391; Dictionary Based Configuration for Logging: PEP 391 Dictionary Based Configuration for Logging.
* PEP 3148; The concurrent.futures module: PEP 3148 The concurrent futures module.
* PEP 3147; PYC Repository Directories: PEP 3147 PYC Repository Directories.
* PEP 3149; ABI Version Tagged .so Files: PEP 3149 ABI Version Tagged so Files.
* PEP 3333; Python Web Server Gateway Interface v1.0.1: PEP 3333 Python Web Server Gateway Interface v1 0 1.
* Other Language Changes: Other Language Changes<12>.
* New, Improved, and Deprecated Modules: New Improved and Deprecated Modules.
* Multi-threading::
* Optimizations: Optimizations<11>.
* Unicode::
* Codecs::
* Documentation::
* IDLE::
* Code Repository::
* Build and C API Changes: Build and C API Changes<5>.
* Porting to Python 3.2: Porting to Python 3 2.
---------- Footnotes ----------
(1)
https://github.com/python/cpython/blob/076ca6c3c8df3030307e548d9be792ce3c1c6eea/Misc/NEWS
(2) https://peps.python.org/pep-0392/
File: python.info, Node: PEP 384 Defining a Stable ABI, Next: PEP 389 Argparse Command Line Parsing Module, Up: What’s New In Python 3 2
1.12.1 PEP 384: Defining a Stable ABI
-------------------------------------
In the past, extension modules built for one Python version were often
not usable with other Python versions. Particularly on Windows, every
feature release of Python required rebuilding all extension modules that
one wanted to use. This requirement was the result of the free access
to Python interpreter internals that extension modules could use.
With Python 3.2, an alternative approach becomes available: extension
modules which restrict themselves to a limited API (by defining
Py_LIMITED_API) cannot use many of the internals, but are constrained to
a set of API functions that are promised to be stable for several
releases. As a consequence, extension modules built for 3.2 in that
mode will also work with 3.3, 3.4, and so on. Extension modules that
make use of details of memory structures can still be built, but will
need to be recompiled for every feature release.
See also
........
PEP 384(1) - Defining a Stable ABI
PEP written by Martin von Löwis.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0384/
File: python.info, Node: PEP 389 Argparse Command Line Parsing Module, Next: PEP 391 Dictionary Based Configuration for Logging, Prev: PEP 384 Defining a Stable ABI, Up: What’s New In Python 3 2
1.12.2 PEP 389: Argparse Command Line Parsing Module
----------------------------------------------------
A new module for command line parsing, *note argparse: 6, was introduced
to overcome the limitations of *note optparse: a0. which did not provide
support for positional arguments (not just options), subcommands,
required options and other common patterns of specifying and validating
options.
This module has already had widespread success in the community as a
third-party module. Being more fully featured than its predecessor, the
*note argparse: 6. module is now the preferred module for command-line
processing. The older module is still being kept available because of
the substantial amount of legacy code that depends on it.
Here’s an annotated example parser showing features like limiting
results to a set of choices, specifying a 'metavar' in the help screen,
validating that one or more positional arguments is present, and making
a required option:
import argparse
parser = argparse.ArgumentParser(
description = 'Manage servers', # main description for help
epilog = 'Tested on Solaris and Linux') # displayed after help
parser.add_argument('action', # argument name
choices = ['deploy', 'start', 'stop'], # three allowed values
help = 'action on each target') # help msg
parser.add_argument('targets',
metavar = 'HOSTNAME', # var name used in help msg
nargs = '+', # require one or more targets
help = 'url for target machines') # help msg explanation
parser.add_argument('-u', '--user', # -u or --user option
required = True, # make it a required argument
help = 'login as user')
Example of calling the parser on a command string:
>>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
>>> result = parser.parse_args(cmd.split())
>>> result.action
'deploy'
>>> result.targets
['sneezy.example.com', 'sleepy.example.com']
>>> result.user
'skycaptain'
Example of the parser’s automatically generated help:
>>> parser.parse_args('-h'.split())
usage: manage_cloud.py [-h] -u USER
{deploy,start,stop} HOSTNAME [HOSTNAME ...]
Manage servers
positional arguments:
{deploy,start,stop} action on each target
HOSTNAME url for target machines
optional arguments:
-h, --help show this help message and exit
-u USER, --user USER login as user
Tested on Solaris and Linux
An especially nice *note argparse: 6. feature is the ability to define
subparsers, each with their own argument patterns and help displays:
import argparse
parser = argparse.ArgumentParser(prog='HELM')
subparsers = parser.add_subparsers()
parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup
parser_l.add_argument('-m', '--missiles', action='store_true')
parser_l.add_argument('-t', '--torpedos', action='store_true')
parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup
aliases=('steer', 'turn')) # equivalent names
parser_m.add_argument('-c', '--course', type=int, required=True)
parser_m.add_argument('-s', '--speed', type=int, default=0)
$ ./helm.py --help # top level help (launch and move)
$ ./helm.py launch --help # help for launch options
$ ./helm.py launch --missiles # set missiles=True and torpedos=False
$ ./helm.py steer --course 180 --speed 5 # set movement parameters
See also
........
PEP 389(1) - New Command Line Parsing Module
PEP written by Steven Bethard.
*note Migrating optparse code to argparse: 11a0. for details on the
differences from *note optparse: a0.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0389/
File: python.info, Node: PEP 391 Dictionary Based Configuration for Logging, Next: PEP 3148 The concurrent futures module, Prev: PEP 389 Argparse Command Line Parsing Module, Up: What’s New In Python 3 2
1.12.3 PEP 391: Dictionary Based Configuration for Logging
----------------------------------------------------------
The *note logging: 87. module provided two kinds of configuration, one
style with function calls for each option or another style driven by an
external file saved in a *note configparser: 22. format. Those options
did not provide the flexibility to create configurations from JSON or
YAML files, nor did they support incremental configuration, which is
needed for specifying logger options from a command line.
To support a more flexible style, the module now offers *note
logging.config.dictConfig(): 11a2. for specifying logging configuration
with plain Python dictionaries. The configuration options include
formatters, handlers, filters, and loggers. Here’s a working example of
a configuration dictionary:
{"version": 1,
"formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
"full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}
},
"handlers": {"console": {
"class": "logging.StreamHandler",
"formatter": "brief",
"level": "INFO",
"stream": "ext://sys.stdout"},
"console_priority": {
"class": "logging.StreamHandler",
"formatter": "full",
"level": "ERROR",
"stream": "ext://sys.stderr"}
},
"root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}
If that dictionary is stored in a file called ‘conf.json’, it can be
loaded and called with code like this:
>>> import json, logging.config
>>> with open('conf.json') as f:
... conf = json.load(f)
...
>>> logging.config.dictConfig(conf)
>>> logging.info("Transaction completed normally")
INFO : root : Transaction completed normally
>>> logging.critical("Abnormal termination")
2011-02-17 11:14:36,694 root CRITICAL Abnormal termination
See also
........
PEP 391(1) - Dictionary Based Configuration for Logging
PEP written by Vinay Sajip.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0391/
File: python.info, Node: PEP 3148 The concurrent futures module, Next: PEP 3147 PYC Repository Directories, Prev: PEP 391 Dictionary Based Configuration for Logging, Up: What’s New In Python 3 2
1.12.4 PEP 3148: The ‘concurrent.futures’ module
------------------------------------------------
Code for creating and managing concurrency is being collected in a new
top-level namespace, 'concurrent'. Its first member is a 'futures'
package which provides a uniform high-level interface for managing
threads and processes.
The design for *note concurrent.futures: 21. was inspired by the
'java.util.concurrent' package. In that model, a running call and its
result are represented by a *note Future: 11a4. object that abstracts
features common to threads, processes, and remote procedure calls. That
object supports status checks (running or done), timeouts,
cancellations, adding callbacks, and access to results or exceptions.
The primary offering of the new module is a pair of executor classes for
launching and managing calls. The goal of the executors is to make it
easier to use existing tools for making parallel calls. They save the
effort needed to setup a pool of resources, launch the calls, create a
results queue, add time-out handling, and limit the total number of
threads, processes, or remote procedure calls.
Ideally, each application should share a single executor across multiple
components so that process and thread limits can be centrally managed.
This solves the design challenge that arises when each component has its
own competing strategy for resource management.
Both classes share a common interface with three methods: *note
submit(): a82. for scheduling a callable and returning a *note Future:
11a4. object; *note map(): de3. for scheduling many asynchronous calls
at a time, and *note shutdown(): 8ec. for freeing resources. The class
is a *note context manager: 5d0. and can be used in a *note with: 5ce.
statement to assure that resources are automatically released when
currently pending futures are done executing.
A simple of example of *note ThreadPoolExecutor: 73d. is a launch of
four parallel threads for copying files:
import concurrent.futures, shutil
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
e.submit(shutil.copy, 'src3.txt', 'dest4.txt')
See also
........
PEP 3148(1) - Futures – Execute Computations Asynchronously
PEP written by Brian Quinlan.
*note Code for Threaded Parallel URL reads: 11a5, an example using
threads to fetch multiple web pages in parallel.
*note Code for computing prime numbers in parallel: 11a6, an example
demonstrating *note ProcessPoolExecutor: 8ed.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3148/
File: python.info, Node: PEP 3147 PYC Repository Directories, Next: PEP 3149 ABI Version Tagged so Files, Prev: PEP 3148 The concurrent futures module, Up: What’s New In Python 3 2
1.12.5 PEP 3147: PYC Repository Directories
-------------------------------------------
Python’s scheme for caching bytecode in '.pyc' files did not work well
in environments with multiple Python interpreters. If one interpreter
encountered a cached file created by another interpreter, it would
recompile the source and overwrite the cached file, thus losing the
benefits of caching.
The issue of “pyc fights” has become more pronounced as it has become
commonplace for Linux distributions to ship with multiple versions of
Python. These conflicts also arise with CPython alternatives such as
Unladen Swallow.
To solve this problem, Python’s import machinery has been extended to
use distinct filenames for each interpreter. Instead of Python 3.2 and
Python 3.3 and Unladen Swallow each competing for a file called
“mymodule.pyc”, they will now look for “mymodule.cpython-32.pyc”,
“mymodule.cpython-33.pyc”, and “mymodule.unladen10.pyc”. And to prevent
all of these new files from cluttering source directories, the 'pyc'
files are now collected in a “__pycache__” directory stored under the
package directory.
Aside from the filenames and target directories, the new scheme has a
few aspects that are visible to the programmer:
* Imported modules now have a *note __cached__: 2d9. attribute which
stores the name of the actual file that was imported:
>>> import collections
>>> collections.__cached__
'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
* The tag that is unique to each interpreter is accessible from the
‘imp’ module:
>>> import imp
>>> imp.get_tag()
'cpython-32'
* Scripts that try to deduce source filename from the imported file
now need to be smarter. It is no longer sufficient to simply strip
the “c” from a “.pyc” filename. Instead, use the new functions in
the ‘imp’ module:
>>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc')
'c:/py32/lib/collections.py'
>>> imp.cache_from_source('c:/py32/lib/collections.py')
'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
* The *note py_compile: b3. and *note compileall: 20. modules have
been updated to reflect the new naming convention and target
directory. The command-line invocation of 'compileall' has new
options: ‘-i’ for specifying a list of files and directories to
compile and ‘-b’ which causes bytecode files to be written to their
legacy location rather than '__pycache__'.
* The *note importlib.abc: 78. module has been updated with new *note
abstract base classes: 11a8. for loading bytecode files. The
obsolete ABCs, ‘PyLoader’ and ‘PyPycLoader’, have been deprecated
(instructions on how to stay Python 3.1 compatible are included
with the documentation).
See also
........
PEP 3147(1) - PYC Repository Directories
PEP written by Barry Warsaw.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3147/
File: python.info, Node: PEP 3149 ABI Version Tagged so Files, Next: PEP 3333 Python Web Server Gateway Interface v1 0 1, Prev: PEP 3147 PYC Repository Directories, Up: What’s New In Python 3 2
1.12.6 PEP 3149: ABI Version Tagged .so Files
---------------------------------------------
The PYC repository directory allows multiple bytecode cache files to be
co-located. This PEP implements a similar mechanism for shared object
files by giving them a common directory and distinct names for each
version.
The common directory is “pyshared” and the file names are made distinct
by identifying the Python implementation (such as CPython, PyPy, Jython,
etc.), the major and minor version numbers, and optional build flags
(such as “d” for debug, “m” for pymalloc, “u” for wide-unicode). For an
arbitrary package “foo”, you may see these files when the distribution
package is installed:
/usr/share/pyshared/foo.cpython-32m.so
/usr/share/pyshared/foo.cpython-33md.so
In Python itself, the tags are accessible from functions in the *note
sysconfig: db. module:
>>> import sysconfig
>>> sysconfig.get_config_var('SOABI') # find the version tag
'cpython-32mu'
>>> sysconfig.get_config_var('EXT_SUFFIX') # find the full filename extension
'.cpython-32mu.so'
See also
........
PEP 3149(1) - ABI Version Tagged .so Files
PEP written by Barry Warsaw.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3149/
File: python.info, Node: PEP 3333 Python Web Server Gateway Interface v1 0 1, Next: Other Language Changes<12>, Prev: PEP 3149 ABI Version Tagged so Files, Up: What’s New In Python 3 2
1.12.7 PEP 3333: Python Web Server Gateway Interface v1.0.1
-----------------------------------------------------------
This informational PEP clarifies how bytes/text issues are to be handled
by the WSGI protocol. The challenge is that string handling in Python 3
is most conveniently handled with the *note str: 447. type even though
the HTTP protocol is itself bytes oriented.
The PEP differentiates so-called 'native strings' that are used for
request/response headers and metadata versus 'byte strings' which are
used for the bodies of requests and responses.
The 'native strings' are always of type *note str: 447. but are
restricted to code points between 'U+0000' through 'U+00FF' which are
translatable to bytes using 'Latin-1' encoding. These strings are used
for the keys and values in the environment dictionary and for response
headers and statuses in the ‘start_response()’ function. They must
follow RFC 2616(1) with respect to encoding. That is, they must either
be 'ISO-8859-1' characters or use RFC 2047(2) MIME encoding.
For developers porting WSGI applications from Python 2, here are the
salient points:
* If the app already used strings for headers in Python 2, no change
is needed.
* If instead, the app encoded output headers or decoded input
headers, then the headers will need to be re-encoded to Latin-1.
For example, an output header encoded in utf-8 was using
‘h.encode('utf-8')’ now needs to convert from bytes to native
strings using ‘h.encode('utf-8').decode('latin-1')’.
* Values yielded by an application or sent using the ‘write()’ method
must be byte strings. The ‘start_response()’ function and environ
must use native strings. The two cannot be mixed.
For server implementers writing CGI-to-WSGI pathways or other CGI-style
protocols, the users must to be able access the environment using native
strings even though the underlying platform may have a different
convention. To bridge this gap, the *note wsgiref: 118. module has a
new function, *note wsgiref.handlers.read_environ(): 11ab. for
transcoding CGI variables from *note os.environ: 11ac. into native
strings and returning a new dictionary.
See also
........
PEP 3333(3) - Python Web Server Gateway Interface v1.0.1
PEP written by Phillip Eby.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2616.html
(2) https://datatracker.ietf.org/doc/html/rfc2047.html
(3) https://peps.python.org/pep-3333/
File: python.info, Node: Other Language Changes<12>, Next: New Improved and Deprecated Modules, Prev: PEP 3333 Python Web Server Gateway Interface v1 0 1, Up: What’s New In Python 3 2
1.12.8 Other Language Changes
-----------------------------
Some smaller changes made to the core Python language are:
* String formatting for *note format(): 61b. and *note str.format():
61d. gained new capabilities for the format character '#'.
Previously, for integers in binary, octal, or hexadecimal, it
caused the output to be prefixed with ‘0b’, ‘0o’, or ‘0x’
respectively. Now it can also handle floats, complex, and Decimal,
causing the output to always have a decimal point even when no
digits follow it.
>>> format(20, '#o')
'0o24'
>>> format(12.34, '#5.0f')
' 12.'
(Suggested by Mark Dickinson and implemented by Eric Smith in
bpo-7094(1).)
* There is also a new *note str.format_map(): dd6. method that
extends the capabilities of the existing *note str.format(): 61d.
method by accepting arbitrary *note mapping: 11ae. objects. This
new method makes it possible to use string formatting with any of
Python’s many dictionary-like objects such as *note defaultdict:
11af, *note Shelf: f95, *note ConfigParser: 1c8, or *note dbm: 31.
It is also useful with custom *note dict: 258. subclasses that
normalize keys before look-up or that supply a ‘__missing__()’
method for unknown keys:
>>> import shelve
>>> d = shelve.open('tmp.shl')
>>> 'The {project_name} status is {status} as of {date}'.format_map(d)
'The testing project status is green as of February 15, 2011'
>>> class LowerCasedDict(dict):
... def __getitem__(self, key):
... return dict.__getitem__(self, key.lower())
...
>>> lcd = LowerCasedDict(part='widgets', quantity=10)
>>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd)
'There are 10 widgets in stock'
>>> class PlaceholderDict(dict):
... def __missing__(self, key):
... return '<{}>'.format(key)
...
>>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict())
'Hello , welcome to '
(Suggested by Raymond Hettinger and implemented by Eric Smith in
bpo-6081(2).)
* The interpreter can now be started with a quiet option, ‘-q’, to
prevent the copyright and version information from being displayed
in the interactive mode. The option can be introspected using the
*note sys.flags: 688. attribute:
$ python -q
>>> sys.flags
sys.flags(debug=0, division_warning=0, inspect=0, interactive=0,
optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
ignore_environment=0, verbose=0, bytes_warning=0, quiet=1)
(Contributed by Marcin Wojdyr in bpo-1772833(3)).
* The *note hasattr(): 4ce. function works by calling *note
getattr(): bd1. and detecting whether an exception is raised. This
technique allows it to detect methods created dynamically by *note
__getattr__(): 4cf. or *note __getattribute__(): bd2. which would
otherwise be absent from the class dictionary. Formerly, 'hasattr'
would catch any exception, possibly masking genuine errors. Now,
'hasattr' has been tightened to only catch *note AttributeError:
348. and let other exceptions pass through:
>>> class A:
... @property
... def f(self):
... return 1 // 0
...
>>> a = A()
>>> hasattr(a, 'f')
Traceback (most recent call last):
...
ZeroDivisionError: integer division or modulo by zero
(Discovered by Yury Selivanov and fixed by Benjamin Peterson;
bpo-9666(4).)
* The *note str(): 447. of a float or complex number is now the same
as its *note repr(): 7f9. Previously, the *note str(): 447. form
was shorter but that just caused confusion and is no longer needed
now that the shortest possible *note repr(): 7f9. is displayed by
default:
>>> import math
>>> repr(math.pi)
'3.141592653589793'
>>> str(math.pi)
'3.141592653589793'
(Proposed and implemented by Mark Dickinson; bpo-9337(5).)
* *note memoryview: 464. objects now have a *note release(): 11b0.
method and they also now support the context management protocol.
This allows timely release of any resources that were acquired when
requesting a buffer from the original object.
>>> with memoryview(b'abcdefgh') as v:
... print(v.tolist())
[97, 98, 99, 100, 101, 102, 103, 104]
(Added by Antoine Pitrou; bpo-9757(6).)
* Previously it was illegal to delete a name from the local namespace
if it occurs as a free variable in a nested block:
def outer(x):
def inner():
return x
inner()
del x
This is now allowed. Remember that the target of an *note except:
18b. clause is cleared, so this code which used to work with Python
2.6, raised a *note SyntaxError: 18d. with Python 3.1 and now works
again:
def f():
def print_error():
print(e)
try:
something
except Exception as e:
print_error()
# implicit "del e" here
(See bpo-4617(7).)
* *note Struct sequence types: 11b1. are now subclasses of tuple.
This means that C structures like those returned by *note
os.stat(): 49c, *note time.gmtime(): 11b2, and *note
sys.version_info: 69c. now work like a *note named tuple: 64a. and
now work with functions and methods that expect a tuple as an
argument. This is a big step forward in making the C structures as
flexible as their pure Python counterparts:
>>> import sys
>>> isinstance(sys.version_info, tuple)
True
>>> 'Version %d.%d.%d %s(%d)' % sys.version_info
'Version 3.2.0 final(0)'
(Suggested by Arfrever Frehtes Taifersar Arahesis and implemented
by Benjamin Peterson in bpo-8413(8).)
* Warnings are now easier to control using the *note PYTHONWARNINGS:
baa. environment variable as an alternative to using ‘-W’ at the
command line:
$ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'
(Suggested by Barry Warsaw and implemented by Philip Jenvey in
bpo-7301(9).)
* A new warning category, *note ResourceWarning: 246, has been added.
It is emitted when potential issues with resource consumption or
cleanup are detected. It is silenced by default in normal release
builds but can be enabled through the means provided by the *note
warnings: 112. module, or on the command line.
A *note ResourceWarning: 246. is issued at interpreter shutdown if
the *note gc.garbage: 11b3. list isn’t empty, and if *note
gc.DEBUG_UNCOLLECTABLE: 11b4. is set, all uncollectable objects are
printed. This is meant to make the programmer aware that their
code contains object finalization issues.
A *note ResourceWarning: 246. is also issued when a *note file
object: 11b5. is destroyed without having been explicitly closed.
While the deallocator for such object ensures it closes the
underlying operating system resource (usually, a file descriptor),
the delay in deallocating the object could produce various issues,
especially under Windows. Here is an example of enabling the
warning from the command line:
$ python -q -Wdefault
>>> f = open("foo", "wb")
>>> del f
__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
(Added by Antoine Pitrou and Georg Brandl in bpo-10093(10) and
bpo-477863(11).)
* *note range: 943. objects now support 'index' and 'count' methods.
This is part of an effort to make more objects fully implement the
*note collections.Sequence: 11b6. *note abstract base class: 11a8.
As a result, the language will have a more uniform API. In
addition, *note range: 943. objects now support slicing and
negative indices, even with values larger than *note sys.maxsize:
11b7. This makes 'range' more interoperable with lists:
>>> range(0, 100, 2).count(10)
1
>>> range(0, 100, 2).index(10)
5
>>> range(0, 100, 2)[5]
10
>>> range(0, 100, 2)[0:5]
range(0, 10, 2)
(Contributed by Daniel Stutzbach in bpo-9213(12), by Alexander
Belopolsky in bpo-2690(13), and by Nick Coghlan in bpo-10889(14).)
* The *note callable(): 11b8. builtin function from Py2.x was
resurrected. It provides a concise, readable alternative to using
an *note abstract base class: 11a8. in an expression like
‘isinstance(x, collections.Callable)’:
>>> callable(max)
True
>>> callable(20)
False
(See bpo-10518(15).)
* Python’s import mechanism can now load modules installed in
directories with non-ASCII characters in the path name. This
solved an aggravating problem with home directories for users with
non-ASCII characters in their usernames.
(Required extensive work by Victor Stinner in bpo-9425(16).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=7094
(2) https://bugs.python.org/issue?@action=redirect&bpo=6081
(3) https://bugs.python.org/issue?@action=redirect&bpo=1772833
(4) https://bugs.python.org/issue?@action=redirect&bpo=9666
(5) https://bugs.python.org/issue?@action=redirect&bpo=9337
(6) https://bugs.python.org/issue?@action=redirect&bpo=9757
(7) https://bugs.python.org/issue?@action=redirect&bpo=4617
(8) https://bugs.python.org/issue?@action=redirect&bpo=8413
(9) https://bugs.python.org/issue?@action=redirect&bpo=7301
(10) https://bugs.python.org/issue?@action=redirect&bpo=10093
(11) https://bugs.python.org/issue?@action=redirect&bpo=477863
(12) https://bugs.python.org/issue?@action=redirect&bpo=9213
(13) https://bugs.python.org/issue?@action=redirect&bpo=2690
(14) https://bugs.python.org/issue?@action=redirect&bpo=10889
(15) https://bugs.python.org/issue?@action=redirect&bpo=10518
(16) https://bugs.python.org/issue?@action=redirect&bpo=9425
File: python.info, Node: New Improved and Deprecated Modules, Next: Multi-threading, Prev: Other Language Changes<12>, Up: What’s New In Python 3 2
1.12.9 New, Improved, and Deprecated Modules
--------------------------------------------
Python’s standard library has undergone significant maintenance efforts
and quality improvements.
The biggest news for Python 3.2 is that the *note email: 3b. package,
*note mailbox: 8b. module, and ‘nntplib’ modules now work correctly with
the bytes/text model in Python 3. For the first time, there is correct
handling of messages with mixed encodings.
Throughout the standard library, there has been more careful attention
to encodings and text versus bytes issues. In particular, interactions
with the operating system are now better able to exchange non-ASCII data
using the Windows MBCS encoding, locale-aware encodings, or UTF-8.
Another significant win is the addition of substantially better support
for 'SSL' connections and security certificates.
In addition, more classes now implement a *note context manager: 5d0. to
support convenient and reliable resource clean-up using a *note with:
5ce. statement.
* Menu:
* email: email<6>.
* elementtree::
* functools: functools<7>.
* itertools: itertools<7>.
* collections: collections<8>.
* threading: threading<9>.
* datetime and time::
* math: math<10>.
* abc: abc<3>.
* io: io<7>.
* reprlib::
* logging: logging<8>.
* csv: csv<4>.
* contextlib: contextlib<8>.
* decimal and fractions::
* ftp::
* popen::
* select: select<3>.
* gzip and zipfile::
* tarfile: tarfile<10>.
* hashlib: hashlib<7>.
* ast: ast<5>.
* os: os<13>.
* shutil: shutil<8>.
* sqlite3: sqlite3<11>.
* html: html<3>.
* socket: socket<11>.
* ssl: ssl<12>.
* nntp::
* certificates::
* imaplib: imaplib<4>.
* http.client: http client<4>.
* unittest: unittest<11>.
* random: random<5>.
* poplib: poplib<4>.
* asyncore: asyncore<2>.
* tempfile: tempfile<5>.
* inspect: inspect<10>.
* pydoc: pydoc<6>.
* dis: dis<5>.
* dbm: dbm<7>.
* ctypes: ctypes<3>.
* site: site<4>.
* sysconfig: sysconfig<3>.
* pdb: pdb<8>.
* configparser: configparser<5>.
* urllib.parse: urllib parse<6>.
* mailbox::
* turtledemo::
File: python.info, Node: email<6>, Next: elementtree, Up: New Improved and Deprecated Modules
1.12.9.1 email
..............
The usability of the *note email: 3b. package in Python 3 has been
mostly fixed by the extensive efforts of R. David Murray. The problem
was that emails are typically read and stored in the form of *note
bytes: 1c2. rather than *note str: 447. text, and they may contain
multiple encodings within a single email. So, the email package had to
be extended to parse and generate email messages in bytes format.
* New functions *note message_from_bytes(): 11bb. and *note
message_from_binary_file(): 11bc, and new classes *note
BytesFeedParser: 11bd. and *note BytesParser: 11be. allow binary
message data to be parsed into model objects.
* Given bytes input to the model, *note get_payload(): 11bf. will by
default decode a message body that has a
‘Content-Transfer-Encoding’ of '8bit' using the charset specified
in the MIME headers and return the resulting string.
* Given bytes input to the model, *note Generator: 11c0. will convert
message bodies that have a ‘Content-Transfer-Encoding’ of '8bit' to
instead have a '7bit' ‘Content-Transfer-Encoding’.
Headers with unencoded non-ASCII bytes are deemed to be RFC
2047(1)-encoded using the 'unknown-8bit' character set.
* A new class *note BytesGenerator: 11c1. produces bytes as output,
preserving any unchanged non-ASCII data that was present in the
input used to build the model, including message bodies with a
‘Content-Transfer-Encoding’ of '8bit'.
* The *note smtplib: ca. *note SMTP: 92d. class now accepts a byte
string for the 'msg' argument to the *note sendmail(): e54. method,
and a new method, *note send_message(): e55. accepts a *note
Message: 27e. object and can optionally obtain the 'from_addr' and
'to_addrs' addresses directly from the object.
(Proposed and implemented by R. David Murray, bpo-4661(2) and
bpo-10321(3).)
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2047.html
(2) https://bugs.python.org/issue?@action=redirect&bpo=4661
(3) https://bugs.python.org/issue?@action=redirect&bpo=10321
File: python.info, Node: elementtree, Next: functools<7>, Prev: email<6>, Up: New Improved and Deprecated Modules
1.12.9.2 elementtree
....................
The *note xml.etree.ElementTree: 125. package and its
‘xml.etree.cElementTree’ counterpart have been updated to version 1.3.
Several new and useful functions and methods have been added:
* *note xml.etree.ElementTree.fromstringlist(): 11c3. which builds an
XML document from a sequence of fragments
* *note xml.etree.ElementTree.register_namespace(): 11c4. for
registering a global namespace prefix
* *note xml.etree.ElementTree.tostringlist(): fef. for string
representation including all sublists
* *note xml.etree.ElementTree.Element.extend(): 11c5. for appending a
sequence of zero or more elements
* *note xml.etree.ElementTree.Element.iterfind(): 11c6. searches an
element and subelements
* *note xml.etree.ElementTree.Element.itertext(): 11c7. creates a
text iterator over an element and its subelements
* *note xml.etree.ElementTree.TreeBuilder.end(): 11c8. closes the
current element
* *note xml.etree.ElementTree.TreeBuilder.doctype(): a97. handles a
doctype declaration
Two methods have been deprecated:
* ‘xml.etree.ElementTree.getchildren()’ use ‘list(elem)’ instead.
* ‘xml.etree.ElementTree.getiterator()’ use ‘Element.iter’ instead.
For details of the update, see Introducing ElementTree(1) on Fredrik
Lundh’s website.
(Contributed by Florent Xicluna and Fredrik Lundh, bpo-6472(2).)
---------- Footnotes ----------
(1)
https://web.archive.org/web/20200703234532/http://effbot.org/zone/elementtree-13-intro.htm
(2) https://bugs.python.org/issue?@action=redirect&bpo=6472
File: python.info, Node: functools<7>, Next: itertools<7>, Prev: elementtree, Up: New Improved and Deprecated Modules
1.12.9.3 functools
..................
* The *note functools: 5f. module includes a new decorator for
caching function calls. *note functools.lru_cache(): 9ee. can save
repeated queries to an external resource whenever the results are
expected to be the same.
For example, adding a caching decorator to a database query
function can save database accesses for popular searches:
>>> import functools
>>> @functools.lru_cache(maxsize=300)
... def get_phone_number(name):
... c = conn.cursor()
... c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
... return c.fetchone()[0]
>>> for name in user_requests:
... get_phone_number(name) # cached lookup
To help with choosing an effective cache size, the wrapped function
is instrumented for tracking cache statistics:
>>> get_phone_number.cache_info()
CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)
If the phonelist table gets updated, the outdated contents of the
cache can be cleared with:
>>> get_phone_number.cache_clear()
(Contributed by Raymond Hettinger and incorporating design ideas
from Jim Baker, Miki Tebeka, and Nick Coghlan; see recipe
498245(1), recipe 577479(2), bpo-10586(3), and bpo-10593(4).)
* The *note functools.wraps(): f58. decorator now adds a
‘__wrapped__’ attribute pointing to the original callable function.
This allows wrapped functions to be introspected. It also copies
*note __annotations__: 11ca. if defined. And now it also
gracefully skips over missing attributes such as *note __doc__:
11cb. which might not be defined for the wrapped callable.
In the above example, the cache can be removed by recovering the
original function:
>>> get_phone_number = get_phone_number.__wrapped__ # uncached function
(By Nick Coghlan and Terrence Cole; bpo-9567(5), bpo-3445(6), and
bpo-8814(7).)
* To help write classes with rich comparison methods, a new decorator
*note functools.total_ordering(): f3d. will use existing equality
and inequality methods to fill in the remaining methods.
For example, supplying '__eq__' and '__lt__' will enable *note
total_ordering(): f3d. to fill-in '__le__', '__gt__' and '__ge__':
@total_ordering
class Student:
def __eq__(self, other):
return ((self.lastname.lower(), self.firstname.lower()) ==
(other.lastname.lower(), other.firstname.lower()))
def __lt__(self, other):
return ((self.lastname.lower(), self.firstname.lower()) <
(other.lastname.lower(), other.firstname.lower()))
With the 'total_ordering' decorator, the remaining comparison
methods are filled in automatically.
(Contributed by Raymond Hettinger.)
* To aid in porting programs from Python 2, the *note
functools.cmp_to_key(): 11cc. function converts an old-style
comparison function to modern *note key function: e04.:
>>> # locale-aware sort order
>>> sorted(iterable, key=cmp_to_key(locale.strcoll))
For sorting examples and a brief sorting tutorial, see the Sorting
HowTo(8) tutorial.
(Contributed by Raymond Hettinger.)
---------- Footnotes ----------
(1)
https://code.activestate.com/recipes/498245-lru-and-lfu-cache-decorators/
(2)
https://code.activestate.com/recipes/577479-simple-caching-decorator/
(3) https://bugs.python.org/issue?@action=redirect&bpo=10586
(4) https://bugs.python.org/issue?@action=redirect&bpo=10593
(5) https://bugs.python.org/issue?@action=redirect&bpo=9567
(6) https://bugs.python.org/issue?@action=redirect&bpo=3445
(7) https://bugs.python.org/issue?@action=redirect&bpo=8814
(8) https://wiki.python.org/moin/HowTo/Sorting/
File: python.info, Node: itertools<7>, Next: collections<8>, Prev: functools<7>, Up: New Improved and Deprecated Modules
1.12.9.4 itertools
..................
* The *note itertools: 81. module has a new *note accumulate(): 9fc.
function modeled on APL’s 'scan' operator and Numpy’s 'accumulate'
function:
>>> from itertools import accumulate
>>> list(accumulate([8, 2, 50]))
[8, 10, 60]
>>> prob_dist = [0.1, 0.4, 0.2, 0.3]
>>> list(accumulate(prob_dist)) # cumulative probability distribution
[0.1, 0.5, 0.7, 1.0]
For an example using *note accumulate(): 9fc, see the *note
examples for the random module: 11ce.
(Contributed by Raymond Hettinger and incorporating design
suggestions from Mark Dickinson.)
File: python.info, Node: collections<8>, Next: threading<9>, Prev: itertools<7>, Up: New Improved and Deprecated Modules
1.12.9.5 collections
....................
* The *note collections.Counter: 108b. class now has two forms of
in-place subtraction, the existing '-=' operator for saturating
subtraction(1) and the new *note subtract(): 11d0. method for
regular subtraction. The former is suitable for multisets(2) which
only have positive counts, and the latter is more suitable for use
cases that allow negative counts:
>>> from collections import Counter
>>> tally = Counter(dogs=5, cats=3)
>>> tally -= Counter(dogs=2, cats=8) # saturating subtraction
>>> tally
Counter({'dogs': 3})
>>> tally = Counter(dogs=5, cats=3)
>>> tally.subtract(dogs=2, cats=8) # regular subtraction
>>> tally
Counter({'dogs': 3, 'cats': -5})
(Contributed by Raymond Hettinger.)
* The *note collections.OrderedDict: 5d7. class has a new method
*note move_to_end(): 11d1. which takes an existing key and moves it
to either the first or last position in the ordered sequence.
The default is to move an item to the last position. This is
equivalent of renewing an entry with ‘od[k] = od.pop(k)’.
A fast move-to-end operation is useful for resequencing entries.
For example, an ordered dictionary can be used to track order of
access by aging entries from the oldest to the most recently
accessed.
>>> from collections import OrderedDict
>>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e'])
>>> list(d)
['a', 'b', 'X', 'd', 'e']
>>> d.move_to_end('X')
>>> list(d)
['a', 'b', 'd', 'e', 'X']
(Contributed by Raymond Hettinger.)
* The *note collections.deque: 5d8. class grew two new methods *note
count(): 11d2. and *note reverse(): 11d3. that make them more
substitutable for *note list: 60d. objects:
>>> from collections import deque
>>> d = deque('simsalabim')
>>> d.count('s')
2
>>> d.reverse()
>>> d
deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])
(Contributed by Raymond Hettinger.)
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/Saturation_arithmetic
(2) https://en.wikipedia.org/wiki/Multiset
File: python.info, Node: threading<9>, Next: datetime and time, Prev: collections<8>, Up: New Improved and Deprecated Modules
1.12.9.6 threading
..................
The *note threading: ed. module has a new *note Barrier: 11d5.
synchronization class for making multiple threads wait until all of them
have reached a common barrier point. Barriers are useful for making
sure that a task with multiple preconditions does not run until all of
the predecessor tasks are complete.
Barriers can work with an arbitrary number of threads. This is a
generalization of a Rendezvous(1) which is defined for only two threads.
Implemented as a two-phase cyclic barrier, *note Barrier: 11d5. objects
are suitable for use in loops. The separate 'filling' and 'draining'
phases assure that all threads get released (drained) before any one of
them can loop back and re-enter the barrier. The barrier fully resets
after each cycle.
Example of using barriers:
from threading import Barrier, Thread
def get_votes(site):
ballots = conduct_election(site)
all_polls_closed.wait() # do not count until all polls are closed
totals = summarize(ballots)
publish(site, totals)
all_polls_closed = Barrier(len(sites))
for site in sites:
Thread(target=get_votes, args=(site,)).start()
In this example, the barrier enforces a rule that votes cannot be
counted at any polling site until all polls are closed. Notice how a
solution with a barrier is similar to one with *note
threading.Thread.join(): 11d6, but the threads stay alive and continue
to do work (summarizing ballots) after the barrier point is crossed.
If any of the predecessor tasks can hang or be delayed, a barrier can be
created with an optional 'timeout' parameter. Then if the timeout
period elapses before all the predecessor tasks reach the barrier point,
all waiting threads are released and a *note BrokenBarrierError: 11d7.
exception is raised:
def get_votes(site):
ballots = conduct_election(site)
try:
all_polls_closed.wait(timeout=midnight - time.now())
except BrokenBarrierError:
lockbox = seal_ballots(ballots)
queue.put(lockbox)
else:
totals = summarize(ballots)
publish(site, totals)
In this example, the barrier enforces a more robust rule. If some
election sites do not finish before midnight, the barrier times-out and
the ballots are sealed and deposited in a queue for later handling.
See Barrier Synchronization Patterns(2) for more examples of how
barriers can be used in parallel computing. Also, there is a simple but
thorough explanation of barriers in The Little Book of Semaphores(3),
'section 3.6'.
(Contributed by Kristján Valur Jónsson with an API review by Jeffrey
Yasskin in bpo-8777(4).)
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/Synchronous_rendezvous
(2)
https://osl.cs.illinois.edu/media/papers/karmani-2009-barrier_synchronization_pattern.pdf
(3) https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf
(4) https://bugs.python.org/issue?@action=redirect&bpo=8777
File: python.info, Node: datetime and time, Next: math<10>, Prev: threading<9>, Up: New Improved and Deprecated Modules
1.12.9.7 datetime and time
..........................
* The *note datetime: 30. module has a new type *note timezone: 10b5.
that implements the *note tzinfo: 5da. interface by returning a
fixed UTC offset and timezone name. This makes it easier to create
timezone-aware datetime objects:
>>> from datetime import datetime, timezone
>>> datetime.now(timezone.utc)
datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)
>>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")
datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)
* Also, *note timedelta: 9cf. objects can now be multiplied by *note
float: 2f1. and divided by *note float: 2f1. and *note int: 259.
objects. And *note timedelta: 9cf. objects can now divide one
another.
* The *note datetime.date.strftime(): c91. method is no longer
restricted to years after 1900. The new supported year range is
from 1000 to 9999 inclusive.
* Whenever a two-digit year is used in a time tuple, the
interpretation has been governed by ‘time.accept2dyear’. The
default is ‘True’ which means that for a two-digit year, the
century is guessed according to the POSIX rules governing the ‘%y’
strptime format.
Starting with Py3.2, use of the century guessing heuristic will
emit a *note DeprecationWarning: 1a5. Instead, it is recommended
that ‘time.accept2dyear’ be set to ‘False’ so that large date
ranges can be used without guesswork:
>>> import time, warnings
>>> warnings.resetwarnings() # remove the default warning filters
>>> time.accept2dyear = True # guess whether 11 means 11 or 2011
>>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
Warning (from warnings module):
...
DeprecationWarning: Century info guessed for a 2-digit year.
'Fri Jan 1 12:34:56 2011'
>>> time.accept2dyear = False # use the full range of allowable dates
>>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
'Fri Jan 1 12:34:56 11'
Several functions now have significantly expanded date ranges.
When ‘time.accept2dyear’ is false, the *note time.asctime(): 11d9.
function will accept any year that fits in a C int, while the *note
time.mktime(): 11da. and *note time.strftime(): 11db. functions
will accept the full range supported by the corresponding operating
system functions.
(Contributed by Alexander Belopolsky and Victor Stinner in
bpo-1289118(1), bpo-5094(2), bpo-6641(3), bpo-2706(4), bpo-1777412(5),
bpo-8013(6), and bpo-10827(7).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1289118
(2) https://bugs.python.org/issue?@action=redirect&bpo=5094
(3) https://bugs.python.org/issue?@action=redirect&bpo=6641
(4) https://bugs.python.org/issue?@action=redirect&bpo=2706
(5) https://bugs.python.org/issue?@action=redirect&bpo=1777412
(6) https://bugs.python.org/issue?@action=redirect&bpo=8013
(7) https://bugs.python.org/issue?@action=redirect&bpo=10827
File: python.info, Node: math<10>, Next: abc<3>, Prev: datetime and time, Up: New Improved and Deprecated Modules
1.12.9.8 math
.............
The *note math: 8e. module has been updated with six new functions
inspired by the C99 standard.
The *note isfinite(): 11dd. function provides a reliable and fast way to
detect special values. It returns ‘True’ for regular numbers and
‘False’ for 'Nan' or 'Infinity':
>>> from math import isfinite
>>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))]
[True, True, False, False]
The *note expm1(): 11de. function computes ‘e**x-1’ for small values of
'x' without incurring the loss of precision that usually accompanies the
subtraction of nearly equal quantities:
>>> from math import expm1
>>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x
0.013765762467652909
The *note erf(): bd8. function computes a probability integral or
Gaussian error function(1). The complementary error function, *note
erfc(): bd9, is ‘1 - erf(x)’:
>>> from math import erf, erfc, sqrt
>>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation
0.682689492137086
>>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation
0.31731050786291404
>>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0))
1.0
The *note gamma(): 11df. function is a continuous extension of the
factorial function. See ‘https://en.wikipedia.org/wiki/Gamma_function’
for details. Because the function is related to factorials, it grows
large even for small values of 'x', so there is also a *note lgamma():
11e0. function for computing the natural logarithm of the gamma
function:
>>> from math import gamma, lgamma
>>> gamma(7.0) # six factorial
720.0
>>> lgamma(801.0) # log(800 factorial)
4551.950730698041
(Contributed by Mark Dickinson.)
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/Error_function
File: python.info, Node: abc<3>, Next: io<7>, Prev: math<10>, Up: New Improved and Deprecated Modules
1.12.9.9 abc
............
The *note abc: 4. module now supports *note abstractclassmethod(): 1080.
and *note abstractstaticmethod(): 1081.
These tools make it possible to define an *note abstract base class:
11a8. that requires a particular *note classmethod(): 166. or *note
staticmethod(): 412. to be implemented:
class Temperature(metaclass=abc.ABCMeta):
@abc.abstractclassmethod
def from_fahrenheit(cls, t):
...
@abc.abstractclassmethod
def from_celsius(cls, t):
...
(Patch submitted by Daniel Urban; bpo-5867(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5867
File: python.info, Node: io<7>, Next: reprlib, Prev: abc<3>, Up: New Improved and Deprecated Modules
1.12.9.10 io
............
The *note io.BytesIO: e9d. has a new method, *note getbuffer(): 11e3,
which provides functionality similar to *note memoryview(): 464. It
creates an editable view of the data without making a copy. The
buffer’s random access and support for slice notation are well-suited to
in-place editing:
>>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11
>>> def change_location(buffer, record_number, location):
... start = record_number * REC_LEN + LOC_START
... buffer[start: start+LOC_LEN] = location
>>> import io
>>> byte_stream = io.BytesIO(
... b'G3805 storeroom Main chassis '
... b'X7899 shipping Reserve cog '
... b'L6988 receiving Primary sprocket'
... )
>>> buffer = byte_stream.getbuffer()
>>> change_location(buffer, 1, b'warehouse ')
>>> change_location(buffer, 0, b'showroom ')
>>> print(byte_stream.getvalue())
b'G3805 showroom Main chassis '
b'X7899 warehouse Reserve cog '
b'L6988 receiving Primary sprocket'
(Contributed by Antoine Pitrou in bpo-5506(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5506
File: python.info, Node: reprlib, Next: logging<8>, Prev: io<7>, Up: New Improved and Deprecated Modules
1.12.9.11 reprlib
.................
When writing a *note __repr__(): 618. method for a custom container, it
is easy to forget to handle the case where a member refers back to the
container itself. Python’s builtin objects such as *note list: 60d. and
*note set: 5d5. handle self-reference by displaying “…” in the recursive
part of the representation string.
To help write such *note __repr__(): 618. methods, the *note reprlib:
bb. module has a new decorator, *note recursive_repr(): 11e5, for
detecting recursive calls to ‘__repr__()’ and substituting a placeholder
string instead:
>>> class MyList(list):
... @recursive_repr()
... def __repr__(self):
... return '<' + '|'.join(map(repr, self)) + '>'
...
>>> m = MyList('abc')
>>> m.append(m)
>>> m.append('x')
>>> print(m)
<'a'|'b'|'c'|...|'x'>
(Contributed by Raymond Hettinger in bpo-9826(1) and bpo-9840(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9826
(2) https://bugs.python.org/issue?@action=redirect&bpo=9840
File: python.info, Node: logging<8>, Next: csv<4>, Prev: reprlib, Up: New Improved and Deprecated Modules
1.12.9.12 logging
.................
In addition to dictionary-based configuration described above, the *note
logging: 87. package has many other improvements.
The logging documentation has been augmented by a *note basic tutorial:
11e7, an *note advanced tutorial: 11e8, and a *note cookbook: 11e9. of
logging recipes. These documents are the fastest way to learn about
logging.
The *note logging.basicConfig(): 9ff. set-up function gained a 'style'
argument to support three different types of string formatting. It
defaults to “%” for traditional %-formatting, can be set to “{” for the
new *note str.format(): 61d. style, or can be set to “$” for the
shell-style formatting provided by *note string.Template: 683. The
following three configurations are equivalent:
>>> from logging import basicConfig
>>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s")
>>> basicConfig(style='{', format="{name} -> {levelname} {message}")
>>> basicConfig(style='$', format="$name -> $levelname: $message")
If no configuration is set-up before a logging event occurs, there is
now a default configuration using a *note StreamHandler: 11ea. directed
to *note sys.stderr: 939. for events of ‘WARNING’ level or higher.
Formerly, an event occurring before a configuration was set-up would
either raise an exception or silently drop the event depending on the
value of *note logging.raiseExceptions: 11eb. The new default handler
is stored in *note logging.lastResort: 11ec.
The use of filters has been simplified. Instead of creating a *note
Filter: 11ed. object, the predicate can be any Python callable that
returns ‘True’ or ‘False’.
There were a number of other improvements that add flexibility and
simplify configuration. See the module documentation for a full listing
of changes in Python 3.2.
File: python.info, Node: csv<4>, Next: contextlib<8>, Prev: logging<8>, Up: New Improved and Deprecated Modules
1.12.9.13 csv
.............
The *note csv: 29. module now supports a new dialect, *note
unix_dialect: 11ef, which applies quoting for all fields and a
traditional Unix style with ‘'\n'’ as the line terminator. The
registered dialect name is ‘unix’.
The *note csv.DictWriter: 11f0. has a new method, *note writeheader():
11f1. for writing-out an initial row to document the field names:
>>> import csv, sys
>>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix')
>>> w.writeheader()
"name","dept"
>>> w.writerows([
... {'name': 'tom', 'dept': 'accounting'},
... {'name': 'susan', 'dept': 'Salesl'}])
"tom","accounting"
"susan","sales"
(New dialect suggested by Jay Talbot in bpo-5975(1), and the new method
suggested by Ed Abraham in bpo-1537721(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5975
(2) https://bugs.python.org/issue?@action=redirect&bpo=1537721
File: python.info, Node: contextlib<8>, Next: decimal and fractions, Prev: csv<4>, Up: New Improved and Deprecated Modules
1.12.9.14 contextlib
....................
There is a new and slightly mind-blowing tool *note ContextDecorator:
11f3. that is helpful for creating a *note context manager: 5d0. that
does double duty as a function decorator.
As a convenience, this new functionality is used by *note
contextmanager(): 11f4. so that no extra effort is needed to support
both roles.
The basic idea is that both context managers and function decorators can
be used for pre-action and post-action wrappers. Context managers wrap
a group of statements using a *note with: 5ce. statement, and function
decorators wrap a group of statements enclosed in a function. So,
occasionally there is a need to write a pre-action or post-action
wrapper that can be used in either role.
For example, it is sometimes useful to wrap functions or groups of
statements with a logger that can track the time of entry and time of
exit. Rather than writing both a function decorator and a context
manager for the task, the *note contextmanager(): 11f4. provides both
capabilities in a single definition:
from contextlib import contextmanager
import logging
logging.basicConfig(level=logging.INFO)
@contextmanager
def track_entry_and_exit(name):
logging.info('Entering: %s', name)
yield
logging.info('Exiting: %s', name)
Formerly, this would have only been usable as a context manager:
with track_entry_and_exit('widget loader'):
print('Some time consuming activity goes here')
load_widget()
Now, it can be used as a decorator as well:
@track_entry_and_exit('widget loader')
def activity():
print('Some time consuming activity goes here')
load_widget()
Trying to fulfill two roles at once places some limitations on the
technique. Context managers normally have the flexibility to return an
argument usable by a *note with: 5ce. statement, but there is no
parallel for function decorators.
In the above example, there is not a clean way for the
'track_entry_and_exit' context manager to return a logging instance for
use in the body of enclosed statements.
(Contributed by Michael Foord in bpo-9110(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9110
File: python.info, Node: decimal and fractions, Next: ftp, Prev: contextlib<8>, Up: New Improved and Deprecated Modules
1.12.9.15 decimal and fractions
...............................
Mark Dickinson crafted an elegant and efficient scheme for assuring that
different numeric datatypes will have the same hash value whenever their
actual values are equal (bpo-8188(1)):
assert hash(Fraction(3, 2)) == hash(1.5) == \
hash(Decimal("1.5")) == hash(complex(1.5, 0))
Some of the hashing details are exposed through a new attribute, *note
sys.hash_info: ff8, which describes the bit width of the hash value, the
prime modulus, the hash values for 'infinity' and 'nan', and the
multiplier used for the imaginary part of a number:
>>> sys.hash_info
sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003)
An early decision to limit the interoperability of various numeric types
has been relaxed. It is still unsupported (and ill-advised) to have
implicit mixing in arithmetic expressions such as ‘Decimal('1.1') +
float('1.1')’ because the latter loses information in the process of
constructing the binary float. However, since existing floating-point
value can be converted losslessly to either a decimal or rational
representation, it makes sense to add them to the constructor and to
support mixed-type comparisons.
* The *note decimal.Decimal: 29f. constructor now accepts *note
float: 2f1. objects directly so there in no longer a need to use
the *note from_float(): 11f6. method (bpo-8257(2)).
* Mixed type comparisons are now fully supported so that *note
Decimal: 29f. objects can be directly compared with *note float:
2f1. and *note fractions.Fraction: 1e9. (bpo-2531(3) and
bpo-8188(4)).
Similar changes were made to *note fractions.Fraction: 1e9. so that the
*note from_float(): 11f7. and *note from_decimal(): 11f8. methods are no
longer needed (bpo-8294(5)):
>>> from decimal import Decimal
>>> from fractions import Fraction
>>> Decimal(1.1)
Decimal('1.100000000000000088817841970012523233890533447265625')
>>> Fraction(1.1)
Fraction(2476979795053773, 2251799813685248)
Another useful change for the *note decimal: 36. module is that the
*note Context.clamp: 1191. attribute is now public. This is useful in
creating contexts that correspond to the decimal interchange formats
specified in IEEE 754 (see bpo-8540(6)).
(Contributed by Mark Dickinson and Raymond Hettinger.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8188
(2) https://bugs.python.org/issue?@action=redirect&bpo=8257
(3) https://bugs.python.org/issue?@action=redirect&bpo=2531
(4) https://bugs.python.org/issue?@action=redirect&bpo=8188
(5) https://bugs.python.org/issue?@action=redirect&bpo=8294
(6) https://bugs.python.org/issue?@action=redirect&bpo=8540
File: python.info, Node: ftp, Next: popen, Prev: decimal and fractions, Up: New Improved and Deprecated Modules
1.12.9.16 ftp
.............
The *note ftplib.FTP: 8f9. class now supports the context management
protocol to unconditionally consume *note socket.error: 104e. exceptions
and to close the FTP connection when done:
>>> from ftplib import FTP
>>> with FTP("ftp1.at.proftpd.org") as ftp:
ftp.login()
ftp.dir()
'230 Anonymous login ok, restrictions apply.'
dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .
dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 ..
dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS
dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora
Other file-like objects such as *note mmap.mmap: 20d. and *note
fileinput.input(): 7fb. also grew auto-closing context managers:
with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
for line in f:
process(line)
(Contributed by Tarek Ziadé and Giampaolo Rodolà in bpo-4972(1), and by
Georg Brandl in bpo-8046(2) and bpo-1286(3).)
The *note FTP_TLS: 8fa. class now accepts a 'context' parameter, which
is a *note ssl.SSLContext: 296. object allowing bundling SSL
configuration options, certificates and private keys into a single
(potentially long-lived) structure.
(Contributed by Giampaolo Rodolà; bpo-8806(4).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4972
(2) https://bugs.python.org/issue?@action=redirect&bpo=8046
(3) https://bugs.python.org/issue?@action=redirect&bpo=1286
(4) https://bugs.python.org/issue?@action=redirect&bpo=8806
File: python.info, Node: popen, Next: select<3>, Prev: ftp, Up: New Improved and Deprecated Modules
1.12.9.17 popen
...............
The *note os.popen(): a87. and *note subprocess.Popen(): 199. functions
now support *note with: 5ce. statements for auto-closing of the file
descriptors.
(Contributed by Antoine Pitrou and Brian Curtin in bpo-7461(1) and
bpo-10554(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=7461
(2) https://bugs.python.org/issue?@action=redirect&bpo=10554
File: python.info, Node: select<3>, Next: gzip and zipfile, Prev: popen, Up: New Improved and Deprecated Modules
1.12.9.18 select
................
The *note select: c1. module now exposes a new, constant attribute,
*note PIPE_BUF: 11fc, which gives the minimum number of bytes which are
guaranteed not to block when *note select.select(): d9f. says a pipe is
ready for writing.
>>> import select
>>> select.PIPE_BUF
512
(Available on Unix systems. Patch by Sébastien Sablé in bpo-9862(1))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9862
File: python.info, Node: gzip and zipfile, Next: tarfile<10>, Prev: select<3>, Up: New Improved and Deprecated Modules
1.12.9.19 gzip and zipfile
..........................
*note gzip.GzipFile: 416. now implements the *note io.BufferedIOBase:
690. *note abstract base class: 11a8. (except for ‘truncate()’). It
also has a *note peek(): 11fe. method and supports unseekable as well as
zero-padded file objects.
The *note gzip: 67. module also gains the *note compress(): 63a. and
*note decompress(): 11ff. functions for easier in-memory compression and
decompression. Keep in mind that text needs to be encoded as *note
bytes: 1c2. before compressing and decompressing:
>>> import gzip
>>> s = 'Three shall be the number thou shalt count, '
>>> s += 'and the number of the counting shall be three'
>>> b = s.encode() # convert to utf-8
>>> len(b)
89
>>> c = gzip.compress(b)
>>> len(c)
77
>>> gzip.decompress(c).decode()[:42] # decompress and convert to text
'Three shall be the number thou shalt count'
(Contributed by Anand B. Pillai in bpo-3488(1); and by Antoine Pitrou,
Nir Aides and Brian Curtin in bpo-9962(2), bpo-1675951(3), bpo-7471(4)
and bpo-2846(5).)
Also, the *note zipfile.ZipExtFile: 417. class was reworked internally
to represent files stored inside an archive. The new implementation is
significantly faster and can be wrapped in an *note io.BufferedReader:
1200. object for more speedups. It also solves an issue where
interleaved calls to 'read' and 'readline' gave the wrong results.
(Patch submitted by Nir Aides in bpo-7610(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=3488
(2) https://bugs.python.org/issue?@action=redirect&bpo=9962
(3) https://bugs.python.org/issue?@action=redirect&bpo=1675951
(4) https://bugs.python.org/issue?@action=redirect&bpo=7471
(5) https://bugs.python.org/issue?@action=redirect&bpo=2846
(6) https://bugs.python.org/issue?@action=redirect&bpo=7610
File: python.info, Node: tarfile<10>, Next: hashlib<7>, Prev: gzip and zipfile, Up: New Improved and Deprecated Modules
1.12.9.20 tarfile
.................
The *note TarFile: 1202. class can now be used as a context manager. In
addition, its *note add(): bf3. method has a new option, 'filter', that
controls which files are added to the archive and allows the file
metadata to be edited.
The new 'filter' option replaces the older, less flexible 'exclude'
parameter which is now deprecated. If specified, the optional 'filter'
parameter needs to be a *note keyword argument: d12. The user-supplied
filter function accepts a *note TarInfo: 1203. object and returns an
updated *note TarInfo: 1203. object, or if it wants the file to be
excluded, the function can return ‘None’:
>>> import tarfile, glob
>>> def myfilter(tarinfo):
... if tarinfo.isfile(): # only save real files
... tarinfo.uname = 'monty' # redact the user name
... return tarinfo
>>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:
... for filename in glob.glob('*.txt'):
... tf.add(filename, filter=myfilter)
... tf.list()
-rw-r--r-- monty/501 902 2011-01-26 17:59:11 annotations.txt
-rw-r--r-- monty/501 123 2011-01-26 17:59:11 general_questions.txt
-rw-r--r-- monty/501 3514 2011-01-26 17:59:11 prion.txt
-rw-r--r-- monty/501 124 2011-01-26 17:59:11 py_todo.txt
-rw-r--r-- monty/501 1399 2011-01-26 17:59:11 semaphore_notes.txt
(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in
bpo-6856(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6856
File: python.info, Node: hashlib<7>, Next: ast<5>, Prev: tarfile<10>, Up: New Improved and Deprecated Modules
1.12.9.21 hashlib
.................
The *note hashlib: 68. module has two new constant attributes listing
the hashing algorithms guaranteed to be present in all implementations
and those available on the current implementation:
>>> import hashlib
>>> hashlib.algorithms_guaranteed
{'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'}
>>> hashlib.algorithms_available
{'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256',
'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2',
'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224',
'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'}
(Suggested by Carl Chenet in bpo-7418(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=7418
File: python.info, Node: ast<5>, Next: os<13>, Prev: hashlib<7>, Up: New Improved and Deprecated Modules
1.12.9.22 ast
.............
The *note ast: 8. module has a wonderful a general-purpose tool for
safely evaluating expression strings using the Python literal syntax.
The *note ast.literal_eval(): c18. function serves as a secure
alternative to the builtin *note eval(): 180. function which is easily
abused. Python 3.2 adds *note bytes: 1c2. and *note set: 5d5. literals
to the list of supported types: strings, bytes, numbers, tuples, lists,
dicts, sets, booleans, and ‘None’.
>>> from ast import literal_eval
>>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}"
>>> literal_eval(request)
{'args': (2, 0.5), 'req': 3, 'func': 'pow'}
>>> request = "os.system('do something harmful')"
>>> literal_eval(request)
Traceback (most recent call last):
...
ValueError: malformed node or string: <_ast.Call object at 0x101739a10>
(Implemented by Benjamin Peterson and Georg Brandl.)
File: python.info, Node: os<13>, Next: shutil<8>, Prev: ast<5>, Up: New Improved and Deprecated Modules
1.12.9.23 os
............
Different operating systems use various encodings for filenames and
environment variables. The *note os: a1. module provides two new
functions, *note fsencode(): c55. and *note fsdecode(): c54, for
encoding and decoding filenames:
>>> import os
>>> filename = 'Sehenswürdigkeiten'
>>> os.fsencode(filename)
b'Sehensw\xc3\xbcrdigkeiten'
Some operating systems allow direct access to encoded bytes in the
environment. If so, the *note os.supports_bytes_environ: 1207. constant
will be true.
For direct access to encoded environment variables (if available), use
the new *note os.getenvb(): 1208. function or use *note os.environb:
1209. which is a bytes version of *note os.environ: 11ac.
(Contributed by Victor Stinner.)
File: python.info, Node: shutil<8>, Next: sqlite3<11>, Prev: os<13>, Up: New Improved and Deprecated Modules
1.12.9.24 shutil
................
The *note shutil.copytree(): a28. function has two new options:
* 'ignore_dangling_symlinks': when ‘symlinks=False’ so that the
function copies a file pointed to by a symlink, not the symlink
itself. This option will silence the error raised if the file
doesn’t exist.
* 'copy_function': is a callable that will be used to copy files.
*note shutil.copy2(): a5a. is used by default.
(Contributed by Tarek Ziadé.)
In addition, the *note shutil: c5. module now supports *note archiving
operations: 120b. for zipfiles, uncompressed tarfiles, gzipped tarfiles,
and bzipped tarfiles. And there are functions for registering
additional archiving file formats (such as xz compressed tarfiles or
custom formats).
The principal functions are *note make_archive(): 4af. and *note
unpack_archive(): 467. By default, both operate on the current
directory (which can be set by *note os.chdir(): 609.) and on any
sub-directories. The archive filename needs to be specified with a full
pathname. The archiving step is non-destructive (the original files are
left unchanged).
>>> import shutil, pprint
>>> os.chdir('mydata') # change to the source directory
>>> f = shutil.make_archive('/var/backup/mydata',
... 'zip') # archive the current directory
>>> f # show the name of archive
'/var/backup/mydata.zip'
>>> os.chdir('tmp') # change to an unpacking
>>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data
>>> pprint.pprint(shutil.get_archive_formats()) # display known formats
[('bztar', "bzip2'ed tar-file"),
('gztar', "gzip'ed tar-file"),
('tar', 'uncompressed tar file'),
('zip', 'ZIP file')]
>>> shutil.register_archive_format( # register a new archive format
... name='xz',
... function=xz.compress, # callable archiving function
... extra_args=[('level', 8)], # arguments to the function
... description='xz compression'
... )
(Contributed by Tarek Ziadé.)
File: python.info, Node: sqlite3<11>, Next: html<3>, Prev: shutil<8>, Up: New Improved and Deprecated Modules
1.12.9.25 sqlite3
.................
The *note sqlite3: cf. module was updated to pysqlite version 2.6.0. It
has two new capabilities.
* The ‘sqlite3.Connection.in_transit’ attribute is true if there is
an active transaction for uncommitted changes.
* The *note sqlite3.Connection.enable_load_extension(): 83a. and
*note sqlite3.Connection.load_extension(): 4b4. methods allows you
to load SQLite extensions from “.so” files. One well-known
extension is the fulltext-search extension distributed with SQLite.
(Contributed by R. David Murray and Shashwat Anand; bpo-8845(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8845
File: python.info, Node: html<3>, Next: socket<11>, Prev: sqlite3<11>, Up: New Improved and Deprecated Modules
1.12.9.26 html
..............
A new *note html: 6b. module was introduced with only a single function,
*note escape(): 1007, which is used for escaping reserved characters
from HTML markup:
>>> import html
>>> html.escape('x > 2 && x < 7')
'x > 2 && x < 7'
File: python.info, Node: socket<11>, Next: ssl<12>, Prev: html<3>, Up: New Improved and Deprecated Modules
1.12.9.27 socket
................
The *note socket: cc. module has two new improvements.
* Socket objects now have a *note detach(): 120f. method which puts
the socket into closed state without actually closing the
underlying file descriptor. The latter can then be reused for
other purposes. (Added by Antoine Pitrou; bpo-8524(1).)
* *note socket.create_connection(): 66e. now supports the context
management protocol to unconditionally consume *note socket.error:
104e. exceptions and to close the socket when done. (Contributed
by Giampaolo Rodolà; bpo-9794(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8524
(2) https://bugs.python.org/issue?@action=redirect&bpo=9794
File: python.info, Node: ssl<12>, Next: nntp, Prev: socket<11>, Up: New Improved and Deprecated Modules
1.12.9.28 ssl
.............
The *note ssl: d0. module added a number of features to satisfy common
requirements for secure (encrypted, authenticated) internet connections:
* A new class, *note SSLContext: 296, serves as a container for
persistent SSL data, such as protocol settings, certificates,
private keys, and various other options. It includes a *note
wrap_socket(): 520. for creating an SSL socket from an SSL context.
* A new function, ‘ssl.match_hostname()’, supports server identity
verification for higher-level protocols by implementing the rules
of HTTPS (from RFC 2818(1)) which are also suitable for other
protocols.
* The *note ssl.wrap_socket(): 520. constructor function now takes a
'ciphers' argument. The 'ciphers' string lists the allowed
encryption algorithms using the format described in the OpenSSL
documentation(2).
* When linked against recent versions of OpenSSL, the *note ssl: d0.
module now supports the Server Name Indication extension to the TLS
protocol, allowing multiple “virtual hosts” using different
certificates on a single IP port. This extension is only supported
in client mode, and is activated by passing the 'server_hostname'
argument to *note ssl.SSLContext.wrap_socket(): 520.
* Various options have been added to the *note ssl: d0. module, such
as *note OP_NO_SSLv2: 835. which disables the insecure and obsolete
SSLv2 protocol.
* The extension now loads all the OpenSSL ciphers and digest
algorithms. If some SSL certificates cannot be verified, they are
reported as an “unknown algorithm” error.
* The version of OpenSSL being used is now accessible using the
module attributes *note ssl.OPENSSL_VERSION: 1211. (a string),
*note ssl.OPENSSL_VERSION_INFO: 1212. (a 5-tuple), and *note
ssl.OPENSSL_VERSION_NUMBER: 1213. (an integer).
(Contributed by Antoine Pitrou in bpo-8850(3), bpo-1589(4), bpo-8322(5),
bpo-5639(6), bpo-4870(7), bpo-8484(8), and bpo-8321(9).)
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2818.html
(2) https://docs.openssl.org/1.0.2/man1/ciphers/#cipher-list-format
(3) https://bugs.python.org/issue?@action=redirect&bpo=8850
(4) https://bugs.python.org/issue?@action=redirect&bpo=1589
(5) https://bugs.python.org/issue?@action=redirect&bpo=8322
(6) https://bugs.python.org/issue?@action=redirect&bpo=5639
(7) https://bugs.python.org/issue?@action=redirect&bpo=4870
(8) https://bugs.python.org/issue?@action=redirect&bpo=8484
(9) https://bugs.python.org/issue?@action=redirect&bpo=8321
File: python.info, Node: nntp, Next: certificates, Prev: ssl<12>, Up: New Improved and Deprecated Modules
1.12.9.29 nntp
..............
The ‘nntplib’ module has a revamped implementation with better bytes and
text semantics as well as more practical APIs. These improvements break
compatibility with the nntplib version in Python 3.1, which was partly
dysfunctional in itself.
Support for secure connections through both implicit (using
‘nntplib.NNTP_SSL’) and explicit (using ‘nntplib.NNTP.starttls()’) TLS
has also been added.
(Contributed by Antoine Pitrou in bpo-9360(1) and Andrew Vant in
bpo-1926(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9360
(2) https://bugs.python.org/issue?@action=redirect&bpo=1926
File: python.info, Node: certificates, Next: imaplib<4>, Prev: nntp, Up: New Improved and Deprecated Modules
1.12.9.30 certificates
......................
*note http.client.HTTPSConnection: b41, *note
urllib.request.HTTPSHandler: 1216. and *note urllib.request.urlopen():
295. now take optional arguments to allow for server certificate
checking against a set of Certificate Authorities, as recommended in
public uses of HTTPS.
(Added by Antoine Pitrou, bpo-9003(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9003
File: python.info, Node: imaplib<4>, Next: http client<4>, Prev: certificates, Up: New Improved and Deprecated Modules
1.12.9.31 imaplib
.................
Support for explicit TLS on standard IMAP4 connections has been added
through the new *note imaplib.IMAP4.starttls: 1218. method.
(Contributed by Lorenzo M. Catucci and Antoine Pitrou, bpo-4471(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4471
File: python.info, Node: http client<4>, Next: unittest<11>, Prev: imaplib<4>, Up: New Improved and Deprecated Modules
1.12.9.32 http.client
.....................
There were a number of small API improvements in the *note http.client:
6f. module. The old-style HTTP 0.9 simple responses are no longer
supported and the 'strict' parameter is deprecated in all classes.
The *note HTTPConnection: b40. and *note HTTPSConnection: b41. classes
now have a 'source_address' parameter for a (host, port) tuple
indicating where the HTTP connection is made from.
Support for certificate checking and HTTPS virtual hosts were added to
*note HTTPSConnection: b41.
The *note request(): ca9. method on connection objects allowed an
optional 'body' argument so that a *note file object: 11b5. could be
used to supply the content of the request. Conveniently, the 'body'
argument now also accepts an *note iterable: 121a. object so long as it
includes an explicit ‘Content-Length’ header. This extended interface
is much more flexible than before.
To establish an HTTPS connection through a proxy server, there is a new
*note set_tunnel(): 121b. method that sets the host and port for HTTP
Connect tunneling.
To match the behavior of *note http.server: 72, the HTTP client library
now also encodes headers with ISO-8859-1 (Latin-1) encoding. It was
already doing that for incoming headers, so now the behavior is
consistent for both incoming and outgoing traffic. (See work by Armin
Ronacher in bpo-10980(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10980
File: python.info, Node: unittest<11>, Next: random<5>, Prev: http client<4>, Up: New Improved and Deprecated Modules
1.12.9.33 unittest
..................
The unittest module has a number of improvements supporting test
discovery for packages, easier experimentation at the interactive
prompt, new testcase methods, improved diagnostic messages for test
failures, and better method names.
* The command-line call ‘python -m unittest’ can now accept file
paths instead of module names for running specific tests
(bpo-10620(1)). The new test discovery can find tests within
packages, locating any test importable from the top-level
directory. The top-level directory can be specified with the ‘-t’
option, a pattern for matching files with ‘-p’, and a directory to
start discovery with ‘-s’:
$ python -m unittest discover -s my_proj_dir -p _test.py
(Contributed by Michael Foord.)
* Experimentation at the interactive prompt is now easier because the
*note unittest.TestCase: 449. class can now be instantiated without
arguments:
>>> from unittest import TestCase
>>> TestCase().assertEqual(pow(2, 3), 8)
(Contributed by Michael Foord.)
* The *note unittest: 106. module has two new methods, *note
assertWarns(): 1158. and *note assertWarnsRegex(): 1159. to verify
that a given warning type is triggered by the code under test:
with self.assertWarns(DeprecationWarning):
legacy_function('XYZ')
(Contributed by Antoine Pitrou, bpo-9754(2).)
Another new method, *note assertCountEqual(): 121d. is used to
compare two iterables to determine if their element counts are
equal (whether the same elements are present with the same number
of occurrences regardless of order):
def test_anagram(self):
self.assertCountEqual('algorithm', 'logarithm')
(Contributed by Raymond Hettinger.)
* A principal feature of the unittest module is an effort to produce
meaningful diagnostics when a test fails. When possible, the
failure is recorded along with a diff of the output. This is
especially helpful for analyzing log files of failed test runs.
However, since diffs can sometime be voluminous, there is a new
*note maxDiff: 121e. attribute that sets maximum length of diffs
displayed.
* In addition, the method names in the module have undergone a number
of clean-ups.
For example, *note assertRegex(): 529. is the new name for
‘assertRegexpMatches()’ which was misnamed because the test uses
*note re.search(): 121f, not *note re.match(): 1220. Other methods
using regular expressions are now named using short form “Regex” in
preference to “Regexp” – this matches the names used in other
unittest implementations, matches Python’s old name for the *note
re: b9. module, and it has unambiguous camel-casing.
(Contributed by Raymond Hettinger and implemented by Ezio Melotti.)
* To improve consistency, some long-standing method aliases are being
deprecated in favor of the preferred names:
Old Name Preferred Name
-----------------------------------------------------------------------
‘assert_()’ *note assertTrue(): 522.
‘assertEquals()’ *note assertEqual(): 524.
‘assertNotEquals()’ *note assertNotEqual(): 525.
‘assertAlmostEquals()’ *note assertAlmostEqual(): 526.
‘assertNotAlmostEquals()’ *note assertNotAlmostEqual(): 527.
Likewise, the ‘TestCase.fail*’ methods deprecated in Python 3.1 are
expected to be removed in Python 3.3.
(Contributed by Ezio Melotti; bpo-9424(3).)
* The ‘assertDictContainsSubset()’ method was deprecated because it
was misimplemented with the arguments in the wrong order. This
created hard-to-debug optical illusions where tests like
‘TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})’
would fail.
(Contributed by Raymond Hettinger.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10620
(2) https://bugs.python.org/issue?@action=redirect&bpo=9754
(3) https://bugs.python.org/issue?@action=redirect&bpo=9424
File: python.info, Node: random<5>, Next: poplib<4>, Prev: unittest<11>, Up: New Improved and Deprecated Modules
1.12.9.34 random
................
The integer methods in the *note random: b8. module now do a better job
of producing uniform distributions. Previously, they computed
selections with ‘int(n*random())’ which had a slight bias whenever 'n'
was not a power of two. Now, multiple selections are made from a range
up to the next power of two and a selection is kept only when it falls
within the range ‘0 <= x < n’. The functions and methods affected are
*note randrange(): 868, *note randint(): 1222, *note choice(): 1223,
*note shuffle(): 742. and *note sample(): 741.
(Contributed by Raymond Hettinger; bpo-9025(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9025
File: python.info, Node: poplib<4>, Next: asyncore<2>, Prev: random<5>, Up: New Improved and Deprecated Modules
1.12.9.35 poplib
................
*note POP3_SSL: 924. class now accepts a 'context' parameter, which is a
*note ssl.SSLContext: 296. object allowing bundling SSL configuration
options, certificates and private keys into a single (potentially
long-lived) structure.
(Contributed by Giampaolo Rodolà; bpo-8807(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8807
File: python.info, Node: asyncore<2>, Next: tempfile<5>, Prev: poplib<4>, Up: New Improved and Deprecated Modules
1.12.9.36 asyncore
..................
‘asyncore.dispatcher’ now provides a ‘handle_accepted()’ method
returning a ‘(sock, addr)’ pair which is called when a connection has
actually been established with a new remote endpoint. This is supposed
to be used as a replacement for old ‘handle_accept()’ and avoids the
user to call ‘accept()’ directly.
(Contributed by Giampaolo Rodolà; bpo-6706(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6706
File: python.info, Node: tempfile<5>, Next: inspect<10>, Prev: asyncore<2>, Up: New Improved and Deprecated Modules
1.12.9.37 tempfile
..................
The *note tempfile: e0. module has a new context manager, *note
TemporaryDirectory: 1227. which provides easy deterministic cleanup of
temporary directories:
with tempfile.TemporaryDirectory() as tmpdirname:
print('created temporary dir:', tmpdirname)
(Contributed by Neil Schemenauer and Nick Coghlan; bpo-5178(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5178
File: python.info, Node: inspect<10>, Next: pydoc<6>, Prev: tempfile<5>, Up: New Improved and Deprecated Modules
1.12.9.38 inspect
.................
* The *note inspect: 7e. module has a new function *note
getgeneratorstate(): 1229. to easily identify the current state of
a generator-iterator:
>>> from inspect import getgeneratorstate
>>> def gen():
... yield 'demo'
...
>>> g = gen()
>>> getgeneratorstate(g)
'GEN_CREATED'
>>> next(g)
'demo'
>>> getgeneratorstate(g)
'GEN_SUSPENDED'
>>> next(g, None)
>>> getgeneratorstate(g)
'GEN_CLOSED'
(Contributed by Rodolpho Eckhardt and Nick Coghlan, bpo-10220(1).)
* To support lookups without the possibility of activating a dynamic
attribute, the *note inspect: 7e. module has a new function, *note
getattr_static(): 490. Unlike *note hasattr(): 4ce, this is a true
read-only search, guaranteed not to change state while it is
searching:
>>> class A:
... @property
... def f(self):
... print('Running')
... return 10
...
>>> a = A()
>>> getattr(a, 'f')
Running
10
>>> inspect.getattr_static(a, 'f')
(Contributed by Michael Foord.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10220
File: python.info, Node: pydoc<6>, Next: dis<5>, Prev: inspect<10>, Up: New Improved and Deprecated Modules
1.12.9.39 pydoc
...............
The *note pydoc: b5. module now provides a much-improved web server
interface, as well as a new command-line option ‘-b’ to automatically
open a browser window to display that server:
$ pydoc3.2 -b
(Contributed by Ron Adam; bpo-2001(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2001
File: python.info, Node: dis<5>, Next: dbm<7>, Prev: pydoc<6>, Up: New Improved and Deprecated Modules
1.12.9.40 dis
.............
The *note dis: 38. module gained two new functions for inspecting code,
*note code_info(): 122c. and *note show_code(): f27. Both provide
detailed code object information for the supplied function, method,
source code string or code object. The former returns a string and the
latter prints it:
>>> import dis, random
>>> dis.show_code(random.choice)
Name: choice
Filename: /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py
Argument count: 2
Kw-only arguments: 0
Number of locals: 3
Stack size: 11
Flags: OPTIMIZED, NEWLOCALS, NOFREE
Constants:
0: 'Choose a random element from a non-empty sequence.'
1: 'Cannot choose from an empty sequence'
Names:
0: _randbelow
1: len
2: ValueError
3: IndexError
Variable names:
0: self
1: seq
2: i
In addition, the *note dis(): b34. function now accepts string arguments
so that the common idiom ‘dis(compile(s, '', 'eval'))’ can be shortened
to ‘dis(s)’:
>>> dis('3*x+1 if x%2==1 else x//2')
1 0 LOAD_NAME 0 (x)
3 LOAD_CONST 0 (2)
6 BINARY_MODULO
7 LOAD_CONST 1 (1)
10 COMPARE_OP 2 (==)
13 POP_JUMP_IF_FALSE 28
16 LOAD_CONST 2 (3)
19 LOAD_NAME 0 (x)
22 BINARY_MULTIPLY
23 LOAD_CONST 1 (1)
26 BINARY_ADD
27 RETURN_VALUE
>> 28 LOAD_NAME 0 (x)
31 LOAD_CONST 0 (2)
34 BINARY_FLOOR_DIVIDE
35 RETURN_VALUE
Taken together, these improvements make it easier to explore how CPython
is implemented and to see for yourself what the language syntax does
under-the-hood.
(Contributed by Nick Coghlan in bpo-9147(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9147
File: python.info, Node: dbm<7>, Next: ctypes<3>, Prev: dis<5>, Up: New Improved and Deprecated Modules
1.12.9.41 dbm
.............
All database modules now support the ‘get()’ and ‘setdefault()’ methods.
(Suggested by Ray Allen in bpo-9523(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9523
File: python.info, Node: ctypes<3>, Next: site<4>, Prev: dbm<7>, Up: New Improved and Deprecated Modules
1.12.9.42 ctypes
................
A new type, *note ctypes.c_ssize_t: 122f. represents the C ‘ssize_t’
datatype.
File: python.info, Node: site<4>, Next: sysconfig<3>, Prev: ctypes<3>, Up: New Improved and Deprecated Modules
1.12.9.43 site
..............
The *note site: c7. module has three new functions useful for reporting
on the details of a given Python installation.
* *note getsitepackages(): 1231. lists all global site-packages
directories.
* *note getuserbase(): 1232. reports on the user’s base directory
where data can be stored.
* *note getusersitepackages(): 1233. reveals the user-specific
site-packages directory path.
>>> import site
>>> site.getsitepackages()
['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages',
'/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python',
'/Library/Python/3.2/site-packages']
>>> site.getuserbase()
'/Users/raymondhettinger/Library/Python/3.2'
>>> site.getusersitepackages()
'/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages'
Conveniently, some of site’s functionality is accessible directly from
the command-line:
$ python -m site --user-base
/Users/raymondhettinger/.local
$ python -m site --user-site
/Users/raymondhettinger/.local/lib/python3.2/site-packages
(Contributed by Tarek Ziadé in bpo-6693(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6693
File: python.info, Node: sysconfig<3>, Next: pdb<8>, Prev: site<4>, Up: New Improved and Deprecated Modules
1.12.9.44 sysconfig
...................
The new *note sysconfig: db. module makes it straightforward to discover
installation paths and configuration variables that vary across
platforms and installations.
The module offers access simple access functions for platform and
version information:
* *note get_platform(): 1235. returning values like 'linux-i586' or
'macosx-10.6-ppc'.
* *note get_python_version(): 1236. returns a Python version string
such as “3.2”.
It also provides access to the paths and variables corresponding to one
of seven named schemes used by ‘distutils’. Those include
'posix_prefix', 'posix_home', 'posix_user', 'nt', 'nt_user', 'os2',
'os2_home':
* *note get_paths(): 1237. makes a dictionary containing installation
paths for the current installation scheme.
* *note get_config_vars(): 1025. returns a dictionary of platform
specific variables.
There is also a convenient command-line interface:
C:\Python32>python -m sysconfig
Platform: "win32"
Python version: "3.2"
Current installation scheme: "nt"
Paths:
data = "C:\Python32"
include = "C:\Python32\Include"
platinclude = "C:\Python32\Include"
platlib = "C:\Python32\Lib\site-packages"
platstdlib = "C:\Python32\Lib"
purelib = "C:\Python32\Lib\site-packages"
scripts = "C:\Python32\Scripts"
stdlib = "C:\Python32\Lib"
Variables:
BINDIR = "C:\Python32"
BINLIBDEST = "C:\Python32\Lib"
EXE = ".exe"
INCLUDEPY = "C:\Python32\Include"
LIBDEST = "C:\Python32\Lib"
SO = ".pyd"
VERSION = "32"
abiflags = ""
base = "C:\Python32"
exec_prefix = "C:\Python32"
platbase = "C:\Python32"
prefix = "C:\Python32"
projectbase = "C:\Python32"
py_version = "3.2"
py_version_nodot = "32"
py_version_short = "3.2"
srcdir = "C:\Python32"
userbase = "C:\Documents and Settings\Raymond\Application Data\Python"
(Moved out of Distutils by Tarek Ziadé.)
File: python.info, Node: pdb<8>, Next: configparser<5>, Prev: sysconfig<3>, Up: New Improved and Deprecated Modules
1.12.9.45 pdb
.............
The *note pdb: a5. debugger module gained a number of usability
improvements:
* ‘pdb.py’ now has a ‘-c’ option that executes commands as given in a
‘.pdbrc’ script file.
* A ‘.pdbrc’ script file can contain ‘continue’ and ‘next’ commands
that continue debugging.
* The *note Pdb: 921. class constructor now accepts a 'nosigint'
argument.
* New commands: ‘l(list)’, ‘ll(long list)’ and ‘source’ for listing
source code.
* New commands: ‘display’ and ‘undisplay’ for showing or hiding the
value of an expression if it has changed.
* New command: ‘interact’ for starting an interactive interpreter
containing the global and local names found in the current scope.
* Breakpoints can be cleared by breakpoint number.
(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)
File: python.info, Node: configparser<5>, Next: urllib parse<6>, Prev: pdb<8>, Up: New Improved and Deprecated Modules
1.12.9.46 configparser
......................
The *note configparser: 22. module was modified to improve usability and
predictability of the default parser and its supported INI syntax. The
old ‘ConfigParser’ class was removed in favor of ‘SafeConfigParser’
which has in turn been renamed to *note ConfigParser: 1c8. Support for
inline comments is now turned off by default and section or option
duplicates are not allowed in a single configuration source.
Config parsers gained a new API based on the mapping protocol:
>>> parser = ConfigParser()
>>> parser.read_string("""
... [DEFAULT]
... location = upper left
... visible = yes
... editable = no
... color = blue
...
... [main]
... title = Main Menu
... color = green
...
... [options]
... title = Options
... """)
>>> parser['main']['color']
'green'
>>> parser['main']['editable']
'no'
>>> section = parser['options']
>>> section['title']
'Options'
>>> section['title'] = 'Options (editable: %(editable)s)'
>>> section['title']
'Options (editable: no)'
The new API is implemented on top of the classical API, so custom parser
subclasses should be able to use it without modifications.
The INI file structure accepted by config parsers can now be customized.
Users can specify alternative option/value delimiters and comment
prefixes, change the name of the 'DEFAULT' section or switch the
interpolation syntax.
There is support for pluggable interpolation including an additional
interpolation handler *note ExtendedInterpolation: 71e.:
>>> parser = ConfigParser(interpolation=ExtendedInterpolation())
>>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'},
... 'custom': {'prefix': '/usr/local'}})
>>> parser.read_string("""
... [buildout]
... parts =
... zope9
... instance
... find-links =
... ${buildout:directory}/downloads/dist
...
... [zope9]
... recipe = plone.recipe.zope9install
... location = /opt/zope
...
... [instance]
... recipe = plone.recipe.zope9instance
... zope9-location = ${zope9:location}
... zope-conf = ${custom:prefix}/etc/zope.conf
... """)
>>> parser['buildout']['find-links']
'\n/home/ambv/zope9/downloads/dist'
>>> parser['instance']['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance = parser['instance']
>>> instance['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance['zope9-location']
'/opt/zope'
A number of smaller features were also introduced, like support for
specifying encoding in read operations, specifying fallback values for
get-functions, or reading directly from dictionaries and strings.
(All changes contributed by Łukasz Langa.)
File: python.info, Node: urllib parse<6>, Next: mailbox, Prev: configparser<5>, Up: New Improved and Deprecated Modules
1.12.9.47 urllib.parse
......................
A number of usability improvements were made for the *note urllib.parse:
10a. module.
The *note urlparse(): 307. function now supports IPv6(1) addresses as
described in RFC 2732(2):
>>> import urllib.parse
>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/')
ParseResult(scheme='http',
netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]',
path='/foo/',
params='',
query='',
fragment='')
The *note urldefrag(): 123b. function now returns a *note named tuple:
64a.:
>>> r = urllib.parse.urldefrag('http://python.org/about/#target')
>>> r
DefragResult(url='http://python.org/about/', fragment='target')
>>> r[0]
'http://python.org/about/'
>>> r.fragment
'target'
And, the *note urlencode(): e8e. function is now much more flexible,
accepting either a string or bytes type for the 'query' argument. If it
is a string, then the 'safe', 'encoding', and 'error' parameters are
sent to *note quote_plus(): 123c. for encoding:
>>> urllib.parse.urlencode([
... ('type', 'telenovela'),
... ('name', '¿Dónde Está Elisa?')],
... encoding='latin-1')
'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F'
As detailed in *note Parsing ASCII Encoded Bytes: 123d, all the *note
urllib.parse: 10a. functions now accept ASCII-encoded byte strings as
input, so long as they are not mixed with regular strings. If
ASCII-encoded byte strings are given as parameters, the return types
will also be an ASCII-encoded byte strings:
>>> urllib.parse.urlparse(b'http://www.python.org:80/about/')
ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80',
path=b'/about/', params=b'', query=b'', fragment=b'')
(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in bpo-2987(3),
bpo-5468(4), and bpo-9873(5).)
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/IPv6
(2) https://datatracker.ietf.org/doc/html/rfc2732.html
(3) https://bugs.python.org/issue?@action=redirect&bpo=2987
(4) https://bugs.python.org/issue?@action=redirect&bpo=5468
(5) https://bugs.python.org/issue?@action=redirect&bpo=9873
File: python.info, Node: mailbox, Next: turtledemo, Prev: urllib parse<6>, Up: New Improved and Deprecated Modules
1.12.9.48 mailbox
.................
Thanks to a concerted effort by R. David Murray, the *note mailbox: 8b.
module has been fixed for Python 3.2. The challenge was that mailbox
had been originally designed with a text interface, but email messages
are best represented with *note bytes: 1c2. because various parts of a
message may have different encodings.
The solution harnessed the *note email: 3b. package’s binary support for
parsing arbitrary email messages. In addition, the solution required a
number of API changes.
As expected, the *note add(): 123f. method for *note mailbox.Mailbox:
1240. objects now accepts binary input.
*note StringIO: f22. and text file input are deprecated. Also, string
input will fail early if non-ASCII characters are used. Previously it
would fail when the email was processed in a later step.
There is also support for binary output. The *note get_file(): 1241.
method now returns a file in the binary mode (where it used to
incorrectly set the file to text-mode). There is also a new *note
get_bytes(): 1242. method that returns a *note bytes: 1c2.
representation of a message corresponding to a given 'key'.
It is still possible to get non-binary output using the old API’s *note
get_string(): 1243. method, but that approach is not very useful.
Instead, it is best to extract messages from a *note Message: 1244.
object or to load them from binary input.
(Contributed by R. David Murray, with efforts from Steffen Daode
Nurpmeso and an initial patch by Victor Stinner in bpo-9124(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=9124
File: python.info, Node: turtledemo, Prev: mailbox, Up: New Improved and Deprecated Modules
1.12.9.49 turtledemo
....................
The demonstration code for the *note turtle: 101. module was moved from
the 'Demo' directory to main library. It includes over a dozen sample
scripts with lively displays. Being on *note sys.path: 3b0, it can now
be run directly from the command-line:
$ python -m turtledemo
(Moved from the Demo directory by Alexander Belopolsky in bpo-10199(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10199
File: python.info, Node: Multi-threading, Next: Optimizations<11>, Prev: New Improved and Deprecated Modules, Up: What’s New In Python 3 2
1.12.10 Multi-threading
-----------------------
* The mechanism for serializing execution of concurrently running
Python threads (generally known as the *note GIL: 159. or Global
Interpreter Lock) has been rewritten. Among the objectives were
more predictable switching intervals and reduced overhead due to
lock contention and the number of ensuing system calls. The notion
of a “check interval” to allow thread switches has been abandoned
and replaced by an absolute duration expressed in seconds. This
parameter is tunable through *note sys.setswitchinterval(): 94a.
It currently defaults to 5 milliseconds.
Additional details about the implementation can be read from a
python-dev mailing-list message(1) (however, “priority requests” as
exposed in this message have not been kept for inclusion).
(Contributed by Antoine Pitrou.)
* Regular and recursive locks now accept an optional 'timeout'
argument to their *note acquire(): 694. method. (Contributed by
Antoine Pitrou; bpo-7316(2).)
* Similarly, *note threading.Semaphore.acquire(): 1247. also gained a
'timeout' argument. (Contributed by Torsten Landschoff;
bpo-850728(3).)
* Regular and recursive lock acquisitions can now be interrupted by
signals on platforms using Pthreads. This means that Python
programs that deadlock while acquiring locks can be successfully
killed by repeatedly sending SIGINT to the process (by pressing
‘Ctrl’+‘C’ in most shells). (Contributed by Reid Kleckner;
bpo-8844(4).)
---------- Footnotes ----------
(1)
https://mail.python.org/pipermail/python-dev/2009-October/093321.html
(2) https://bugs.python.org/issue?@action=redirect&bpo=7316
(3) https://bugs.python.org/issue?@action=redirect&bpo=850728
(4) https://bugs.python.org/issue?@action=redirect&bpo=8844
File: python.info, Node: Optimizations<11>, Next: Unicode, Prev: Multi-threading, Up: What’s New In Python 3 2
1.12.11 Optimizations
---------------------
A number of small performance enhancements have been added:
* Python’s peephole optimizer now recognizes patterns such ‘x in {1,
2, 3}’ as being a test for membership in a set of constants. The
optimizer recasts the *note set: 5d5. as a *note frozenset: 5d6.
and stores the pre-built constant.
Now that the speed penalty is gone, it is practical to start
writing membership tests using set-notation. This style is both
semantically clear and operationally fast:
extension = name.rpartition('.')[2]
if extension in {'xml', 'html', 'xhtml', 'css'}:
handle(name)
(Patch and additional tests contributed by Dave Malcolm;
bpo-6690(1)).
* Serializing and unserializing data using the *note pickle: a6.
module is now several times faster.
(Contributed by Alexandre Vassalotti, Antoine Pitrou and the
Unladen Swallow team in bpo-9410(2) and bpo-3873(3).)
* The Timsort algorithm(4) used in *note list.sort(): bcf. and *note
sorted(): bce. now runs faster and uses less memory when called
with a *note key function: e04. Previously, every element of a
list was wrapped with a temporary object that remembered the key
value associated with each element. Now, two arrays of keys and
values are sorted in parallel. This saves the memory consumed by
the sort wrappers, and it saves time lost to delegating
comparisons.
(Patch by Daniel Stutzbach in bpo-9915(5).)
* JSON decoding performance is improved and memory consumption is
reduced whenever the same string is repeated for multiple keys.
Also, JSON encoding now uses the C speedups when the ‘sort_keys’
argument is true.
(Contributed by Antoine Pitrou in bpo-7451(6) and by Raymond
Hettinger and Antoine Pitrou in bpo-10314(7).)
* Recursive locks (created with the *note threading.RLock(): 2e2.
API) now benefit from a C implementation which makes them as fast
as regular locks, and between 10x and 15x faster than their
previous pure Python implementation.
(Contributed by Antoine Pitrou; bpo-3001(8).)
* The fast-search algorithm in stringlib is now used by the *note
split(): ea3, *note rsplit(): 1249, *note splitlines(): 124a. and
*note replace(): 19a. methods on *note bytes: 1c2, *note bytearray:
53a. and *note str: 447. objects. Likewise, the algorithm is also
used by *note rfind(): ea2, *note rindex(): 124b, *note rsplit():
1249. and *note rpartition(): 124c.
(Patch by Florent Xicluna in bpo-7622(9) and bpo-7462(10).)
* Integer to string conversions now work two “digits” at a time,
reducing the number of division and modulo operations.
(bpo-6713(11) by Gawain Bolton, Mark Dickinson, and Victor
Stinner.)
There were several other minor optimizations. Set differencing now runs
faster when one operand is much larger than the other (patch by Andress
Bennetts in bpo-8685(12)). The ‘array.repeat()’ method has a faster
implementation (bpo-1569291(13) by Alexander Belopolsky). The *note
BaseHTTPRequestHandler: 10c1. has more efficient buffering (bpo-3709(14)
by Andrew Schaaf). The *note operator.attrgetter(): e32. function has
been sped-up (bpo-10160(15) by Christos Georgiou). And *note
ConfigParser: 1c8. loads multi-line arguments a bit faster (bpo-7113(16)
by Łukasz Langa).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6690
(2) https://bugs.python.org/issue?@action=redirect&bpo=9410
(3) https://bugs.python.org/issue?@action=redirect&bpo=3873
(4) https://en.wikipedia.org/wiki/Timsort
(5) https://bugs.python.org/issue?@action=redirect&bpo=9915
(6) https://bugs.python.org/issue?@action=redirect&bpo=7451
(7) https://bugs.python.org/issue?@action=redirect&bpo=10314
(8) https://bugs.python.org/issue?@action=redirect&bpo=3001
(9) https://bugs.python.org/issue?@action=redirect&bpo=7622
(10) https://bugs.python.org/issue?@action=redirect&bpo=7462
(11) https://bugs.python.org/issue?@action=redirect&bpo=6713
(12) https://bugs.python.org/issue?@action=redirect&bpo=8685
(13) https://bugs.python.org/issue?@action=redirect&bpo=1569291
(14) https://bugs.python.org/issue?@action=redirect&bpo=3709
(15) https://bugs.python.org/issue?@action=redirect&bpo=10160
(16) https://bugs.python.org/issue?@action=redirect&bpo=7113
File: python.info, Node: Unicode, Next: Codecs, Prev: Optimizations<11>, Up: What’s New In Python 3 2
1.12.12 Unicode
---------------
Python has been updated to Unicode 6.0.0(1). The update to the standard
adds over 2,000 new characters including emoji(2) symbols which are
important for mobile phones.
In addition, the updated standard has altered the character properties
for two Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric
character (U+19DA), making the former eligible for use in identifiers
while disqualifying the latter. For more information, see Unicode
Character Database Changes(3).
---------- Footnotes ----------
(1) https://unicode.org/versions/Unicode6.0.0/
(2) https://en.wikipedia.org/wiki/Emoji
(3) https://www.unicode.org/versions/Unicode6.0.0/#Database_Changes
File: python.info, Node: Codecs, Next: Documentation, Prev: Unicode, Up: What’s New In Python 3 2
1.12.13 Codecs
--------------
Support was added for 'cp720' Arabic DOS encoding (bpo-1616979(1)).
MBCS encoding no longer ignores the error handler argument. In the
default strict mode, it raises an *note UnicodeDecodeError: a12. when it
encounters an undecodable byte sequence and an *note UnicodeEncodeError:
673. for an unencodable character.
The MBCS codec supports ‘'strict'’ and ‘'ignore'’ error handlers for
decoding, and ‘'strict'’ and ‘'replace'’ for encoding.
To emulate Python3.1 MBCS encoding, select the ‘'ignore'’ handler for
decoding and the ‘'replace'’ handler for encoding.
On Mac OS X, Python decodes command line arguments with ‘'utf-8'’ rather
than the locale encoding.
By default, *note tarfile: de. uses ‘'utf-8'’ encoding on Windows
(instead of ‘'mbcs'’) and the ‘'surrogateescape'’ error handler on all
operating systems.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1616979
File: python.info, Node: Documentation, Next: IDLE, Prev: Codecs, Up: What’s New In Python 3 2
1.12.14 Documentation
---------------------
The documentation continues to be improved.
* A table of quick links has been added to the top of lengthy
sections such as *note Built-in Functions: 1250. In the case of
*note itertools: 81, the links are accompanied by tables of
cheatsheet-style summaries to provide an overview and memory jog
without having to read all of the docs.
* In some cases, the pure Python source code can be a helpful adjunct
to the documentation, so now many modules now feature quick links
to the latest version of the source code. For example, the *note
functools: 5f. module documentation has a quick link at the top
labeled:
'Source code' Lib/functools.py(1).
(Contributed by Raymond Hettinger; see rationale(2).)
* The docs now contain more examples and recipes. In particular,
*note re: b9. module has an extensive section, *note Regular
Expression Examples: 1251. Likewise, the *note itertools: 81.
module continues to be updated with new *note Itertools Recipes:
1252.
* The *note datetime: 30. module now has an auxiliary implementation
in pure Python. No functionality was changed. This just provides
an easier-to-read alternate implementation.
(Contributed by Alexander Belopolsky in bpo-9528(3).)
* The unmaintained ‘Demo’ directory has been removed. Some demos
were integrated into the documentation, some were moved to the
‘Tools/demo’ directory, and others were removed altogether.
(Contributed by Georg Brandl in bpo-7962(4).)
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/functools.py
(2)
https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/
(3) https://bugs.python.org/issue?@action=redirect&bpo=9528
(4) https://bugs.python.org/issue?@action=redirect&bpo=7962
File: python.info, Node: IDLE, Next: Code Repository, Prev: Documentation, Up: What’s New In Python 3 2
1.12.15 IDLE
------------
* The format menu now has an option to clean source files by
stripping trailing whitespace.
(Contributed by Raymond Hettinger; bpo-5150(1).)
* IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa
AquaTk.
(Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren;
bpo-6075(2).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5150
(2) https://bugs.python.org/issue?@action=redirect&bpo=6075
File: python.info, Node: Code Repository, Next: Build and C API Changes<5>, Prev: IDLE, Up: What’s New In Python 3 2
1.12.16 Code Repository
-----------------------
In addition to the existing Subversion code repository at
‘https://svn.python.org’ there is now a Mercurial(1) repository at
‘https://hg.python.org/’.
After the 3.2 release, there are plans to switch to Mercurial as the
primary repository. This distributed version control system should make
it easier for members of the community to create and share external
changesets. See PEP 385(2) for details.
To learn to use the new version control system, see the Quick Start(3)
or the Guide to Mercurial Workflows(4).
---------- Footnotes ----------
(1) https://www.mercurial-scm.org/
(2) https://peps.python.org/pep-0385/
(3) https://www.mercurial-scm.org/wiki/QuickStart
(4) https://www.mercurial-scm.org/guide
File: python.info, Node: Build and C API Changes<5>, Next: Porting to Python 3 2, Prev: Code Repository, Up: What’s New In Python 3 2
1.12.17 Build and C API Changes
-------------------------------
Changes to Python’s build process and to the C API include:
* The 'idle', 'pydoc' and '2to3' scripts are now installed with a
version-specific suffix on ‘make altinstall’ (bpo-10679(1)).
* The C functions that access the Unicode Database now accept and
return characters from the full Unicode range, even on narrow
unicode builds (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and
others). A visible difference in Python is that *note
unicodedata.numeric(): 1256. now returns the correct value for
large code points, and *note repr(): 7f9. may consider more
characters as printable.
(Reported by Bupjoe Lee and fixed by Amaury Forgeot D’Arc;
bpo-5127(2).)
* Computed gotos are now enabled by default on supported compilers
(which are detected by the configure script). They can still be
disabled selectively by specifying ‘--without-computed-gotos’.
(Contributed by Antoine Pitrou; bpo-9203(3).)
* The option ‘--with-wctype-functions’ was removed. The built-in
unicode database is now used for all functions.
(Contributed by Amaury Forgeot D’Arc; bpo-9210(4).)
* Hash values are now values of a new type, *note Py_hash_t: 1257,
which is defined to be the same size as a pointer. Previously they
were of type long, which on some 64-bit operating systems is still
only 32 bits long. As a result of this fix, *note set: 5d5. and
*note dict: 258. can now hold more than ‘2**32’ entries on builds
with 64-bit pointers (previously, they could grow to that size but
their performance degraded catastrophically).
(Suggested by Raymond Hettinger and implemented by Benjamin
Peterson; bpo-9778(5).)
* A new macro ‘Py_VA_COPY’ copies the state of the variable argument
list. It is equivalent to C99 'va_copy' but available on all
Python platforms (bpo-2443(6)).
* A new C API function ‘PySys_SetArgvEx()’ allows an embedded
interpreter to set *note sys.argv: 1258. without also modifying
*note sys.path: 3b0. (bpo-5753(7)).
* ‘PyEval_CallObject()’ is now only available in macro form. The
function declaration, which was kept for backwards compatibility
reasons, is now removed – the macro was introduced in 1997
(bpo-8276(8)).
* There is a new function *note PyLong_AsLongLongAndOverflow(): 1259.
which is analogous to *note PyLong_AsLongAndOverflow(): 125a. They
both serve to convert Python *note int: 259. into a native
fixed-width type while providing detection of cases where the
conversion won’t fit (bpo-7767(9)).
* The *note PyUnicode_CompareWithASCIIString(): 125b. function now
returns 'not equal' if the Python string is 'NUL' terminated.
* There is a new function *note PyErr_NewExceptionWithDoc(): 125c.
that is like *note PyErr_NewException(): 125d. but allows a
docstring to be specified. This lets C exceptions have the same
self-documenting capabilities as their pure Python counterparts
(bpo-7033(10)).
* When compiled with the ‘--with-valgrind’ option, the pymalloc
allocator will be automatically disabled when running under
Valgrind. This gives improved memory leak detection when running
under Valgrind, while taking advantage of pymalloc at other times
(bpo-2422(11)).
* Removed the ‘O?’ format from the 'PyArg_Parse' functions. The
format is no longer used and it had never been documented
(bpo-8837(12)).
There were a number of other small changes to the C-API. See the
Misc/NEWS(13) file for a complete list.
Also, there were a number of updates to the Mac OS X build, see
Mac/BuildScript/README.txt(14) for details. For users running a
32/64-bit build, there is a known problem with the default Tcl/Tk on Mac
OS X 10.6. Accordingly, we recommend installing an updated alternative
such as ActiveState Tcl/Tk 8.5.9(15). See
‘https://www.python.org/download/mac/tcltk/’ for additional details.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=10679
(2) https://bugs.python.org/issue?@action=redirect&bpo=5127
(3) https://bugs.python.org/issue?@action=redirect&bpo=9203
(4) https://bugs.python.org/issue?@action=redirect&bpo=9210
(5) https://bugs.python.org/issue?@action=redirect&bpo=9778
(6) https://bugs.python.org/issue?@action=redirect&bpo=2443
(7) https://bugs.python.org/issue?@action=redirect&bpo=5753
(8) https://bugs.python.org/issue?@action=redirect&bpo=8276
(9) https://bugs.python.org/issue?@action=redirect&bpo=7767
(10) https://bugs.python.org/issue?@action=redirect&bpo=7033
(11) https://bugs.python.org/issue?@action=redirect&bpo=2422
(12) https://bugs.python.org/issue?@action=redirect&bpo=8837
(13) https://github.com/python/cpython/blob/v3.2.6/Misc/NEWS
(14)
https://github.com/python/cpython/blob/v3.2.6/Mac/BuildScript/README.txt
(15)
https://web.archive.org/web/20101208191259/https://www.activestate.com/activetcl/downloads
File: python.info, Node: Porting to Python 3 2, Prev: Build and C API Changes<5>, Up: What’s New In Python 3 2
1.12.18 Porting to Python 3.2
-----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code:
* The *note configparser: 22. module has a number of clean-ups. The
major change is to replace the old ‘ConfigParser’ class with
long-standing preferred alternative ‘SafeConfigParser’. In
addition there are a number of smaller incompatibilities:
* The interpolation syntax is now validated on *note get():
125f. and *note set(): 1260. operations. In the default
interpolation scheme, only two tokens with percent signs are
valid: ‘%(name)s’ and ‘%%’, the latter being an escaped
percent sign.
* The *note set(): 1260. and *note add_section(): 1261. methods
now verify that values are actual strings. Formerly,
unsupported types could be introduced unintentionally.
* Duplicate sections or options from a single source now raise
either *note DuplicateSectionError: 1262. or *note
DuplicateOptionError: 1263. Formerly, duplicates would
silently overwrite a previous entry.
* Inline comments are now disabled by default so now the ';'
character can be safely used in values.
* Comments now can be indented. Consequently, for ';' or '#' to
appear at the start of a line in multiline values, it has to
be interpolated. This keeps comment prefix characters in
values from being mistaken as comments.
* ‘""’ is now a valid value and is no longer automatically
converted to an empty string. For empty strings, use ‘"option
="’ in a line.
* The ‘nntplib’ module was reworked extensively, meaning that its
APIs are often incompatible with the 3.1 APIs.
* *note bytearray: 53a. objects can no longer be used as filenames;
instead, they should be converted to *note bytes: 1c2.
* The ‘array.tostring()’ and ‘array.fromstring()’ have been renamed
to *note array.tobytes(): 1264. and *note array.frombytes(): 1265.
for clarity. The old names have been deprecated. (See
bpo-8990(1).)
* ‘PyArg_Parse*()’ functions:
* “t#” format has been removed: use “s#” or “s*” instead
* “w” and “w#” formats has been removed: use “w*” instead
* The ‘PyCObject’ type, deprecated in 3.1, has been removed. To wrap
opaque C pointers in Python objects, the *note PyCapsule: 1266. API
should be used instead; the new type has a well-defined interface
for passing typing safety information and a less complicated
signature for calling a destructor.
* The ‘sys.setfilesystemencoding()’ function was removed because it
had a flawed design.
* The *note random.seed(): 1267. function and method now salt string
seeds with an sha512 hash function. To access the previous version
of 'seed' in order to reproduce Python 3.1 sequences, set the
'version' argument to '1', ‘random.seed(s, version=1)’.
* The previously deprecated ‘string.maketrans()’ function has been
removed in favor of the static methods *note bytes.maketrans():
1268. and *note bytearray.maketrans(): 1269. This change solves
the confusion around which types were supported by the *note
string: d3. module. Now, *note str: 447, *note bytes: 1c2, and
*note bytearray: 53a. each have their own 'maketrans' and
'translate' methods with intermediate translation tables of the
appropriate type.
(Contributed by Georg Brandl; bpo-5675(2).)
* The previously deprecated ‘contextlib.nested()’ function has been
removed in favor of a plain *note with: 5ce. statement which can
accept multiple context managers. The latter technique is faster
(because it is built-in), and it does a better job finalizing
multiple context managers when one of them raises an exception:
with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
for line in infile:
if '' in line:
outfile.write(line)
(Contributed by Georg Brandl and Mattias Brändström; appspot issue
53094(3).)
* *note struct.pack(): 126a. now only allows bytes for the ‘s’ string
pack code. Formerly, it would accept text arguments and implicitly
encode them to bytes using UTF-8. This was problematic because it
made assumptions about the correct encoding and because a
variable-length encoding can fail when writing to fixed length
segment of a structure.
Code such as ‘struct.pack('<6sHHBBB', 'GIF87a', x, y)’ should be
rewritten with to use bytes instead of text,
‘struct.pack('<6sHHBBB', b'GIF87a', x, y)’.
(Discovered by David Beazley and fixed by Victor Stinner;
bpo-10783(4).)
* The *note xml.etree.ElementTree: 125. class now raises an *note
xml.etree.ElementTree.ParseError: 126b. when a parse fails.
Previously it raised an *note xml.parsers.expat.ExpatError: 126c.
* The new, longer *note str(): 447. value on floats may break
doctests which rely on the old output format.
* In *note subprocess.Popen: 199, the default value for 'close_fds'
is now ‘True’ under Unix; under Windows, it is ‘True’ if the three
standard streams are set to ‘None’, ‘False’ otherwise. Previously,
'close_fds' was always ‘False’ by default, which produced difficult
to solve bugs or race conditions when open file descriptors would
leak into the child process.
* Support for legacy HTTP 0.9 has been removed from *note
urllib.request: 10b. and *note http.client: 6f. Such support is
still present on the server side (in *note http.server: 72.).
(Contributed by Antoine Pitrou, bpo-10711(5).)
* SSL sockets in timeout mode now raise *note socket.timeout: 830.
when a timeout occurs, rather than a generic *note SSLError: 126d.
(Contributed by Antoine Pitrou, bpo-10272(6).)
* The misleading functions ‘PyEval_AcquireLock()’ and
‘PyEval_ReleaseLock()’ have been officially deprecated. The
thread-state aware APIs (such as *note PyEval_SaveThread(): 3a4.
and *note PyEval_RestoreThread(): 3a5.) should be used instead.
* Due to security risks, ‘asyncore.handle_accept()’ has been
deprecated, and a new function, ‘asyncore.handle_accepted()’, was
added to replace it.
(Contributed by Giampaolo Rodola in bpo-6706(7).)
* Due to the new *note GIL: 159. implementation,
‘PyEval_InitThreads()’ cannot be called before *note
Py_Initialize(): 8ab. anymore.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8990
(2) https://bugs.python.org/issue?@action=redirect&bpo=5675
(3) https://codereview.appspot.com/53094
(4) https://bugs.python.org/issue?@action=redirect&bpo=10783
(5) https://bugs.python.org/issue?@action=redirect&bpo=10711
(6) https://bugs.python.org/issue?@action=redirect&bpo=10272
(7) https://bugs.python.org/issue?@action=redirect&bpo=6706
File: python.info, Node: What’s New In Python 3 1, Next: What’s New In Python 3 0, Prev: What’s New In Python 3 2, Up: What’s New in Python
1.13 What’s New In Python 3.1
=============================
Author: Raymond Hettinger
This article explains the new features in Python 3.1, compared to 3.0.
Python 3.1 was released on June 27, 2009.
* Menu:
* PEP 372; Ordered Dictionaries: PEP 372 Ordered Dictionaries.
* PEP 378; Format Specifier for Thousands Separator: PEP 378 Format Specifier for Thousands Separator.
* Other Language Changes: Other Language Changes<13>.
* New, Improved, and Deprecated Modules: New Improved and Deprecated Modules<2>.
* Optimizations: Optimizations<12>.
* IDLE: IDLE<2>.
* Build and C API Changes: Build and C API Changes<6>.
* Porting to Python 3.1: Porting to Python 3 1.
File: python.info, Node: PEP 372 Ordered Dictionaries, Next: PEP 378 Format Specifier for Thousands Separator, Up: What’s New In Python 3 1
1.13.1 PEP 372: Ordered Dictionaries
------------------------------------
Regular Python dictionaries iterate over key/value pairs in arbitrary
order. Over the years, a number of authors have written alternative
implementations that remember the order that the keys were originally
inserted. Based on the experiences from those implementations, a new
*note collections.OrderedDict: 5d7. class has been introduced.
The OrderedDict API is substantially the same as regular dictionaries
but will iterate over keys and values in a guaranteed order depending on
when a key was first inserted. If a new entry overwrites an existing
entry, the original insertion position is left unchanged. Deleting an
entry and reinserting it will move it to the end.
The standard library now supports use of ordered dictionaries in several
modules. The *note configparser: 22. module uses them by default. This
lets configuration files be read, modified, and then written back in
their original order. The '_asdict()' method for *note
collections.namedtuple(): 1ca. now returns an ordered dictionary with
the values appearing in the same order as the underlying tuple indices.
The *note json: 82. module is being built-out with an
'object_pairs_hook' to allow OrderedDicts to be built by the decoder.
Support was also added for third-party tools like PyYAML(1).
See also
........
PEP 372(2) - Ordered Dictionaries
PEP written by Armin Ronacher and Raymond Hettinger.
Implementation written by Raymond Hettinger.
Since an ordered dictionary remembers its insertion order, it can be
used in conjunction with sorting to make a sorted dictionary:
>>> # regular unsorted dictionary
>>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2}
>>> # dictionary sorted by key
>>> OrderedDict(sorted(d.items(), key=lambda t: t[0]))
OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)])
>>> # dictionary sorted by value
>>> OrderedDict(sorted(d.items(), key=lambda t: t[1]))
OrderedDict([('pear', 1), ('orange', 2), ('banana', 3), ('apple', 4)])
>>> # dictionary sorted by length of the key string
>>> OrderedDict(sorted(d.items(), key=lambda t: len(t[0])))
OrderedDict([('pear', 1), ('apple', 4), ('orange', 2), ('banana', 3)])
The new sorted dictionaries maintain their sort order when entries are
deleted. But when new keys are added, the keys are appended to the end
and the sort is not maintained.
---------- Footnotes ----------
(1) https://pyyaml.org/
(2) https://peps.python.org/pep-0372/
File: python.info, Node: PEP 378 Format Specifier for Thousands Separator, Next: Other Language Changes<13>, Prev: PEP 372 Ordered Dictionaries, Up: What’s New In Python 3 1
1.13.2 PEP 378: Format Specifier for Thousands Separator
--------------------------------------------------------
The built-in *note format(): 61b. function and the *note str.format():
61d. method use a mini-language that now includes a simple, non-locale
aware way to format a number with a thousands separator. That provides
a way to humanize a program’s output, improving its professional
appearance and readability:
>>> format(1234567, ',d')
'1,234,567'
>>> format(1234567.89, ',.2f')
'1,234,567.89'
>>> format(12345.6 + 8901234.12j, ',f')
'12,345.600000+8,901,234.120000j'
>>> format(Decimal('1234567.89'), ',f')
'1,234,567.89'
The supported types are *note int: 259, *note float: 2f1, *note complex:
2f2. and *note decimal.Decimal: 29f.
Discussions are underway about how to specify alternative separators
like dots, spaces, apostrophes, or underscores. Locale-aware
applications should use the existing 'n' format specifier which already
has some support for thousands separators.
See also
........
PEP 378(1) - Format Specifier for Thousands Separator
PEP written by Raymond Hettinger and implemented by Eric Smith and
Mark Dickinson.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0378/
File: python.info, Node: Other Language Changes<13>, Next: New Improved and Deprecated Modules<2>, Prev: PEP 378 Format Specifier for Thousands Separator, Up: What’s New In Python 3 1
1.13.3 Other Language Changes
-----------------------------
Some smaller changes made to the core Python language are:
* Directories and zip archives containing a ‘__main__.py’ file can
now be executed directly by passing their name to the interpreter.
The directory/zipfile is automatically inserted as the first entry
in sys.path. (Suggestion and initial patch by Andy Chu; revised
patch by Phillip J. Eby and Nick Coghlan; bpo-1739468(1).)
* The *note int(): 259. type gained a ‘bit_length’ method that
returns the number of bits necessary to represent its argument in
binary:
>>> n = 37
>>> bin(37)
'0b100101'
>>> n.bit_length()
6
>>> n = 2**123-1
>>> n.bit_length()
123
>>> (n+1).bit_length()
124
(Contributed by Fredrik Johansson, Victor Stinner, Raymond
Hettinger, and Mark Dickinson; bpo-3439(2).)
* The fields in *note format(): 61b. strings can now be automatically
numbered:
>>> 'Sir {} of {}'.format('Gallahad', 'Camelot')
'Sir Gallahad of Camelot'
Formerly, the string would have required numbered fields such as:
‘'Sir {0} of {1}'’.
(Contributed by Eric Smith; bpo-5237(3).)
* The ‘string.maketrans()’ function is deprecated and is replaced by
new static methods, *note bytes.maketrans(): 1268. and *note
bytearray.maketrans(): 1269. This change solves the confusion
around which types were supported by the *note string: d3. module.
Now, *note str: 447, *note bytes: 1c2, and *note bytearray: 53a.
each have their own 'maketrans' and 'translate' methods with
intermediate translation tables of the appropriate type.
(Contributed by Georg Brandl; bpo-5675(4).)
* The syntax of the *note with: 5ce. statement now allows multiple
context managers in a single statement:
>>> with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
... for line in infile:
... if '' in line:
... outfile.write(line)
With the new syntax, the ‘contextlib.nested()’ function is no
longer needed and is now deprecated.
(Contributed by Georg Brandl and Mattias Brändström; appspot issue
53094(5).)
* ‘round(x, n)’ now returns an integer if 'x' is an integer.
Previously it returned a float:
>>> round(1123, -2)
1100
(Contributed by Mark Dickinson; bpo-4707(6).)
* Python now uses David Gay’s algorithm for finding the shortest
floating-point representation that doesn’t change its value. This
should help mitigate some of the confusion surrounding binary
floating-point numbers.
The significance is easily seen with a number like ‘1.1’ which does
not have an exact equivalent in binary floating point. Since there
is no exact equivalent, an expression like ‘float('1.1')’ evaluates
to the nearest representable value which is ‘0x1.199999999999ap+0’
in hex or ‘1.100000000000000088817841970012523233890533447265625’
in decimal. That nearest value was and still is used in subsequent
floating-point calculations.
What is new is how the number gets displayed. Formerly, Python
used a simple approach. The value of ‘repr(1.1)’ was computed as
‘format(1.1, '.17g')’ which evaluated to ‘'1.1000000000000001'’.
The advantage of using 17 digits was that it relied on IEEE-754
guarantees to assure that ‘eval(repr(1.1))’ would round-trip
exactly to its original value. The disadvantage is that many
people found the output to be confusing (mistaking intrinsic
limitations of binary floating-point representation as being a
problem with Python itself).
The new algorithm for ‘repr(1.1)’ is smarter and returns ‘'1.1'’.
Effectively, it searches all equivalent string representations
(ones that get stored with the same underlying float value) and
returns the shortest representation.
The new algorithm tends to emit cleaner representations when
possible, but it does not change the underlying values. So, it is
still the case that ‘1.1 + 2.2 != 3.3’ even though the
representations may suggest otherwise.
The new algorithm depends on certain features in the underlying
floating-point implementation. If the required features are not
found, the old algorithm will continue to be used. Also, the text
pickle protocols assure cross-platform portability by using the old
algorithm.
(Contributed by Eric Smith and Mark Dickinson; bpo-1580(7))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1739468
(2) https://bugs.python.org/issue?@action=redirect&bpo=3439
(3) https://bugs.python.org/issue?@action=redirect&bpo=5237
(4) https://bugs.python.org/issue?@action=redirect&bpo=5675
(5) https://codereview.appspot.com/53094
(6) https://bugs.python.org/issue?@action=redirect&bpo=4707
(7) https://bugs.python.org/issue?@action=redirect&bpo=1580
File: python.info, Node: New Improved and Deprecated Modules<2>, Next: Optimizations<12>, Prev: Other Language Changes<13>, Up: What’s New In Python 3 1
1.13.4 New, Improved, and Deprecated Modules
--------------------------------------------
* Added a *note collections.Counter: 108b. class to support
convenient counting of unique items in a sequence or iterable:
>>> Counter(['red', 'blue', 'red', 'green', 'blue', 'blue'])
Counter({'blue': 3, 'red': 2, 'green': 1})
(Contributed by Raymond Hettinger; bpo-1696199(1).)
* Added a new module, *note tkinter.ttk: f9. for access to the Tk
themed widget set. The basic idea of ttk is to separate, to the
extent possible, the code implementing a widget’s behavior from the
code implementing its appearance.
(Contributed by Guilherme Polo; bpo-2983(2).)
* The *note gzip.GzipFile: 416. and *note bz2.BZ2File: 863. classes
now support the context management protocol:
>>> # Automatically close file after writing
>>> with gzip.GzipFile(filename, "wb") as f:
... f.write(b"xxx")
(Contributed by Antoine Pitrou.)
* The *note decimal: 36. module now supports methods for creating a
decimal object from a binary *note float: 2f1. The conversion is
exact but can sometimes be surprising:
>>> Decimal.from_float(1.1)
Decimal('1.100000000000000088817841970012523233890533447265625')
The long decimal result shows the actual binary fraction being
stored for '1.1'. The fraction has many digits because '1.1'
cannot be exactly represented in binary.
(Contributed by Raymond Hettinger and Mark Dickinson.)
* The *note itertools: 81. module grew two new functions. The *note
itertools.combinations_with_replacement(): 1274. function is one of
four for generating combinatorics including permutations and
Cartesian products. The *note itertools.compress(): 1275. function
mimics its namesake from APL. Also, the existing *note
itertools.count(): 1276. function now has an optional 'step'
argument and can accept any type of counting sequence including
*note fractions.Fraction: 1e9. and *note decimal.Decimal: 29f.:
>>> [p+q for p,q in combinations_with_replacement('LOVE', 2)]
['LL', 'LO', 'LV', 'LE', 'OO', 'OV', 'OE', 'VV', 'VE', 'EE']
>>> list(compress(data=range(10), selectors=[0,0,1,1,0,1,0,1,0,0]))
[2, 3, 5, 7]
>>> c = count(start=Fraction(1,2), step=Fraction(1,6))
>>> [next(c), next(c), next(c), next(c)]
[Fraction(1, 2), Fraction(2, 3), Fraction(5, 6), Fraction(1, 1)]
(Contributed by Raymond Hettinger.)
* *note collections.namedtuple(): 1ca. now supports a keyword
argument 'rename' which lets invalid fieldnames be automatically
converted to positional names in the form _0, _1, etc. This is
useful when the field names are being created by an external source
such as a CSV header, SQL field list, or user input:
>>> query = input()
SELECT region, dept, count(*) FROM main GROUPBY region, dept
>>> cursor.execute(query)
>>> query_fields = [desc[0] for desc in cursor.description]
>>> UserQuery = namedtuple('UserQuery', query_fields, rename=True)
>>> pprint.pprint([UserQuery(*row) for row in cursor])
[UserQuery(region='South', dept='Shipping', _2=185),
UserQuery(region='North', dept='Accounting', _2=37),
UserQuery(region='West', dept='Sales', _2=419)]
(Contributed by Raymond Hettinger; bpo-1818(3).)
* The *note re.sub(): 2a5, *note re.subn(): 2a6. and *note
re.split(): 2a4. functions now accept a flags parameter.
(Contributed by Gregory Smith.)
* The *note logging: 87. module now implements a simple *note
logging.NullHandler: 1277. class for applications that are not
using logging but are calling library code that does. Setting-up a
null handler will suppress spurious warnings such as “No handlers
could be found for logger foo”:
>>> h = logging.NullHandler()
>>> logging.getLogger("foo").addHandler(h)
(Contributed by Vinay Sajip; bpo-4384(4)).
* The *note runpy: be. module which supports the ‘-m’ command line
switch now supports the execution of packages by looking for and
executing a ‘__main__’ submodule when a package name is supplied.
(Contributed by Andi Vajda; bpo-4195(5).)
* The *note pdb: a5. module can now access and display source code
loaded via *note zipimport: 132. (or any other conformant PEP
302(6) loader).
(Contributed by Alexander Belopolsky; bpo-4201(7).)
* *note functools.partial: 410. objects can now be pickled.
(Suggested by Antoine Pitrou and Jesse Noller. Implemented by Jack
Diederich; bpo-5228(8).)
* Add *note pydoc: b5. help topics for symbols so that ‘help('@')’
works as expected in the interactive environment.
(Contributed by David Laban; bpo-4739(9).)
* The *note unittest: 106. module now supports skipping individual
tests or classes of tests. And it supports marking a test as an
expected failure, a test that is known to be broken, but shouldn’t
be counted as a failure on a TestResult:
class TestGizmo(unittest.TestCase):
@unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")
def test_gizmo_on_windows(self):
...
@unittest.expectedFailure
def test_gimzo_without_required_library(self):
...
Also, tests for exceptions have been builtout to work with context
managers using the *note with: 5ce. statement:
def test_division_by_zero(self):
with self.assertRaises(ZeroDivisionError):
x / 0
In addition, several new assertion methods were added including
*note assertSetEqual(): 1278, *note assertDictEqual(): 1279,
‘assertDictContainsSubset()’, *note assertListEqual(): 127a, *note
assertTupleEqual(): 127b, *note assertSequenceEqual(): 127c, *note
assertRaisesRegexp(): 52a, *note assertIsNone(): 127d, and *note
assertIsNotNone(): 127e.
(Contributed by Benjamin Peterson and Antoine Pitrou.)
* The *note io: 7f. module has three new constants for the *note
seek(): 127f. method: *note SEEK_SET: 1280, *note SEEK_CUR: 1281,
and *note SEEK_END: 1282.
* The *note sys.version_info: 69c. tuple is now a named tuple:
>>> sys.version_info
sys.version_info(major=3, minor=1, micro=0, releaselevel='alpha', serial=2)
(Contributed by Ross Light; bpo-4285(10).)
* The ‘nntplib’ and *note imaplib: 74. modules now support IPv6.
(Contributed by Derek Morr; bpo-1655(11) and bpo-1664(12).)
* The *note pickle: a6. module has been adapted for better
interoperability with Python 2.x when used with protocol 2 or
lower. The reorganization of the standard library changed the
formal reference for many objects. For example, ‘__builtin__.set’
in Python 2 is called ‘builtins.set’ in Python 3. This change
confounded efforts to share data between different versions of
Python. But now when protocol 2 or lower is selected, the pickler
will automatically use the old Python 2 names for both loading and
dumping. This remapping is turned-on by default but can be
disabled with the 'fix_imports' option:
>>> s = {1, 2, 3}
>>> pickle.dumps(s, protocol=0)
b'c__builtin__\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
>>> pickle.dumps(s, protocol=0, fix_imports=False)
b'cbuiltins\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
An unfortunate but unavoidable side-effect of this change is that
protocol 2 pickles produced by Python 3.1 won’t be readable with
Python 3.0. The latest pickle protocol, protocol 3, should be used
when migrating data between Python 3.x implementations, as it
doesn’t attempt to remain compatible with Python 2.x.
(Contributed by Alexandre Vassalotti and Antoine Pitrou,
bpo-6137(13).)
* A new module, *note importlib: 77. was added. It provides a
complete, portable, pure Python reference implementation of the
*note import: 5de. statement and its counterpart, the *note
__import__(): 8d3. function. It represents a substantial step
forward in documenting and defining the actions that take place
during imports.
(Contributed by Brett Cannon.)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1696199
(2) https://bugs.python.org/issue?@action=redirect&bpo=2983
(3) https://bugs.python.org/issue?@action=redirect&bpo=1818
(4) https://bugs.python.org/issue?@action=redirect&bpo=4384
(5) https://bugs.python.org/issue?@action=redirect&bpo=4195
(6) https://peps.python.org/pep-0302/
(7) https://bugs.python.org/issue?@action=redirect&bpo=4201
(8) https://bugs.python.org/issue?@action=redirect&bpo=5228
(9) https://bugs.python.org/issue?@action=redirect&bpo=4739
(10) https://bugs.python.org/issue?@action=redirect&bpo=4285
(11) https://bugs.python.org/issue?@action=redirect&bpo=1655
(12) https://bugs.python.org/issue?@action=redirect&bpo=1664
(13) https://bugs.python.org/issue?@action=redirect&bpo=6137
File: python.info, Node: Optimizations<12>, Next: IDLE<2>, Prev: New Improved and Deprecated Modules<2>, Up: What’s New In Python 3 1
1.13.5 Optimizations
--------------------
Major performance enhancements have been added:
* The new I/O library (as defined in PEP 3116(1)) was mostly written
in Python and quickly proved to be a problematic bottleneck in
Python 3.0. In Python 3.1, the I/O library has been entirely
rewritten in C and is 2 to 20 times faster depending on the task at
hand. The pure Python version is still available for
experimentation purposes through the ‘_pyio’ module.
(Contributed by Amaury Forgeot d’Arc and Antoine Pitrou.)
* Added a heuristic so that tuples and dicts containing only
untrackable objects are not tracked by the garbage collector. This
can reduce the size of collections and therefore the garbage
collection overhead on long-running programs, depending on their
particular use of datatypes.
(Contributed by Antoine Pitrou, bpo-4688(2).)
* Enabling a configure option named ‘--with-computed-gotos’ on
compilers that support it (notably: gcc, SunPro, icc), the bytecode
evaluation loop is compiled with a new dispatch mechanism which
gives speedups of up to 20%, depending on the system, the compiler,
and the benchmark.
(Contributed by Antoine Pitrou along with a number of other
participants, bpo-4753(3)).
* The decoding of UTF-8, UTF-16 and LATIN-1 is now two to four times
faster.
(Contributed by Antoine Pitrou and Amaury Forgeot d’Arc,
bpo-4868(4).)
* The *note json: 82. module now has a C extension to substantially
improve its performance. In addition, the API was modified so that
json works only with *note str: 447, not with *note bytes: 1c2.
That change makes the module closely match the JSON
specification(5) which is defined in terms of Unicode.
(Contributed by Bob Ippolito and converted to Py3.1 by Antoine
Pitrou and Benjamin Peterson; bpo-4136(6).)
* Unpickling now interns the attribute names of pickled objects.
This saves memory and allows pickles to be smaller.
(Contributed by Jake McGuire and Antoine Pitrou; bpo-5084(7).)
---------- Footnotes ----------
(1) https://peps.python.org/pep-3116/
(2) https://bugs.python.org/issue?@action=redirect&bpo=4688
(3) https://bugs.python.org/issue?@action=redirect&bpo=4753
(4) https://bugs.python.org/issue?@action=redirect&bpo=4868
(5) https://json.org/
(6) https://bugs.python.org/issue?@action=redirect&bpo=4136
(7) https://bugs.python.org/issue?@action=redirect&bpo=5084
File: python.info, Node: IDLE<2>, Next: Build and C API Changes<6>, Prev: Optimizations<12>, Up: What’s New In Python 3 1
1.13.6 IDLE
-----------
* IDLE’s format menu now provides an option to strip trailing
whitespace from a source file.
(Contributed by Roger D. Serwy; bpo-5150(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5150
File: python.info, Node: Build and C API Changes<6>, Next: Porting to Python 3 1, Prev: IDLE<2>, Up: What’s New In Python 3 1
1.13.7 Build and C API Changes
------------------------------
Changes to Python’s build process and to the C API include:
* Integers are now stored internally either in base ‘2**15’ or in
base ‘2**30’, the base being determined at build time. Previously,
they were always stored in base ‘2**15’. Using base ‘2**30’ gives
significant performance improvements on 64-bit machines, but
benchmark results on 32-bit machines have been mixed. Therefore,
the default is to use base ‘2**30’ on 64-bit machines and base
‘2**15’ on 32-bit machines; on Unix, there’s a new configure option
‘--enable-big-digits’ that can be used to override this default.
Apart from the performance improvements this change should be
invisible to end users, with one exception: for testing and
debugging purposes there’s a new *note sys.int_info: 1286. that
provides information about the internal format, giving the number
of bits per digit and the size in bytes of the C type used to store
each digit:
>>> import sys
>>> sys.int_info
sys.int_info(bits_per_digit=30, sizeof_digit=4)
(Contributed by Mark Dickinson; bpo-4258(1).)
* The *note PyLong_AsUnsignedLongLong(): 1287. function now handles a
negative 'pylong' by raising *note OverflowError: 87f. instead of
*note TypeError: 534.
(Contributed by Mark Dickinson and Lisandro Dalcrin; bpo-5175(2).)
* Deprecated ‘PyNumber_Int()’. Use *note PyNumber_Long(): a67.
instead.
(Contributed by Mark Dickinson; bpo-4910(3).)
* Added a new *note PyOS_string_to_double(): 1288. function to
replace the deprecated functions ‘PyOS_ascii_strtod()’ and
‘PyOS_ascii_atof()’.
(Contributed by Mark Dickinson; bpo-5914(4).)
* Added *note PyCapsule: 1266. as a replacement for the ‘PyCObject’
API. The principal difference is that the new type has a well
defined interface for passing typing safety information and a less
complicated signature for calling a destructor. The old type had a
problematic API and is now deprecated.
(Contributed by Larry Hastings; bpo-5630(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4258
(2) https://bugs.python.org/issue?@action=redirect&bpo=5175
(3) https://bugs.python.org/issue?@action=redirect&bpo=4910
(4) https://bugs.python.org/issue?@action=redirect&bpo=5914
(5) https://bugs.python.org/issue?@action=redirect&bpo=5630
File: python.info, Node: Porting to Python 3 1, Prev: Build and C API Changes<6>, Up: What’s New In Python 3 1
1.13.8 Porting to Python 3.1
----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code:
* The new floating-point string representations can break existing
doctests. For example:
def e():
'''Compute the base of natural logarithms.
>>> e()
2.7182818284590451
'''
return sum(1/math.factorial(x) for x in reversed(range(30)))
doctest.testmod()
**********************************************************************
Failed example:
e()
Expected:
2.7182818284590451
Got:
2.718281828459045
**********************************************************************
* The automatic name remapping in the pickle module for protocol 2 or
lower can make Python 3.1 pickles unreadable in Python 3.0. One
solution is to use protocol 3. Another solution is to set the
'fix_imports' option to ‘False’. See the discussion above for more
details.
File: python.info, Node: What’s New In Python 3 0, Next: What’s New in Python 2 7, Prev: What’s New In Python 3 1, Up: What’s New in Python
1.14 What’s New In Python 3.0
=============================
Author: Guido van Rossum
This article explains the new features in Python 3.0, compared to 2.6.
Python 3.0, also known as “Python 3000” or “Py3K”, is the first ever
'intentionally backwards incompatible' Python release. Python 3.0 was
released on December 3, 2008. There are more changes than in a typical
release, and more that are important for all Python users.
Nevertheless, after digesting the changes, you’ll find that Python
really hasn’t changed all that much – by and large, we’re mostly fixing
well-known annoyances and warts, and removing a lot of old cruft.
This article doesn’t attempt to provide a complete specification of all
new features, but instead tries to give a convenient overview. For full
details, you should refer to the documentation for Python 3.0, and/or
the many PEPs referenced in the text. If you want to understand the
complete implementation and design rationale for a particular feature,
PEPs usually have more details than the regular documentation; but note
that PEPs usually are not kept up-to-date once a feature has been fully
implemented.
Due to time constraints this document is not as complete as it should
have been. As always for a new release, the ‘Misc/NEWS’ file in the
source distribution contains a wealth of detailed information about
every small thing that was changed.
* Menu:
* Common Stumbling Blocks::
* Overview Of Syntax Changes::
* Changes Already Present In Python 2.6: Changes Already Present In Python 2 6.
* Library Changes::
* PEP 3101; A New Approach To String Formatting: PEP 3101 A New Approach To String Formatting.
* Changes To Exceptions::
* Miscellaneous Other Changes::
* Build and C API Changes: Build and C API Changes<7>.
* Performance::
* Porting To Python 3.0: Porting To Python 3 0.
File: python.info, Node: Common Stumbling Blocks, Next: Overview Of Syntax Changes, Up: What’s New In Python 3 0
1.14.1 Common Stumbling Blocks
------------------------------
This section lists those few changes that are most likely to trip you up
if you’re used to Python 2.5.
* Menu:
* Print Is A Function::
* Views And Iterators Instead Of Lists::
* Ordering Comparisons::
* Integers::
* Text Vs. Data Instead Of Unicode Vs. 8-bit: Text Vs Data Instead Of Unicode Vs 8-bit.
File: python.info, Node: Print Is A Function, Next: Views And Iterators Instead Of Lists, Up: Common Stumbling Blocks
1.14.1.1 Print Is A Function
............................
The ‘print’ statement has been replaced with a *note print(): f70.
function, with keyword arguments to replace most of the special syntax
of the old ‘print’ statement ( PEP 3105(1)). Examples:
Old: print "The answer is", 2*2
New: print("The answer is", 2*2)
Old: print x, # Trailing comma suppresses newline
New: print(x, end=" ") # Appends a space instead of a newline
Old: print # Prints a newline
New: print() # You must call the function!
Old: print >>sys.stderr, "fatal error"
New: print("fatal error", file=sys.stderr)
Old: print (x, y) # prints repr((x, y))
New: print((x, y)) # Not the same as print(x, y)!
You can also customize the separator between items, e.g.:
print("There are <", 2**32, "> possibilities!", sep="")
which produces:
There are <4294967296> possibilities!
Note:
* The *note print(): f70. function doesn’t support the “softspace”
feature of the old ‘print’ statement. For example, in Python 2.x,
‘print "A\n", "B"’ would write ‘"A\nB\n"’; but in Python 3.0,
‘print("A\n", "B")’ writes ‘"A\n B\n"’.
* Initially, you’ll be finding yourself typing the old ‘print x’ a
lot in interactive mode. Time to retrain your fingers to type
‘print(x)’ instead!
* When using the ‘2to3’ source-to-source conversion tool, all ‘print’
statements are automatically converted to *note print(): f70.
function calls, so this is mostly a non-issue for larger projects.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3105/
File: python.info, Node: Views And Iterators Instead Of Lists, Next: Ordering Comparisons, Prev: Print Is A Function, Up: Common Stumbling Blocks
1.14.1.2 Views And Iterators Instead Of Lists
.............................................
Some well-known APIs no longer return lists:
* *note dict: 258. methods *note dict.keys(): 7c8, *note
dict.items(): 7ca. and *note dict.values(): 7c9. return “views”
instead of lists. For example, this no longer works: ‘k =
d.keys(); k.sort()’. Use ‘k = sorted(d)’ instead (this works in
Python 2.5 too and is just as efficient).
* Also, the ‘dict.iterkeys()’, ‘dict.iteritems()’ and
‘dict.itervalues()’ methods are no longer supported.
* *note map(): 860. and *note filter(): 861. return iterators. If
you really need a list and the input sequences are all of equal
length, a quick fix is to wrap *note map(): 860. in *note list():
60d, e.g. ‘list(map(...))’, but a better fix is often to use a
list comprehension (especially when the original code uses *note
lambda: 128f.), or rewriting the code so it doesn’t need a list at
all. Particularly tricky is *note map(): 860. invoked for the side
effects of the function; the correct transformation is to use a
regular *note for: 2ec. loop (since creating a list would just be
wasteful).
If the input sequences are not of equal length, *note map(): 860.
will stop at the termination of the shortest of the sequences. For
full compatibility with *note map(): 860. from Python 2.x, also
wrap the sequences in *note itertools.zip_longest(): 1290, e.g.
‘map(func, *sequences)’ becomes ‘list(map(func,
itertools.zip_longest(*sequences)))’.
* *note range(): 943. now behaves like ‘xrange()’ used to behave,
except it works with values of arbitrary size. The latter no
longer exists.
* *note zip(): 7cb. now returns an iterator.
File: python.info, Node: Ordering Comparisons, Next: Integers, Prev: Views And Iterators Instead Of Lists, Up: Common Stumbling Blocks
1.14.1.3 Ordering Comparisons
.............................
Python 3.0 has simplified the rules for ordering comparisons:
* The ordering comparison operators (‘<’, ‘<=’, ‘>=’, ‘>’) raise a
TypeError exception when the operands don’t have a meaningful
natural ordering. Thus, expressions like ‘1 < ''’, ‘0 > None’ or
‘len <= len’ are no longer valid, and e.g. ‘None < None’ raises
*note TypeError: 534. instead of returning ‘False’. A corollary is
that sorting a heterogeneous list no longer makes sense – all the
elements must be comparable to each other. Note that this does not
apply to the ‘==’ and ‘!=’ operators: objects of different
incomparable types always compare unequal to each other.
* *note sorted(): bce. and *note list.sort(): bcf. no longer accept
the 'cmp' argument providing a comparison function. Use the 'key'
argument instead. N.B. the 'key' and 'reverse' arguments are now
“keyword-only”.
* The ‘cmp()’ function should be treated as gone, and the ‘__cmp__()’
special method is no longer supported. Use *note __lt__(): 1292.
for sorting, *note __eq__(): afa. with *note __hash__(): afb, and
other rich comparisons as needed. (If you really need the ‘cmp()’
functionality, you could use the expression ‘(a > b) - (a < b)’ as
the equivalent for ‘cmp(a, b)’.)
File: python.info, Node: Integers, Next: Text Vs Data Instead Of Unicode Vs 8-bit, Prev: Ordering Comparisons, Up: Common Stumbling Blocks
1.14.1.4 Integers
.................
* PEP 237(1): Essentially, ‘long’ renamed to *note int: 259. That
is, there is only one built-in integral type, named *note int:
259.; but it behaves mostly like the old ‘long’ type.
* PEP 238(2): An expression like ‘1/2’ returns a float. Use ‘1//2’
to get the truncating behavior. (The latter syntax has existed for
years, at least since Python 2.2.)
* The ‘sys.maxint’ constant was removed, since there is no longer a
limit to the value of integers. However, *note sys.maxsize: 11b7.
can be used as an integer larger than any practical list or string
index. It conforms to the implementation’s “natural” integer size
and is typically the same as ‘sys.maxint’ in previous releases on
the same platform (assuming the same build options).
* The *note repr(): 7f9. of a long integer doesn’t include the
trailing ‘L’ anymore, so code that unconditionally strips that
character will chop off the last digit instead. (Use *note str():
447. instead.)
* Octal literals are no longer of the form ‘0720’; use ‘0o720’
instead.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0237/
(2) https://peps.python.org/pep-0238/
File: python.info, Node: Text Vs Data Instead Of Unicode Vs 8-bit, Prev: Integers, Up: Common Stumbling Blocks
1.14.1.5 Text Vs. Data Instead Of Unicode Vs. 8-bit
...................................................
Everything you thought you knew about binary data and Unicode has
changed.
* Python 3.0 uses the concepts of 'text' and (binary) 'data' instead
of Unicode strings and 8-bit strings. All text is Unicode; however
'encoded' Unicode is represented as binary data. The type used to
hold text is *note str: 447, the type used to hold data is *note
bytes: 1c2. The biggest difference with the 2.x situation is that
any attempt to mix text and data in Python 3.0 raises *note
TypeError: 534, whereas if you were to mix Unicode and 8-bit
strings in Python 2.x, it would work if the 8-bit string happened
to contain only 7-bit (ASCII) bytes, but you would get *note
UnicodeDecodeError: a12. if it contained non-ASCII values. This
value-specific behavior has caused numerous sad faces over the
years.
* As a consequence of this change in philosophy, pretty much all code
that uses Unicode, encodings or binary data most likely has to
change. The change is for the better, as in the 2.x world there
were numerous bugs having to do with mixing encoded and unencoded
text. To be prepared in Python 2.x, start using ‘unicode’ for all
unencoded text, and *note str: 447. for binary or encoded data
only. Then the ‘2to3’ tool will do most of the work for you.
* You can no longer use ‘u"..."’ literals for Unicode text. However,
you must use ‘b"..."’ literals for binary data.
* As the *note str: 447. and *note bytes: 1c2. types cannot be mixed,
you must always explicitly convert between them. Use *note
str.encode(): 8d4. to go from *note str: 447. to *note bytes: 1c2,
and *note bytes.decode(): 8d5. to go from *note bytes: 1c2. to
*note str: 447. You can also use ‘bytes(s, encoding=...)’ and
‘str(b, encoding=...)’, respectively.
* Like *note str: 447, the *note bytes: 1c2. type is immutable.
There is a separate 'mutable' type to hold buffered binary data,
*note bytearray: 53a. Nearly all APIs that accept *note bytes:
1c2. also accept *note bytearray: 53a. The mutable API is based on
*note collections.MutableSequence: 1a1.
* All backslashes in raw string literals are interpreted literally.
This means that ‘'\U'’ and ‘'\u'’ escapes in raw strings are not
treated specially. For example, ‘r'\u20ac'’ is a string of 6
characters in Python 3.0, whereas in 2.6, ‘ur'\u20ac'’ was the
single “euro” character. (Of course, this change only affects raw
string literals; the euro character is ‘'\u20ac'’ in Python 3.0.)
* The built-in ‘basestring’ abstract type was removed. Use *note
str: 447. instead. The *note str: 447. and *note bytes: 1c2. types
don’t have functionality enough in common to warrant a shared base
class. The ‘2to3’ tool (see below) replaces every occurrence of
‘basestring’ with *note str: 447.
* Files opened as text files (still the default mode for *note
open(): 517.) always use an encoding to map between strings (in
memory) and bytes (on disk). Binary files (opened with a ‘b’ in
the mode argument) always use bytes in memory. This means that if
a file is opened using an incorrect mode or encoding, I/O will
likely fail loudly, instead of silently producing incorrect data.
It also means that even Unix users will have to specify the correct
mode (text or binary) when opening a file. There is a
platform-dependent default encoding, which on Unixy platforms can
be set with the ‘LANG’ environment variable (and sometimes also
with some other platform-specific locale-related environment
variables). In many cases, but not all, the system default is
UTF-8; you should never count on this default. Any application
reading or writing more than pure ASCII text should probably have a
way to override the encoding. There is no longer any need for
using the encoding-aware streams in the *note codecs: 1b. module.
* The initial values of *note sys.stdin: 539, *note sys.stdout: ad6.
and *note sys.stderr: 939. are now unicode-only text files (i.e.,
they are instances of *note io.TextIOBase: 691.). To read and
write bytes data with these streams, you need to use their *note
io.TextIOBase.buffer: 1295. attribute.
* Filenames are passed to and returned from APIs as (Unicode)
strings. This can present platform-specific problems because on
some platforms filenames are arbitrary byte strings. (On the other
hand, on Windows filenames are natively stored as Unicode.) As a
work-around, most APIs (e.g. *note open(): 517. and many functions
in the *note os: a1. module) that take filenames accept *note
bytes: 1c2. objects as well as strings, and a few APIs have a way
to ask for a *note bytes: 1c2. return value. Thus, *note
os.listdir(): 10ee. returns a list of *note bytes: 1c2. instances
if the argument is a *note bytes: 1c2. instance, and *note
os.getcwdb(): a91. returns the current working directory as a *note
bytes: 1c2. instance. Note that when *note os.listdir(): 10ee.
returns a list of strings, filenames that cannot be decoded
properly are omitted rather than raising *note UnicodeError: 1296.
* Some system APIs like *note os.environ: 11ac. and *note sys.argv:
1258. can also present problems when the bytes made available by
the system is not interpretable using the default encoding.
Setting the ‘LANG’ variable and rerunning the program is probably
the best approach.
* PEP 3138(1): The *note repr(): 7f9. of a string no longer escapes
non-ASCII characters. It still escapes control characters and code
points with non-printable status in the Unicode standard, however.
* PEP 3120(2): The default source encoding is now UTF-8.
* PEP 3131(3): Non-ASCII letters are now allowed in identifiers.
(However, the standard library remains ASCII-only with the
exception of contributor names in comments.)
* The ‘StringIO’ and ‘cStringIO’ modules are gone. Instead, import
the *note io: 7f. module and use *note io.StringIO: f22. or *note
io.BytesIO: e9d. for text and data respectively.
* See also the *note Unicode HOWTO: 1297, which was updated for
Python 3.0.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3138/
(2) https://peps.python.org/pep-3120/
(3) https://peps.python.org/pep-3131/
File: python.info, Node: Overview Of Syntax Changes, Next: Changes Already Present In Python 2 6, Prev: Common Stumbling Blocks, Up: What’s New In Python 3 0
1.14.2 Overview Of Syntax Changes
---------------------------------
This section gives a brief overview of every 'syntactic' change in
Python 3.0.
* Menu:
* New Syntax::
* Changed Syntax::
* Removed Syntax::
File: python.info, Node: New Syntax, Next: Changed Syntax, Up: Overview Of Syntax Changes
1.14.2.1 New Syntax
...................
* PEP 3107(1): Function argument and return value annotations. This
provides a standardized way of annotating a function’s parameters
and return value. There are no semantics attached to such
annotations except that they can be introspected at runtime using
the ‘__annotations__’ attribute. The intent is to encourage
experimentation through metaclasses, decorators or frameworks.
* PEP 3102(2): Keyword-only arguments. Named parameters occurring
after ‘*args’ in the parameter list 'must' be specified using
keyword syntax in the call. You can also use a bare ‘*’ in the
parameter list to indicate that you don’t accept a variable-length
argument list, but you do have keyword-only arguments.
* Keyword arguments are allowed after the list of base classes in a
class definition. This is used by the new convention for
specifying a metaclass (see next section), but can be used for
other purposes as well, as long as the metaclass supports it.
* PEP 3104(3): *note nonlocal: 129a. statement. Using ‘nonlocal x’
you can now assign directly to a variable in an outer (but
non-global) scope. ‘nonlocal’ is a new reserved word.
* PEP 3132(4): Extended Iterable Unpacking. You can now write things
like ‘a, b, *rest = some_sequence’. And even ‘*rest, a = stuff’.
The ‘rest’ object is always a (possibly empty) list; the right-hand
side may be any iterable. Example:
(a, *rest, b) = range(5)
This sets 'a' to ‘0’, 'b' to ‘4’, and 'rest' to ‘[1, 2, 3]’.
* Dictionary comprehensions: ‘{k: v for k, v in stuff}’ means the
same thing as ‘dict(stuff)’ but is more flexible. (This is PEP
274(5) vindicated. :-)
* Set literals, e.g. ‘{1, 2}’. Note that ‘{}’ is an empty
dictionary; use ‘set()’ for an empty set. Set comprehensions are
also supported; e.g., ‘{x for x in stuff}’ means the same thing as
‘set(stuff)’ but is more flexible.
* New octal literals, e.g. ‘0o720’ (already in 2.6). The old octal
literals (‘0720’) are gone.
* New binary literals, e.g. ‘0b1010’ (already in 2.6), and there is
a new corresponding built-in function, *note bin(): 129b.
* Bytes literals are introduced with a leading ‘b’ or ‘B’, and there
is a new corresponding built-in function, *note bytes(): 1c2.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3107/
(2) https://peps.python.org/pep-3102/
(3) https://peps.python.org/pep-3104/
(4) https://peps.python.org/pep-3132/
(5) https://peps.python.org/pep-0274/
File: python.info, Node: Changed Syntax, Next: Removed Syntax, Prev: New Syntax, Up: Overview Of Syntax Changes
1.14.2.2 Changed Syntax
.......................
* PEP 3109(1) and PEP 3134(2): new *note raise: 5e9. statement
syntax: ‘raise [EXPR [from EXPR]]’. See below.
* ‘as’ and *note with: 5ce. are now reserved words. (Since 2.6,
actually.)
* ‘True’, ‘False’, and ‘None’ are reserved words. (2.6 partially
enforced the restrictions on ‘None’ already.)
* Change from *note except: 18b. 'exc', 'var' to ‘except’ 'exc' ‘as’
'var'. See PEP 3110(3).
* PEP 3115(4): New Metaclass Syntax. Instead of:
class C:
__metaclass__ = M
...
you must now use:
class C(metaclass=M):
...
The module-global ‘__metaclass__’ variable is no longer supported.
(It was a crutch to make it easier to default to new-style classes
without deriving every class from *note object: a8c.)
* List comprehensions no longer support the syntactic form ‘[... for
VAR in ITEM1, ITEM2, ...]’. Use ‘[... for VAR in (ITEM1, ITEM2,
...)]’ instead. Also note that list comprehensions have different
semantics: they are closer to syntactic sugar for a generator
expression inside a *note list(): 60d. constructor, and in
particular the loop control variables are no longer leaked into the
surrounding scope.
* The 'ellipsis' (‘...’) can be used as an atomic expression
anywhere. (Previously it was only allowed in slices.) Also, it
'must' now be spelled as ‘...’. (Previously it could also be
spelled as ‘. . .’, by a mere accident of the grammar.)
---------- Footnotes ----------
(1) https://peps.python.org/pep-3109/
(2) https://peps.python.org/pep-3134/
(3) https://peps.python.org/pep-3110/
(4) https://peps.python.org/pep-3115/
File: python.info, Node: Removed Syntax, Prev: Changed Syntax, Up: Overview Of Syntax Changes
1.14.2.3 Removed Syntax
.......................
* PEP 3113(1): Tuple parameter unpacking removed. You can no longer
write ‘def foo(a, (b, c)): ...’. Use ‘def foo(a, b_c): b, c = b_c’
instead.
* Removed backticks (use *note repr(): 7f9. instead).
* Removed ‘<>’ (use ‘!=’ instead).
* Removed keyword: *note exec(): 17f. is no longer a keyword; it
remains as a function. (Fortunately the function syntax was also
accepted in 2.x.) Also note that *note exec(): 17f. no longer
takes a stream argument; instead of ‘exec(f)’ you can use
‘exec(f.read())’.
* Integer literals no longer support a trailing ‘l’ or ‘L’.
* String literals no longer support a leading ‘u’ or ‘U’.
* The *note from: 129e. 'module' *note import: 5de. ‘*’ syntax is
only allowed at the module level, no longer inside functions.
* The only acceptable syntax for relative imports is ‘from .[MODULE]
import NAME’. All *note import: 5de. forms not starting with ‘.’
are interpreted as absolute imports. ( PEP 328(2))
* Classic classes are gone.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3113/
(2) https://peps.python.org/pep-0328/
File: python.info, Node: Changes Already Present In Python 2 6, Next: Library Changes, Prev: Overview Of Syntax Changes, Up: What’s New In Python 3 0
1.14.3 Changes Already Present In Python 2.6
--------------------------------------------
Since many users presumably make the jump straight from Python 2.5 to
Python 3.0, this section reminds the reader of new features that were
originally designed for Python 3.0 but that were back-ported to Python
2.6. The corresponding sections in *note What’s New in Python 2.6:
12a0. should be consulted for longer descriptions.
* *note PEP 343; The ‘with’ statement: 12a1. The *note with: 5ce.
statement is now a standard feature and no longer needs to be
imported from the *note __future__: 0. Also check out *note
Writing Context Managers: 12a2. and *note The contextlib module:
12a3.
* *note PEP 366; Explicit Relative Imports From a Main Module: 12a4.
This enhances the usefulness of the *note -m: 5dd. option when the
referenced module lives in a package.
* *note PEP 370; Per-user site-packages Directory: 12a5.
* *note PEP 371; The multiprocessing Package: 12a6.
* *note PEP 3101; Advanced String Formatting: 12a7. Note: the 2.6
description mentions the *note format(): 61b. method for both 8-bit
and Unicode strings. In 3.0, only the *note str: 447. type (text
strings with Unicode support) supports this method; the *note
bytes: 1c2. type does not. The plan is to eventually make this the
only API for string formatting, and to start deprecating the ‘%’
operator in Python 3.1.
* *note PEP 3105; print As a Function: 12a8. This is now a standard
feature and no longer needs to be imported from *note __future__:
0. More details were given above.
* *note PEP 3110; Exception-Handling Changes: 12a9. The *note
except: 18b. 'exc' ‘as’ 'var' syntax is now standard and ‘except’
'exc', 'var' is no longer supported. (Of course, the ‘as’ 'var'
part is still optional.)
* *note PEP 3112; Byte Literals: 12aa. The ‘b"..."’ string literal
notation (and its variants like ‘b'...'’, ‘b"""..."""’, and
‘br"..."’) now produces a literal of type *note bytes: 1c2.
* *note PEP 3116; New I/O Library: 12ab. The *note io: 7f. module is
now the standard way of doing file I/O. The built-in *note open():
517. function is now an alias for *note io.open(): 518. and has
additional keyword arguments 'encoding', 'errors', 'newline' and
'closefd'. Also note that an invalid 'mode' argument now raises
*note ValueError: 204, not *note IOError: 104b. The binary file
object underlying a text file object can be accessed as ‘f.buffer’
(but beware that the text object maintains a buffer of itself in
order to speed up the encoding and decoding operations).
* *note PEP 3118; Revised Buffer Protocol: 12ac. The old builtin
‘buffer()’ is now really gone; the new builtin *note memoryview():
464. provides (mostly) similar functionality.
* *note PEP 3119; Abstract Base Classes: 12ad. The *note abc: 4.
module and the ABCs defined in the *note collections: 1d. module
plays a somewhat more prominent role in the language now, and
built-in collection types like *note dict: 258. and *note list:
60d. conform to the *note collections.MutableMapping: 10a2. and
*note collections.MutableSequence: 1a1. ABCs, respectively.
* *note PEP 3127; Integer Literal Support and Syntax: 12ae. As
mentioned above, the new octal literal notation is the only one
supported, and binary literals have been added.
* *note PEP 3129; Class Decorators: 12af.
* *note PEP 3141; A Type Hierarchy for Numbers: 12b0. The *note
numbers: 9e. module is another new use of ABCs, defining Python’s
“numeric tower”. Also note the new *note fractions: 5d. module
which implements *note numbers.Rational: 12b1.
File: python.info, Node: Library Changes, Next: PEP 3101 A New Approach To String Formatting, Prev: Changes Already Present In Python 2 6, Up: What’s New In Python 3 0
1.14.4 Library Changes
----------------------
Due to time constraints, this document does not exhaustively cover the
very extensive changes to the standard library. PEP 3108(1) is the
reference for the major changes to the library. Here’s a capsule
review:
* Many old modules were removed. Some, like ‘gopherlib’ (no longer
used) and ‘md5’ (replaced by *note hashlib: 68.), were already
deprecated by PEP 4(2). Others were removed as a result of the
removal of support for various platforms such as Irix, BeOS and Mac
OS 9 (see PEP 11(3)). Some modules were also selected for removal
in Python 3.0 due to lack of use or because a better replacement
exists. See PEP 3108(4) for an exhaustive list.
* The ‘bsddb3’ package was removed because its presence in the core
standard library has proved over time to be a particular burden for
the core developers due to testing instability and Berkeley DB’s
release schedule. However, the package is alive and well,
externally maintained at
‘https://www.jcea.es/programacion/pybsddb.htm’.
* Some modules were renamed because their old name disobeyed PEP
8(5), or for various other reasons. Here’s the list:
Old Name New Name
--------------------------------------------------------
_winreg winreg
ConfigParser configparser
copy_reg copyreg
Queue queue
SocketServer socketserver
markupbase _markupbase
repr reprlib
test.test_support test.support
* A common pattern in Python 2.x is to have one version of a module
implemented in pure Python, with an optional accelerated version
implemented as a C extension; for example, *note pickle: a6. and
‘cPickle’. This places the burden of importing the accelerated
version and falling back on the pure Python version on each user of
these modules. In Python 3.0, the accelerated versions are
considered implementation details of the pure Python versions.
Users should always import the standard version, which attempts to
import the accelerated version and falls back to the pure Python
version. The *note pickle: a6. / ‘cPickle’ pair received this
treatment. The *note profile: af. module is on the list for 3.1.
The ‘StringIO’ module has been turned into a class in the *note io:
7f. module.
* Some related modules have been grouped into packages, and usually
the submodule names have been simplified. The resulting new
packages are:
* *note dbm: 31. (‘anydbm’, ‘dbhash’, ‘dbm’, ‘dumbdbm’, ‘gdbm’,
‘whichdb’).
* *note html: 6b. (‘HTMLParser’, ‘htmlentitydefs’).
* *note http: 6e. (‘httplib’, ‘BaseHTTPServer’, ‘CGIHTTPServer’,
‘SimpleHTTPServer’, ‘Cookie’, ‘cookielib’).
* *note tkinter: f0. (all ‘Tkinter’-related modules except *note
turtle: 101.). The target audience of *note turtle: 101.
doesn’t really care about *note tkinter: f0. Also note that
as of Python 2.6, the functionality of *note turtle: 101. has
been greatly enhanced.
* *note urllib: 108. (‘urllib’, ‘urllib2’, ‘urlparse’,
‘robotparse’).
* *note xmlrpc: 12d. (‘xmlrpclib’, ‘DocXMLRPCServer’,
‘SimpleXMLRPCServer’).
Some other changes to standard library modules, not covered by PEP
3108(6):
* Killed ‘sets’. Use the built-in *note set(): 5d5. class.
* Cleanup of the *note sys: d9. module: removed ‘sys.exitfunc()’,
‘sys.exc_clear()’, ‘sys.exc_type’, ‘sys.exc_value’,
‘sys.exc_traceback’. (Note that *note sys.last_type: 4bb. etc.
remain.)
* Cleanup of the *note array.array: 1a0. type: the ‘read()’ and
‘write()’ methods are gone; use *note fromfile(): 12b3. and *note
tofile(): 12b4. instead. Also, the ‘'c'’ typecode for array is
gone – use either ‘'b'’ for bytes or ‘'u'’ for Unicode characters.
* Cleanup of the *note operator: 9f. module: removed
‘sequenceIncludes()’ and ‘isCallable()’.
* Cleanup of the ‘thread’ module: ‘acquire_lock()’ and
‘release_lock()’ are gone; use *note acquire(): 694. and *note
release(): 12b5. instead.
* Cleanup of the *note random: b8. module: removed the ‘jumpahead()’
API.
* The ‘new’ module is gone.
* The functions ‘os.tmpnam()’, ‘os.tempnam()’ and ‘os.tmpfile()’ have
been removed in favor of the *note tempfile: e0. module.
* The *note tokenize: fb. module has been changed to work with bytes.
The main entry point is now *note tokenize.tokenize(): 4d5, instead
of generate_tokens.
* ‘string.letters’ and its friends (‘string.lowercase’ and
‘string.uppercase’) are gone. Use *note string.ascii_letters:
12b6. etc. instead. (The reason for the removal is that
‘string.letters’ and friends had locale-specific behavior, which is
a bad idea for such attractively named global “constants”.)
* Renamed module ‘__builtin__’ to *note builtins: 12. (removing the
underscores, adding an ‘s’). The ‘__builtins__’ variable found in
most global namespaces is unchanged. To modify a builtin, you
should use *note builtins: 12, not ‘__builtins__’!
---------- Footnotes ----------
(1) https://peps.python.org/pep-3108/
(2) https://peps.python.org/pep-0004/
(3) https://peps.python.org/pep-0011/
(4) https://peps.python.org/pep-3108/
(5) https://peps.python.org/pep-0008/
(6) https://peps.python.org/pep-3108/
File: python.info, Node: PEP 3101 A New Approach To String Formatting, Next: Changes To Exceptions, Prev: Library Changes, Up: What’s New In Python 3 0
'PEP 3101': A New Approach To String Formatting
* A new system for built-in string formatting operations replaces the
‘%’ string formatting operator. (However, the ‘%’ operator is
still supported; it will be deprecated in Python 3.1 and removed
from the language at some later time.) Read PEP 3101(1) for the
full scoop.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3101/
File: python.info, Node: Changes To Exceptions, Next: Miscellaneous Other Changes, Prev: PEP 3101 A New Approach To String Formatting, Up: What’s New In Python 3 0
1.14.6 Changes To Exceptions
----------------------------
The APIs for raising and catching exception have been cleaned up and new
powerful features added:
* PEP 352(1): All exceptions must be derived (directly or indirectly)
from *note BaseException: 5b7. This is the root of the exception
hierarchy. This is not new as a recommendation, but the
'requirement' to inherit from *note BaseException: 5b7. is new.
(Python 2.6 still allowed classic classes to be raised, and placed
no restriction on what you can catch.) As a consequence, string
exceptions are finally truly and utterly dead.
* Almost all exceptions should actually derive from *note Exception:
9d9.; *note BaseException: 5b7. should only be used as a base class
for exceptions that should only be handled at the top level, such
as *note SystemExit: d40. or *note KeyboardInterrupt: 9d1. The
recommended idiom for handling all exceptions except for this
latter category is to use *note except: 18b. *note Exception: 9d9.
* ‘StandardError’ was removed.
* Exceptions no longer behave as sequences. Use the *note args: 569.
attribute instead.
* PEP 3109(2): Raising exceptions. You must now use ‘raise
EXCEPTION(ARGS)’ instead of ‘raise EXCEPTION, ARGS’. Additionally,
you can no longer explicitly specify a traceback; instead, if you
'have' to do this, you can assign directly to the *note
__traceback__: 12b9. attribute (see below).
* PEP 3110(3): Catching exceptions. You must now use ‘except
SOMEEXCEPTION as VARIABLE’ instead of ‘except SOMEEXCEPTION,
VARIABLE’. Moreover, the 'variable' is explicitly deleted when the
*note except: 18b. block is left.
* PEP 3134(4): Exception chaining. There are two cases: implicit
chaining and explicit chaining. Implicit chaining happens when an
exception is raised in an *note except: 18b. or *note finally: 9ca.
handler block. This usually happens due to a bug in the handler
block; we call this a 'secondary' exception. In this case, the
original exception (that was being handled) is saved as the *note
__context__: 12ba. attribute of the secondary exception. Explicit
chaining is invoked with this syntax:
raise SecondaryException() from primary_exception
(where 'primary_exception' is any expression that produces an
exception object, probably an exception that was previously
caught). In this case, the primary exception is stored on the
*note __cause__: 12bb. attribute of the secondary exception. The
traceback printed when an unhandled exception occurs walks the
chain of ‘__cause__’ and *note __context__: 12ba. attributes and
prints a separate traceback for each component of the chain, with
the primary exception at the top. (Java users may recognize this
behavior.)
* PEP 3134(5): Exception objects now store their traceback as the
*note __traceback__: 12b9. attribute. This means that an exception
object now contains all the information pertaining to an exception,
and there are fewer reasons to use *note sys.exc_info(): 686.
(though the latter is not removed).
* A few exception messages are improved when Windows fails to load an
extension module. For example, ‘error code 193’ is now ‘%1 is not
a valid Win32 application’. Strings now deal with non-English
locales.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0352/
(2) https://peps.python.org/pep-3109/
(3) https://peps.python.org/pep-3110/
(4) https://peps.python.org/pep-3134/
(5) https://peps.python.org/pep-3134/
File: python.info, Node: Miscellaneous Other Changes, Next: Build and C API Changes<7>, Prev: Changes To Exceptions, Up: What’s New In Python 3 0
1.14.7 Miscellaneous Other Changes
----------------------------------
* Menu:
* Operators And Special Methods::
* Builtins::
File: python.info, Node: Operators And Special Methods, Next: Builtins, Up: Miscellaneous Other Changes
1.14.7.1 Operators And Special Methods
......................................
* ‘!=’ now returns the opposite of ‘==’, unless ‘==’ returns *note
NotImplemented: 7cd.
* The concept of “unbound methods” has been removed from the
language. When referencing a method as a class attribute, you now
get a plain function object.
* ‘__getslice__()’, ‘__setslice__()’ and ‘__delslice__()’ were
killed. The syntax ‘a[i:j]’ now translates to
‘a.__getitem__(slice(i, j))’ (or *note __setitem__(): 12be. or
*note __delitem__(): 12bf, when used as an assignment or deletion
target, respectively).
* PEP 3114(1): the standard *note next(): 7d3. method has been
renamed to *note __next__(): 12c0.
* The ‘__oct__()’ and ‘__hex__()’ special methods are removed – *note
oct(): 12c1. and *note hex(): 12c2. use *note __index__(): 718. now
to convert the argument to an integer.
* Removed support for ‘__members__’ and ‘__methods__’.
* The function attributes named ‘func_X’ have been renamed to use the
‘__X__’ form, freeing up these names in the function attribute
namespace for user-defined attributes. To wit, ‘func_closure’,
‘func_code’, ‘func_defaults’, ‘func_dict’, ‘func_doc’,
‘func_globals’, ‘func_name’ were renamed to *note __closure__:
12c3, *note __code__: 29c, *note __defaults__: 12c4, *note
__dict__: 12c5, *note __doc__: 11cb, *note __globals__: 12c6, *note
__name__: 12c7, respectively.
* ‘__nonzero__()’ is now *note __bool__(): 12c8.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3114/
File: python.info, Node: Builtins, Prev: Operators And Special Methods, Up: Miscellaneous Other Changes
1.14.7.2 Builtins
.................
* PEP 3135(1): New *note super(): 4d7. You can now invoke *note
super(): 4d7. without arguments and (assuming this is in a regular
instance method defined inside a *note class: 12ca. statement) the
right class and instance will automatically be chosen. With
arguments, the behavior of *note super(): 4d7. is unchanged.
* PEP 3111(2): ‘raw_input()’ was renamed to *note input(): 12cb.
That is, the new *note input(): 12cb. function reads a line from
*note sys.stdin: 539. and returns it with the trailing newline
stripped. It raises *note EOFError: 12cc. if the input is
terminated prematurely. To get the old behavior of *note input():
12cb, use ‘eval(input())’.
* A new built-in function *note next(): 7d3. was added to call the
*note __next__(): 12c0. method on an object.
* The *note round(): 12cd. function rounding strategy and return type
have changed. Exact halfway cases are now rounded to the nearest
even result instead of away from zero. (For example, ‘round(2.5)’
now returns ‘2’ rather than ‘3’.) ‘round(x[, n])’ now delegates to
‘x.__round__([n])’ instead of always returning a float. It
generally returns an integer when called with a single argument and
a value of the same type as ‘x’ when called with two arguments.
* Moved ‘intern()’ to *note sys.intern(): 12ce.
* Removed: ‘apply()’. Instead of ‘apply(f, args)’ use ‘f(*args)’.
* Removed *note callable(): 11b8. Instead of ‘callable(f)’ you can
use ‘isinstance(f, collections.Callable)’. The
‘operator.isCallable()’ function is also gone.
* Removed ‘coerce()’. This function no longer serves a purpose now
that classic classes are gone.
* Removed ‘execfile()’. Instead of ‘execfile(fn)’ use
‘exec(open(fn).read())’.
* Removed the ‘file’ type. Use *note open(): 517. There are now
several different kinds of streams that open can return in the
*note io: 7f. module.
* Removed ‘reduce()’. Use *note functools.reduce(): 12cf. if you
really need it; however, 99 percent of the time an explicit *note
for: 2ec. loop is more readable.
* Removed ‘reload()’. Use ‘imp.reload()’.
* Removed. ‘dict.has_key()’ – use the *note in: 2ee. operator
instead.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3135/
(2) https://peps.python.org/pep-3111/
File: python.info, Node: Build and C API Changes<7>, Next: Performance, Prev: Miscellaneous Other Changes, Up: What’s New In Python 3 0
1.14.8 Build and C API Changes
------------------------------
Due to time constraints, here is a 'very' incomplete list of changes to
the C API.
* Support for several platforms was dropped, including but not
limited to Mac OS 9, BeOS, RISCOS, Irix, and Tru64.
* PEP 3118(1): New Buffer API.
* PEP 3121(2): Extension Module Initialization & Finalization.
* PEP 3123(3): Making *note PyObject_HEAD: 12d1. conform to standard
C.
* No more C API support for restricted execution.
* ‘PyNumber_Coerce()’, ‘PyNumber_CoerceEx()’, ‘PyMember_Get()’, and
‘PyMember_Set()’ C APIs are removed.
* New C API *note PyImport_ImportModuleNoBlock(): 3b9, works like
*note PyImport_ImportModule(): 3ba. but won’t block on the import
lock (returning an error instead).
* Renamed the boolean conversion C-level slot and method:
‘nb_nonzero’ is now ‘nb_bool’.
* Removed ‘METH_OLDARGS’ and ‘WITH_CYCLE_GC’ from the C API.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3118/
(2) https://peps.python.org/pep-3121/
(3) https://peps.python.org/pep-3123/
File: python.info, Node: Performance, Next: Porting To Python 3 0, Prev: Build and C API Changes<7>, Up: What’s New In Python 3 0
1.14.9 Performance
------------------
The net result of the 3.0 generalizations is that Python 3.0 runs the
pystone benchmark around 10% slower than Python 2.5. Most likely the
biggest cause is the removal of special-casing for small integers.
There’s room for improvement, but it will happen after 3.0 is released!
File: python.info, Node: Porting To Python 3 0, Prev: Performance, Up: What’s New In Python 3 0
1.14.10 Porting To Python 3.0
-----------------------------
For porting existing Python 2.5 or 2.6 source code to Python 3.0, the
best strategy is the following:
0. (Prerequisite:) Start with excellent test coverage.
1. Port to Python 2.6. This should be no more work than the average
port from Python 2.x to Python 2.(x+1). Make sure all your tests
pass.
2. (Still using 2.6:) Turn on the ‘-3’ command line switch. This
enables warnings about features that will be removed (or change) in
3.0. Run your test suite again, and fix code that you get warnings
about until there are no warnings left, and all your tests still
pass.
3. Run the ‘2to3’ source-to-source translator over your source code
tree. Run the result of the translation under Python 3.0.
Manually fix up any remaining issues, fixing problems until all
tests pass again.
It is not recommended to try to write source code that runs unchanged
under both Python 2.6 and 3.0; you’d have to use a very contorted coding
style, e.g. avoiding ‘print’ statements, metaclasses, and much more.
If you are maintaining a library that needs to support both Python 2.6
and Python 3.0, the best approach is to modify step 3 above by editing
the 2.6 version of the source code and running the ‘2to3’ translator
again, rather than editing the 3.0 version of the source code.
For porting C extensions to Python 3.0, please see *note Porting
Extension Modules to Python 3: 12d4.
File: python.info, Node: What’s New in Python 2 7, Next: What’s New in Python 2 6, Prev: What’s New In Python 3 0, Up: What’s New in Python
1.15 What’s New in Python 2.7
=============================
Author: A.M. Kuchling (amk at amk.ca)
This article explains the new features in Python 2.7. Python 2.7 was
released on July 3, 2010.
Numeric handling has been improved in many ways, for both floating-point
numbers and for the *note Decimal: 29f. class. There are some useful
additions to the standard library, such as a greatly enhanced *note
unittest: 106. module, the *note argparse: 6. module for parsing
command-line options, convenient *note OrderedDict: 5d7. and *note
Counter: 108b. classes in the *note collections: 1d. module, and many
other improvements.
Python 2.7 is planned to be the last of the 2.x releases, so we worked
on making it a good release for the long term. To help with porting to
Python 3, several new features from the Python 3.x series have been
included in 2.7.
This article doesn’t attempt to provide a complete specification of the
new features, but instead provides a convenient overview. For full
details, you should refer to the documentation for Python 2.7 at
‘https://docs.python.org’. If you want to understand the rationale for
the design and implementation, refer to the PEP for a particular new
feature or the issue on ‘https://bugs.python.org’ in which a change was
discussed. Whenever possible, “What’s New in Python” links to the
bug/patch item for each change.
* Menu:
* The Future for Python 2.x: The Future for Python 2 x.
* Changes to the Handling of Deprecation Warnings::
* Python 3.1 Features: Python 3 1 Features.
* PEP 372; Adding an Ordered Dictionary to collections: PEP 372 Adding an Ordered Dictionary to collections.
* PEP 378; Format Specifier for Thousands Separator: PEP 378 Format Specifier for Thousands Separator<2>.
* PEP 389; The argparse Module for Parsing Command Lines: PEP 389 The argparse Module for Parsing Command Lines.
* PEP 391; Dictionary-Based Configuration For Logging: PEP 391 Dictionary-Based Configuration For Logging.
* PEP 3106; Dictionary Views: PEP 3106 Dictionary Views.
* PEP 3137; The memoryview Object: PEP 3137 The memoryview Object.
* Other Language Changes: Other Language Changes<14>.
* New and Improved Modules::
* Build and C API Changes: Build and C API Changes<8>.
* Other Changes and Fixes::
* Porting to Python 2.7: Porting to Python 2 7.
* New Features Added to Python 2.7 Maintenance Releases: New Features Added to Python 2 7 Maintenance Releases.
* Acknowledgements::
File: python.info, Node: The Future for Python 2 x, Next: Changes to the Handling of Deprecation Warnings, Up: What’s New in Python 2 7
1.15.1 The Future for Python 2.x
--------------------------------
Python 2.7 is the last major release in the 2.x series, as the Python
maintainers have shifted the focus of their new feature development
efforts to the Python 3.x series. This means that while Python 2
continues to receive bug fixes, and to be updated to build correctly on
new hardware and versions of supported operated systems, there will be
no new full feature releases for the language or standard library.
However, while there is a large common subset between Python 2.7 and
Python 3, and many of the changes involved in migrating to that common
subset, or directly to Python 3, can be safely automated, some other
changes (notably those associated with Unicode handling) may require
careful consideration, and preferably robust automated regression test
suites, to migrate effectively.
This means that Python 2.7 will remain in place for a long time,
providing a stable and supported base platform for production systems
that have not yet been ported to Python 3. The full expected lifecycle
of the Python 2.7 series is detailed in PEP 373(1).
Some key consequences of the long-term significance of 2.7 are:
* As noted above, the 2.7 release has a much longer period of
maintenance when compared to earlier 2.x versions. Python 2.7 is
currently expected to remain supported by the core development team
(receiving security updates and other bug fixes) until at least
2020 (10 years after its initial release, compared to the more
typical support period of 18–24 months).
* As the Python 2.7 standard library ages, making effective use of
the Python Package Index (either directly or via a redistributor)
becomes more important for Python 2 users. In addition to a wide
variety of third party packages for various tasks, the available
packages include backports of new modules and features from the
Python 3 standard library that are compatible with Python 2, as
well as various tools and libraries that can make it easier to
migrate to Python 3. The Python Packaging User Guide(2) provides
guidance on downloading and installing software from the Python
Package Index.
* While the preferred approach to enhancing Python 2 is now the
publication of new packages on the Python Package Index, this
approach doesn’t necessarily work in all cases, especially those
related to network security. In exceptional cases that cannot be
handled adequately by publishing new or updated packages on PyPI,
the Python Enhancement Proposal process may be used to make the
case for adding new features directly to the Python 2 standard
library. Any such additions, and the maintenance releases where
they were added, will be noted in the *note New Features Added to
Python 2.7 Maintenance Releases: 12d9. section below.
For projects wishing to migrate from Python 2 to Python 3, or for
library and framework developers wishing to support users on both Python
2 and Python 3, there are a variety of tools and guides available to
help decide on a suitable approach and manage some of the technical
details involved. The recommended starting point is the pyporting-howto
HOWTO guide.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0373/
(2) https://packaging.python.org
File: python.info, Node: Changes to the Handling of Deprecation Warnings, Next: Python 3 1 Features, Prev: The Future for Python 2 x, Up: What’s New in Python 2 7
1.15.2 Changes to the Handling of Deprecation Warnings
------------------------------------------------------
For Python 2.7, a policy decision was made to silence warnings only of
interest to developers by default. *note DeprecationWarning: 1a5. and
its descendants are now ignored unless otherwise requested, preventing
users from seeing warnings triggered by an application. This change was
also made in the branch that became Python 3.2. (Discussed on
stdlib-sig and carried out in bpo-7319(1).)
In previous releases, *note DeprecationWarning: 1a5. messages were
enabled by default, providing Python developers with a clear indication
of where their code may break in a future major version of Python.
However, there are increasingly many users of Python-based applications
who are not directly involved in the development of those applications.
*note DeprecationWarning: 1a5. messages are irrelevant to such users,
making them worry about an application that’s actually working correctly
and burdening application developers with responding to these concerns.
You can re-enable display of *note DeprecationWarning: 1a5. messages by
running Python with the *note -Wdefault: 8c6. (short form: *note -Wd:
8c6.) switch, or by setting the *note PYTHONWARNINGS: baa. environment
variable to ‘"default"’ (or ‘"d"’) before running Python. Python code
can also re-enable them by calling ‘warnings.simplefilter('default')’.
The ‘unittest’ module also automatically reenables deprecation warnings
when running tests.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=7319
File: python.info, Node: Python 3 1 Features, Next: PEP 372 Adding an Ordered Dictionary to collections, Prev: Changes to the Handling of Deprecation Warnings, Up: What’s New in Python 2 7
1.15.3 Python 3.1 Features
--------------------------
Much as Python 2.6 incorporated features from Python 3.0, version 2.7
incorporates some of the new features in Python 3.1. The 2.x series
continues to provide tools for migrating to the 3.x series.
A partial list of 3.1 features that were backported to 2.7:
* The syntax for set literals (‘{1,2,3}’ is a mutable set).
* Dictionary and set comprehensions (‘{i: i*2 for i in range(3)}’).
* Multiple context managers in a single *note with: 5ce. statement.
* A new version of the *note io: 7f. library, rewritten in C for
performance.
* The ordered-dictionary type described in *note PEP 372; Adding an
Ordered Dictionary to collections: 12dc.
* The new ‘","’ format specifier described in *note PEP 378; Format
Specifier for Thousands Separator: 12dd.
* The *note memoryview: 464. object.
* A small subset of the *note importlib: 77. module, *note described
below: 12de.
* The *note repr(): 7f9. of a float ‘x’ is shorter in many cases:
it’s now based on the shortest decimal string that’s guaranteed to
round back to ‘x’. As in previous versions of Python, it’s
guaranteed that ‘float(repr(x))’ recovers ‘x’.
* Float-to-string and string-to-float conversions are correctly
rounded. The *note round(): 12cd. function is also now correctly
rounded.
* The *note PyCapsule: 1266. type, used to provide a C API for
extension modules.
* The *note PyLong_AsLongAndOverflow(): 125a. C API function.
Other new Python3-mode warnings include:
* ‘operator.isCallable()’ and ‘operator.sequenceIncludes()’, which
are not supported in 3.x, now trigger warnings.
* The ‘-3’ switch now automatically enables the ‘-Qwarn’ switch that
causes warnings about using classic division with integers and long
integers.
File: python.info, Node: PEP 372 Adding an Ordered Dictionary to collections, Next: PEP 378 Format Specifier for Thousands Separator<2>, Prev: Python 3 1 Features, Up: What’s New in Python 2 7
1.15.4 PEP 372: Adding an Ordered Dictionary to collections
-----------------------------------------------------------
Regular Python dictionaries iterate over key/value pairs in arbitrary
order. Over the years, a number of authors have written alternative
implementations that remember the order that the keys were originally
inserted. Based on the experiences from those implementations, 2.7
introduces a new *note OrderedDict: 5d7. class in the *note collections:
1d. module.
The *note OrderedDict: 5d7. API provides the same interface as regular
dictionaries but iterates over keys and values in a guaranteed order
depending on when a key was first inserted:
>>> from collections import OrderedDict
>>> d = OrderedDict([('first', 1),
... ('second', 2),
... ('third', 3)])
>>> d.items()
[('first', 1), ('second', 2), ('third', 3)]
If a new entry overwrites an existing entry, the original insertion
position is left unchanged:
>>> d['second'] = 4
>>> d.items()
[('first', 1), ('second', 4), ('third', 3)]
Deleting an entry and reinserting it will move it to the end:
>>> del d['second']
>>> d['second'] = 5
>>> d.items()
[('first', 1), ('third', 3), ('second', 5)]
The *note popitem(): 12e0. method has an optional 'last' argument that
defaults to ‘True’. If 'last' is true, the most recently added key is
returned and removed; if it’s false, the oldest key is selected:
>>> od = OrderedDict([(x,0) for x in range(20)])
>>> od.popitem()
(19, 0)
>>> od.popitem()
(18, 0)
>>> od.popitem(last=False)
(0, 0)
>>> od.popitem(last=False)
(1, 0)
Comparing two ordered dictionaries checks both the keys and values, and
requires that the insertion order was the same:
>>> od1 = OrderedDict([('first', 1),
... ('second', 2),
... ('third', 3)])
>>> od2 = OrderedDict([('third', 3),
... ('first', 1),
... ('second', 2)])
>>> od1 == od2
False
>>> # Move 'third' key to the end
>>> del od2['third']; od2['third'] = 3
>>> od1 == od2
True
Comparing an *note OrderedDict: 5d7. with a regular dictionary ignores
the insertion order and just compares the keys and values.
How does the *note OrderedDict: 5d7. work? It maintains a doubly linked
list of keys, appending new keys to the list as they’re inserted. A
secondary dictionary maps keys to their corresponding list node, so
deletion doesn’t have to traverse the entire linked list and therefore
remains 'O'(1).
The standard library now supports use of ordered dictionaries in several
modules.
* The *note ConfigParser: 22. module uses them by default, meaning
that configuration files can now be read, modified, and then
written back in their original order.
* The *note _asdict(): 9e0. method for *note
collections.namedtuple(): 1ca. now returns an ordered dictionary
with the values appearing in the same order as the underlying tuple
indices.
* The *note json: 82. module’s *note JSONDecoder: d47. class
constructor was extended with an 'object_pairs_hook' parameter to
allow ‘OrderedDict’ instances to be built by the decoder. Support
was also added for third-party tools like PyYAML(1).
See also
........
PEP 372(2) - Adding an ordered dictionary to collections
PEP written by Armin Ronacher and Raymond Hettinger; implemented by
Raymond Hettinger.
---------- Footnotes ----------
(1) https://pyyaml.org/
(2) https://peps.python.org/pep-0372/
File: python.info, Node: PEP 378 Format Specifier for Thousands Separator<2>, Next: PEP 389 The argparse Module for Parsing Command Lines, Prev: PEP 372 Adding an Ordered Dictionary to collections, Up: What’s New in Python 2 7
1.15.5 PEP 378: Format Specifier for Thousands Separator
--------------------------------------------------------
To make program output more readable, it can be useful to add separators
to large numbers, rendering them as 18,446,744,073,709,551,616 instead
of 18446744073709551616.
The fully general solution for doing this is the *note locale: 86.
module, which can use different separators (“,” in North America, “.” in
Europe) and different grouping sizes, but *note locale: 86. is
complicated to use and unsuitable for multi-threaded applications where
different threads are producing output for different locales.
Therefore, a simple comma-grouping mechanism has been added to the
mini-language used by the *note str.format(): 61d. method. When
formatting a floating-point number, simply include a comma between the
width and the precision:
>>> '{:20,.2f}'.format(18446744073709551616.0)
'18,446,744,073,709,551,616.00'
When formatting an integer, include the comma after the width:
>>> '{:20,d}'.format(18446744073709551616)
'18,446,744,073,709,551,616'
This mechanism is not adaptable at all; commas are always used as the
separator and the grouping is always into three-digit groups. The
comma-formatting mechanism isn’t as general as the *note locale: 86.
module, but it’s easier to use.
See also
........
PEP 378(1) - Format Specifier for Thousands Separator
PEP written by Raymond Hettinger; implemented by Eric Smith.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0378/
File: python.info, Node: PEP 389 The argparse Module for Parsing Command Lines, Next: PEP 391 Dictionary-Based Configuration For Logging, Prev: PEP 378 Format Specifier for Thousands Separator<2>, Up: What’s New in Python 2 7
1.15.6 PEP 389: The argparse Module for Parsing Command Lines
-------------------------------------------------------------
The *note argparse: 6. module for parsing command-line arguments was
added as a more powerful replacement for the *note optparse: a0. module.
This means Python now supports three different modules for parsing
command-line arguments: *note getopt: 61, *note optparse: a0, and *note
argparse: 6. The *note getopt: 61. module closely resembles the C
library’s ‘getopt()’ function, so it remains useful if you’re writing a
Python prototype that will eventually be rewritten in C. *note optparse:
a0. becomes redundant, but there are no plans to remove it because there
are many scripts still using it, and there’s no automated way to update
these scripts. (Making the *note argparse: 6. API consistent with *note
optparse: a0.’s interface was discussed but rejected as too messy and
difficult.)
In short, if you’re writing a new script and don’t need to worry about
compatibility with earlier versions of Python, use *note argparse: 6.
instead of *note optparse: a0.
Here’s an example:
import argparse
parser = argparse.ArgumentParser(description='Command-line example.')
# Add optional switches
parser.add_argument('-v', action='store_true', dest='is_verbose',
help='produce verbose output')
parser.add_argument('-o', action='store', dest='output',
metavar='FILE',
help='direct output to FILE instead of stdout')
parser.add_argument('-C', action='store', type=int, dest='context',
metavar='NUM', default=0,
help='display NUM lines of added context')
# Allow any number of additional arguments.
parser.add_argument(nargs='*', action='store', dest='inputs',
help='input filenames (default is stdin)')
args = parser.parse_args()
print args.__dict__
Unless you override it, ‘-h’ and ‘--help’ switches are automatically
added, and produce neatly formatted output:
-> ./python.exe argparse-example.py --help
usage: argparse-example.py [-h] [-v] [-o FILE] [-C NUM] [inputs [inputs ...]]
Command-line example.
positional arguments:
inputs input filenames (default is stdin)
optional arguments:
-h, --help show this help message and exit
-v produce verbose output
-o FILE direct output to FILE instead of stdout
-C NUM display NUM lines of added context
As with *note optparse: a0, the command-line switches and arguments are
returned as an object with attributes named by the 'dest' parameters:
-> ./python.exe argparse-example.py -v
{'output': None,
'is_verbose': True,
'context': 0,
'inputs': []}
-> ./python.exe argparse-example.py -v -o /tmp/output -C 4 file1 file2
{'output': '/tmp/output',
'is_verbose': True,
'context': 4,
'inputs': ['file1', 'file2']}
*note argparse: 6. has much fancier validation than *note optparse: a0.;
you can specify an exact number of arguments as an integer, 0 or more
arguments by passing ‘'*'’, 1 or more by passing ‘'+'’, or an optional
argument with ‘'?'’. A top-level parser can contain sub-parsers to
define subcommands that have different sets of switches, as in ‘svn
commit’, ‘svn checkout’, etc. You can specify an argument’s type as
*note FileType: f16, which will automatically open files for you and
understands that ‘'-'’ means standard input or output.
See also
........
*note argparse: 6. documentation
The documentation page of the argparse module.
*note Migrating optparse code to argparse: 11a0.
Part of the Python documentation, describing how to convert code
that uses *note optparse: a0.
PEP 389(1) - argparse - New Command Line Parsing Module
PEP written and implemented by Steven Bethard.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0389/
File: python.info, Node: PEP 391 Dictionary-Based Configuration For Logging, Next: PEP 3106 Dictionary Views, Prev: PEP 389 The argparse Module for Parsing Command Lines, Up: What’s New in Python 2 7
1.15.7 PEP 391: Dictionary-Based Configuration For Logging
----------------------------------------------------------
The *note logging: 87. module is very flexible; applications can define
a tree of logging subsystems, and each logger in this tree can filter
out certain messages, format them differently, and direct messages to a
varying number of handlers.
All this flexibility can require a lot of configuration. You can write
Python statements to create objects and set their properties, but a
complex set-up requires verbose but boring code. *note logging: 87.
also supports a *note fileConfig(): b51. function that parses a file,
but the file format doesn’t support configuring filters, and it’s
messier to generate programmatically.
Python 2.7 adds a *note dictConfig(): 11a2. function that uses a
dictionary to configure logging. There are many ways to produce a
dictionary from different sources: construct one with code; parse a file
containing JSON; or use a YAML parsing library if one is installed. For
more information see *note Configuration functions: 12e4.
The following example configures two loggers, the root logger and a
logger named “network”. Messages sent to the root logger will be sent
to the system log using the syslog protocol, and messages to the
“network” logger will be written to a ‘network.log’ file that will be
rotated once the log reaches 1MB.
import logging
import logging.config
configdict = {
'version': 1, # Configuration schema in use; must be 1 for now
'formatters': {
'standard': {
'format': ('%(asctime)s %(name)-15s '
'%(levelname)-8s %(message)s')}},
'handlers': {'netlog': {'backupCount': 10,
'class': 'logging.handlers.RotatingFileHandler',
'filename': '/logs/network.log',
'formatter': 'standard',
'level': 'INFO',
'maxBytes': 1000000},
'syslog': {'class': 'logging.handlers.SysLogHandler',
'formatter': 'standard',
'level': 'ERROR'}},
# Specify all the subordinate loggers
'loggers': {
'network': {
'handlers': ['netlog']
}
},
# Specify properties of the root logger
'root': {
'handlers': ['syslog']
},
}
# Set up configuration
logging.config.dictConfig(configdict)
# As an example, log two error messages
logger = logging.getLogger('/')
logger.error('Database not found')
netlogger = logging.getLogger('network')
netlogger.error('Connection failed')
Three smaller enhancements to the *note logging: 87. module, all
implemented by Vinay Sajip, are:
* The *note SysLogHandler: 658. class now supports syslogging over
TCP. The constructor has a 'socktype' parameter giving the type of
socket to use, either *note socket.SOCK_DGRAM: 12e5. for UDP or
*note socket.SOCK_STREAM: 12e6. for TCP. The default protocol
remains UDP.
* *note Logger: b4f. instances gained a *note getChild(): 12e7.
method that retrieves a descendant logger using a relative path.
For example, once you retrieve a logger by doing ‘log =
getLogger('app')’, calling ‘log.getChild('network.listen')’ is
equivalent to ‘getLogger('app.network.listen')’.
* The *note LoggerAdapter: 12e8. class gained an *note
isEnabledFor(): 12e9. method that takes a 'level' and returns
whether the underlying logger would process a message of that level
of importance.
See also
........
PEP 391(1) - Dictionary-Based Configuration For Logging
PEP written and implemented by Vinay Sajip.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0391/
File: python.info, Node: PEP 3106 Dictionary Views, Next: PEP 3137 The memoryview Object, Prev: PEP 391 Dictionary-Based Configuration For Logging, Up: What’s New in Python 2 7
1.15.8 PEP 3106: Dictionary Views
---------------------------------
The dictionary methods *note keys(): 7c8, *note values(): 7c9, and *note
items(): 7ca. are different in Python 3.x. They return an object called
a 'view' instead of a fully materialized list.
It’s not possible to change the return values of *note keys(): 7c8,
*note values(): 7c9, and *note items(): 7ca. in Python 2.7 because too
much code would break. Instead the 3.x versions were added under the
new names ‘viewkeys()’, ‘viewvalues()’, and ‘viewitems()’.
>>> d = dict((i*10, chr(65+i)) for i in range(26))
>>> d
{0: 'A', 130: 'N', 10: 'B', 140: 'O', 20: ..., 250: 'Z'}
>>> d.viewkeys()
dict_keys([0, 130, 10, 140, 20, 150, 30, ..., 250])
Views can be iterated over, but the key and item views also behave like
sets. The ‘&’ operator performs intersection, and ‘|’ performs a union:
>>> d1 = dict((i*10, chr(65+i)) for i in range(26))
>>> d2 = dict((i**.5, i) for i in range(1000))
>>> d1.viewkeys() & d2.viewkeys()
set([0.0, 10.0, 20.0, 30.0])
>>> d1.viewkeys() | range(0, 30)
set([0, 1, 130, 3, 4, 5, 6, ..., 120, 250])
The view keeps track of the dictionary and its contents change as the
dictionary is modified:
>>> vk = d.viewkeys()
>>> vk
dict_keys([0, 130, 10, ..., 250])
>>> d[260] = '&'
>>> vk
dict_keys([0, 130, 260, 10, ..., 250])
However, note that you can’t add or remove keys while you’re iterating
over the view:
>>> for k in vk:
... d[k*2] = k
...
Traceback (most recent call last):
File "", line 1, in
RuntimeError: dictionary changed size during iteration
You can use the view methods in Python 2.x code, and the 2to3 converter
will change them to the standard *note keys(): 7c8, *note values(): 7c9,
and *note items(): 7ca. methods.
See also
........
PEP 3106(1) - Revamping dict.keys(), .values() and .items()
PEP written by Guido van Rossum. Backported to 2.7 by Alexandre
Vassalotti; bpo-1967(2).
---------- Footnotes ----------
(1) https://peps.python.org/pep-3106/
(2) https://bugs.python.org/issue?@action=redirect&bpo=1967
File: python.info, Node: PEP 3137 The memoryview Object, Next: Other Language Changes<14>, Prev: PEP 3106 Dictionary Views, Up: What’s New in Python 2 7
1.15.9 PEP 3137: The memoryview Object
--------------------------------------
The *note memoryview: 464. object provides a view of another object’s
memory content that matches the *note bytes: 1c2. type’s interface.
>>> import string
>>> m = memoryview(string.letters)
>>> m
>>> len(m) # Returns length of underlying object
52
>>> m[0], m[25], m[26] # Indexing returns one byte
('a', 'z', 'A')
>>> m2 = m[0:26] # Slicing returns another memoryview
>>> m2
The content of the view can be converted to a string of bytes or a list
of integers:
>>> m2.tobytes()
'abcdefghijklmnopqrstuvwxyz'
>>> m2.tolist()
[97, 98, 99, 100, 101, 102, 103, ... 121, 122]
>>>
*note memoryview: 464. objects allow modifying the underlying object if
it’s a mutable object.
>>> m2[0] = 75
Traceback (most recent call last):
File "", line 1, in
TypeError: cannot modify read-only memory
>>> b = bytearray(string.letters) # Creating a mutable object
>>> b
bytearray(b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ')
>>> mb = memoryview(b)
>>> mb[0] = '*' # Assign to view, changing the bytearray.
>>> b[0:5] # The bytearray has been changed.
bytearray(b'*bcde')
>>>
See also
........
PEP 3137(1) - Immutable Bytes and Mutable Buffer
PEP written by Guido van Rossum. Implemented by Travis Oliphant,
Antoine Pitrou and others. Backported to 2.7 by Antoine Pitrou;
bpo-2396(2).
---------- Footnotes ----------
(1) https://peps.python.org/pep-3137/
(2) https://bugs.python.org/issue?@action=redirect&bpo=2396
File: python.info, Node: Other Language Changes<14>, Next: New and Improved Modules, Prev: PEP 3137 The memoryview Object, Up: What’s New in Python 2 7
1.15.10 Other Language Changes
------------------------------
Some smaller changes made to the core Python language are:
* The syntax for set literals has been backported from Python 3.x.
Curly brackets are used to surround the contents of the resulting
mutable set; set literals are distinguished from dictionaries by
not containing colons and values. ‘{}’ continues to represent an
empty dictionary; use ‘set()’ for an empty set.
>>> {1, 2, 3, 4, 5}
set([1, 2, 3, 4, 5])
>>> set() # empty set
set([])
>>> {} # empty dict
{}
Backported by Alexandre Vassalotti; bpo-2335(1).
* Dictionary and set comprehensions are another feature backported
from 3.x, generalizing list/generator comprehensions to use the
literal syntax for sets and dictionaries.
>>> {x: x*x for x in range(6)}
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}
>>> {('a'*x) for x in range(6)}
set(['', 'a', 'aa', 'aaa', 'aaaa', 'aaaaa'])
Backported by Alexandre Vassalotti; bpo-2333(2).
* The *note with: 5ce. statement can now use multiple context
managers in one statement. Context managers are processed from
left to right and each one is treated as beginning a new ‘with’
statement. This means that:
with A() as a, B() as b:
... suite of statements ...
is equivalent to:
with A() as a:
with B() as b:
... suite of statements ...
The ‘contextlib.nested()’ function provides a very similar
function, so it’s no longer necessary and has been deprecated.
(Proposed in ‘https://codereview.appspot.com/53094’; implemented by
Georg Brandl.)
* Conversions between floating-point numbers and strings are now
correctly rounded on most platforms. These conversions occur in
many different places: *note str(): 447. on floats and complex
numbers; the *note float: 2f1. and *note complex: 2f2.
constructors; numeric formatting; serializing and deserializing
floats and complex numbers using the *note marshal: 8d, *note
pickle: a6. and *note json: 82. modules; parsing of float and
imaginary literals in Python code; and *note Decimal: 29f.-to-float
conversion.
Related to this, the *note repr(): 7f9. of a floating-point number
'x' now returns a result based on the shortest decimal string
that’s guaranteed to round back to 'x' under correct rounding (with
round-half-to-even rounding mode). Previously it gave a string
based on rounding x to 17 decimal digits.
The rounding library responsible for this improvement works on
Windows and on Unix platforms using the gcc, icc, or suncc
compilers. There may be a small number of platforms where correct
operation of this code cannot be guaranteed, so the code is not
used on such systems. You can find out which code is being used by
checking *note sys.float_repr_style: 12ed, which will be ‘short’ if
the new code is in use and ‘legacy’ if it isn’t.
Implemented by Eric Smith and Mark Dickinson, using David Gay’s
‘dtoa.c’ library; bpo-7117(3).
* Conversions from long integers and regular integers to floating
point now round differently, returning the floating-point number
closest to the number. This doesn’t matter for small integers that
can be converted exactly, but for large numbers that will
unavoidably lose precision, Python 2.7 now approximates more
closely. For example, Python 2.6 computed the following:
>>> n = 295147905179352891391
>>> float(n)
2.9514790517935283e+20
>>> n - long(float(n))
65535L
Python 2.7’s floating-point result is larger, but much closer to
the true value:
>>> n = 295147905179352891391
>>> float(n)
2.9514790517935289e+20
>>> n - long(float(n))
-1L
(Implemented by Mark Dickinson; bpo-3166(4).)
Integer division is also more accurate in its rounding behaviours.
(Also implemented by Mark Dickinson; bpo-1811(5).)
* Implicit coercion for complex numbers has been removed; the
interpreter will no longer ever attempt to call a ‘__coerce__()’
method on complex objects. (Removed by Meador Inge and Mark
Dickinson; bpo-5211(6).)
* The *note str.format(): 61d. method now supports automatic
numbering of the replacement fields. This makes using *note
str.format(): 61d. more closely resemble using ‘%s’ formatting:
>>> '{}:{}:{}'.format(2009, 04, 'Sunday')
'2009:4:Sunday'
>>> '{}:{}:{day}'.format(2009, 4, day='Sunday')
'2009:4:Sunday'
The auto-numbering takes the fields from left to right, so the
first ‘{...}’ specifier will use the first argument to *note
str.format(): 61d, the next specifier will use the next argument,
and so on. You can’t mix auto-numbering and explicit numbering –
either number all of your specifier fields or none of them – but
you can mix auto-numbering and named fields, as in the second
example above. (Contributed by Eric Smith; bpo-5237(7).)
Complex numbers now correctly support usage with *note format():
61b, and default to being right-aligned. Specifying a precision or
comma-separation applies to both the real and imaginary parts of
the number, but a specified field width and alignment is applied to
the whole of the resulting ‘1.5+3j’ output. (Contributed by Eric
Smith; bpo-1588(8) and bpo-7988(9).)
The ‘F’ format code now always formats its output using uppercase
characters, so it will now produce ‘INF’ and ‘NAN’. (Contributed
by Eric Smith; bpo-3382(10).)
A low-level change: the *note object.__format__(): 61a. method now
triggers a *note PendingDeprecationWarning: 8c7. if it’s passed a
format string, because the ‘__format__()’ method for *note object:
a8c. converts the object to a string representation and formats
that. Previously the method silently applied the format string to
the string representation, but that could hide mistakes in Python
code. If you’re supplying formatting information such as an
alignment or precision, presumably you’re expecting the formatting
to be applied in some object-specific way. (Fixed by Eric Smith;
bpo-7994(11).)
* The *note int(): 259. and ‘long()’ types gained a ‘bit_length’
method that returns the number of bits necessary to represent its
argument in binary:
>>> n = 37
>>> bin(n)
'0b100101'
>>> n.bit_length()
6
>>> n = 2**123-1
>>> n.bit_length()
123
>>> (n+1).bit_length()
124
(Contributed by Fredrik Johansson and Victor Stinner;
bpo-3439(12).)
* The *note import: 5de. statement will no longer try an absolute
import if a relative import (e.g. ‘from .os import sep’) fails.
This fixes a bug, but could possibly break certain ‘import’
statements that were only working by accident. (Fixed by Meador
Inge; bpo-7902(13).)
* It’s now possible for a subclass of the built-in ‘unicode’ type to
override the ‘__unicode__()’ method. (Implemented by Victor
Stinner; bpo-1583863(14).)
* The *note bytearray: 53a. type’s *note translate(): 12ee. method
now accepts ‘None’ as its first argument. (Fixed by Georg Brandl;
bpo-4759(15).)
* When using *note @classmethod: 166. and *note @staticmethod: 412.
to wrap methods as class or static methods, the wrapper object now
exposes the wrapped function as their *note __func__: 12ef.
attribute. (Contributed by Amaury Forgeot d’Arc, after a
suggestion by George Sakkis; bpo-5982(16).)
* When a restricted set of attributes were set using ‘__slots__’,
deleting an unset attribute would not raise *note AttributeError:
348. as you would expect. Fixed by Benjamin Peterson;
bpo-7604(17).)
* Two new encodings are now supported: “cp720”, used primarily for
Arabic text; and “cp858”, a variant of CP 850 that adds the euro
symbol. (CP720 contributed by Alexander Belchenko and Amaury
Forgeot d’Arc in bpo-1616979(18); CP858 contributed by Tim Hatch in
bpo-8016(19).)
* The ‘file’ object will now set the ‘filename’ attribute on the
*note IOError: 104b. exception when trying to open a directory on
POSIX platforms (noted by Jan Kaliszewski; bpo-4764(20)), and now
explicitly checks for and forbids writing to read-only file objects
instead of trusting the C library to catch and report the error
(fixed by Stefan Krah; bpo-5677(21)).
* The Python tokenizer now translates line endings itself, so the
*note compile(): 192. built-in function now accepts code using any
line-ending convention. Additionally, it no longer requires that
the code end in a newline.
* Extra parentheses in function definitions are illegal in Python
3.x, meaning that you get a syntax error from ‘def f((x)): pass’.
In Python3-warning mode, Python 2.7 will now warn about this odd
usage. (Noted by James Lingard; bpo-7362(22).)
* It’s now possible to create weak references to old-style class
objects. New-style classes were always weak-referenceable. (Fixed
by Antoine Pitrou; bpo-8268(23).)
* When a module object is garbage-collected, the module’s dictionary
is now only cleared if no one else is holding a reference to the
dictionary (bpo-7140(24)).
* Menu:
* Interpreter Changes::
* Optimizations: Optimizations<13>.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2335
(2) https://bugs.python.org/issue?@action=redirect&bpo=2333
(3) https://bugs.python.org/issue?@action=redirect&bpo=7117
(4) https://bugs.python.org/issue?@action=redirect&bpo=3166
(5) https://bugs.python.org/issue?@action=redirect&bpo=1811
(6) https://bugs.python.org/issue?@action=redirect&bpo=5211
(7) https://bugs.python.org/issue?@action=redirect&bpo=5237
(8) https://bugs.python.org/issue?@action=redirect&bpo=1588
(9) https://bugs.python.org/issue?@action=redirect&bpo=7988
(10) https://bugs.python.org/issue?@action=redirect&bpo=3382
(11) https://bugs.python.org/issue?@action=redirect&bpo=7994
(12) https://bugs.python.org/issue?@action=redirect&bpo=3439
(13) https://bugs.python.org/issue?@action=redirect&bpo=7902
(14) https://bugs.python.org/issue?@action=redirect&bpo=1583863
(15) https://bugs.python.org/issue?@action=redirect&bpo=4759
(16) https://bugs.python.org/issue?@action=redirect&bpo=5982
(17) https://bugs.python.org/issue?@action=redirect&bpo=7604
(18) https://bugs.python.org/issue?@action=redirect&bpo=1616979
(19) https://bugs.python.org/issue?@action=redirect&bpo=8016
(20) https://bugs.python.org/issue?@action=redirect&bpo=4764
(21) https://bugs.python.org/issue?@action=redirect&bpo=5677
(22) https://bugs.python.org/issue?@action=redirect&bpo=7362
(23) https://bugs.python.org/issue?@action=redirect&bpo=8268
(24) https://bugs.python.org/issue?@action=redirect&bpo=7140
File: python.info, Node: Interpreter Changes, Next: Optimizations<13>, Up: Other Language Changes<14>
1.15.10.1 Interpreter Changes
.............................
A new environment variable, *note PYTHONWARNINGS: baa, allows
controlling warnings. It should be set to a string containing warning
settings, equivalent to those used with the *note -W: 8c6. switch,
separated by commas. (Contributed by Brian Curtin; bpo-7301(1).)
For example, the following setting will print warnings every time they
occur, but turn warnings from the *note Cookie: 71. module into an
error. (The exact syntax for setting an environment variable varies
across operating systems and shells.)
export PYTHONWARNINGS=all,error:::Cookie:0
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=7301
File: python.info, Node: Optimizations<13>, Prev: Interpreter Changes, Up: Other Language Changes<14>
1.15.10.2 Optimizations
.......................
Several performance enhancements have been added:
* A new opcode was added to perform the initial setup for *note with:
5ce. statements, looking up the *note __enter__(): 5c4. and *note
__exit__(): 12f3. methods. (Contributed by Benjamin Peterson.)
* The garbage collector now performs better for one common usage
pattern: when many objects are being allocated without deallocating
any of them. This would previously take quadratic time for garbage
collection, but now the number of full garbage collections is
reduced as the number of objects on the heap grows. The new logic
only performs a full garbage collection pass when the middle
generation has been collected 10 times and when the number of
survivor objects from the middle generation exceeds 10% of the
number of objects in the oldest generation. (Suggested by Martin
von Löwis and implemented by Antoine Pitrou; bpo-4074(1).)
* The garbage collector tries to avoid tracking simple containers
which can’t be part of a cycle. In Python 2.7, this is now true
for tuples and dicts containing atomic types (such as ints,
strings, etc.). Transitively, a dict containing tuples of atomic
types won’t be tracked either. This helps reduce the cost of each
garbage collection by decreasing the number of objects to be
considered and traversed by the collector. (Contributed by Antoine
Pitrou; bpo-4688(2).)
* Long integers are now stored internally either in base ‘2**15’ or
in base ‘2**30’, the base being determined at build time.
Previously, they were always stored in base ‘2**15’. Using base
‘2**30’ gives significant performance improvements on 64-bit
machines, but benchmark results on 32-bit machines have been mixed.
Therefore, the default is to use base ‘2**30’ on 64-bit machines
and base ‘2**15’ on 32-bit machines; on Unix, there’s a new
configure option ‘--enable-big-digits’ that can be used to override
this default.
Apart from the performance improvements this change should be
invisible to end users, with one exception: for testing and
debugging purposes there’s a new structseq ‘sys.long_info’ that
provides information about the internal format, giving the number
of bits per digit and the size in bytes of the C type used to store
each digit:
>>> import sys
>>> sys.long_info
sys.long_info(bits_per_digit=30, sizeof_digit=4)
(Contributed by Mark Dickinson; bpo-4258(3).)
Another set of changes made long objects a few bytes smaller: 2
bytes smaller on 32-bit systems and 6 bytes on 64-bit.
(Contributed by Mark Dickinson; bpo-5260(4).)
* The division algorithm for long integers has been made faster by
tightening the inner loop, doing shifts instead of multiplications,
and fixing an unnecessary extra iteration. Various benchmarks show
speedups of between 50% and 150% for long integer divisions and
modulo operations. (Contributed by Mark Dickinson; bpo-5512(5).)
Bitwise operations are also significantly faster (initial patch by
Gregory Smith; bpo-1087418(6)).
* The implementation of ‘%’ checks for the left-side operand being a
Python string and special-cases it; this results in a 1–3%
performance increase for applications that frequently use ‘%’ with
strings, such as templating libraries. (Implemented by Collin
Winter; bpo-5176(7).)
* List comprehensions with an ‘if’ condition are compiled into faster
bytecode. (Patch by Antoine Pitrou, back-ported to 2.7 by Jeffrey
Yasskin; bpo-4715(8).)
* Converting an integer or long integer to a decimal string was made
faster by special-casing base 10 instead of using a generalized
conversion function that supports arbitrary bases. (Patch by
Gawain Bolton; bpo-6713(9).)
* The ‘split()’, ‘replace()’, ‘rindex()’, ‘rpartition()’, and
‘rsplit()’ methods of string-like types (strings, Unicode strings,
and *note bytearray: 53a. objects) now use a fast reverse-search
algorithm instead of a character-by-character scan. This is
sometimes faster by a factor of 10. (Added by Florent Xicluna;
bpo-7462(10) and bpo-7622(11).)
* The *note pickle: a6. and ‘cPickle’ modules now automatically
intern the strings used for attribute names, reducing memory usage
of the objects resulting from unpickling. (Contributed by Jake
McGuire; bpo-5084(12).)
* The ‘cPickle’ module now special-cases dictionaries, nearly halving
the time required to pickle them. (Contributed by Collin Winter;
bpo-5670(13).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4074
(2) https://bugs.python.org/issue?@action=redirect&bpo=4688
(3) https://bugs.python.org/issue?@action=redirect&bpo=4258
(4) https://bugs.python.org/issue?@action=redirect&bpo=5260
(5) https://bugs.python.org/issue?@action=redirect&bpo=5512
(6) https://bugs.python.org/issue?@action=redirect&bpo=1087418
(7) https://bugs.python.org/issue?@action=redirect&bpo=5176
(8) https://bugs.python.org/issue?@action=redirect&bpo=4715
(9) https://bugs.python.org/issue?@action=redirect&bpo=6713
(10) https://bugs.python.org/issue?@action=redirect&bpo=7462
(11) https://bugs.python.org/issue?@action=redirect&bpo=7622
(12) https://bugs.python.org/issue?@action=redirect&bpo=5084
(13) https://bugs.python.org/issue?@action=redirect&bpo=5670
File: python.info, Node: New and Improved Modules, Next: Build and C API Changes<8>, Prev: Other Language Changes<14>, Up: What’s New in Python 2 7
1.15.11 New and Improved Modules
--------------------------------
As in every release, Python’s standard library received a number of
enhancements and bug fixes. Here’s a partial list of the most notable
changes, sorted alphabetically by module name. Consult the ‘Misc/NEWS’
file in the source tree for a more complete list of changes, or look
through the Subversion logs for all the details.
* The *note bdb: f. module’s base debugging class *note Bdb: 12f5.
gained a feature for skipping modules. The constructor now takes
an iterable containing glob-style patterns such as ‘django.*’; the
debugger will not step into stack frames from a module that matches
one of these patterns. (Contributed by Maru Newby after a
suggestion by Senthil Kumaran; bpo-5142(1).)
* The *note binascii: 10. module now supports the buffer API, so it
can be used with *note memoryview: 464. instances and other similar
buffer objects. (Backported from 3.x by Florent Xicluna;
bpo-7703(2).)
* Updated module: the ‘bsddb’ module has been updated from
4.7.2devel9 to version 4.8.4 of the pybsddb package(3). The new
version features better Python 3.x compatibility, various bug
fixes, and adds several new BerkeleyDB flags and methods. (Updated
by Jesús Cea Avión; bpo-8156(4). The pybsddb changelog can be read
at ‘https://hg.jcea.es/pybsddb/file/tip/ChangeLog’.)
* The *note bz2: 13. module’s *note BZ2File: 863. now supports the
context management protocol, so you can write ‘with
bz2.BZ2File(...) as f:’. (Contributed by Hagen Fürstenau;
bpo-3860(5).)
* New class: the *note Counter: 108b. class in the *note collections:
1d. module is useful for tallying data. *note Counter: 108b.
instances behave mostly like dictionaries but return zero for
missing keys instead of raising a *note KeyError: 33f.:
>>> from collections import Counter
>>> c = Counter()
>>> for letter in 'here is a sample of english text':
... c[letter] += 1
...
>>> c
Counter({' ': 6, 'e': 5, 's': 3, 'a': 2, 'i': 2, 'h': 2,
'l': 2, 't': 2, 'g': 1, 'f': 1, 'm': 1, 'o': 1, 'n': 1,
'p': 1, 'r': 1, 'x': 1})
>>> c['e']
5
>>> c['z']
0
There are three additional *note Counter: 108b. methods. *note
most_common(): 12f6. returns the N most common elements and their
counts. *note elements(): 12f7. returns an iterator over the
contained elements, repeating each element as many times as its
count. *note subtract(): 11d0. takes an iterable and subtracts one
for each element instead of adding; if the argument is a dictionary
or another ‘Counter’, the counts are subtracted.
>>> c.most_common(5)
[(' ', 6), ('e', 5), ('s', 3), ('a', 2), ('i', 2)]
>>> c.elements() ->
'a', 'a', ' ', ' ', ' ', ' ', ' ', ' ',
'e', 'e', 'e', 'e', 'e', 'g', 'f', 'i', 'i',
'h', 'h', 'm', 'l', 'l', 'o', 'n', 'p', 's',
's', 's', 'r', 't', 't', 'x'
>>> c['e']
5
>>> c.subtract('very heavy on the letter e')
>>> c['e'] # Count is now lower
-1
Contributed by Raymond Hettinger; bpo-1696199(6).
New class: *note OrderedDict: 5d7. is described in the earlier
section *note PEP 372; Adding an Ordered Dictionary to collections:
12dc.
New method: The *note deque: 5d8. data type now has a *note
count(): 11d2. method that returns the number of contained elements
equal to the supplied argument 'x', and a *note reverse(): 11d3.
method that reverses the elements of the deque in-place. *note
deque: 5d8. also exposes its maximum length as the read-only *note
maxlen: 12f8. attribute. (Both features added by Raymond
Hettinger.)
The *note namedtuple: 1ca. class now has an optional 'rename'
parameter. If 'rename' is true, field names that are invalid
because they’ve been repeated or aren’t legal Python identifiers
will be renamed to legal names that are derived from the field’s
position within the list of fields:
>>> from collections import namedtuple
>>> T = namedtuple('T', ['field1', '$illegal', 'for', 'field2'], rename=True)
>>> T._fields
('field1', '_1', '_2', 'field2')
(Added by Raymond Hettinger; bpo-1818(7).)
Finally, the *note Mapping: 8c9. abstract base class now returns
*note NotImplemented: 7cd. if a mapping is compared to another type
that isn’t a ‘Mapping’. (Fixed by Daniel Stutzbach; bpo-8729(8).)
* Constructors for the parsing classes in the *note ConfigParser: 22.
module now take an 'allow_no_value' parameter, defaulting to false;
if true, options without values will be allowed. For example:
>>> import ConfigParser, StringIO
>>> sample_config = """
... [mysqld]
... user = mysql
... pid-file = /var/run/mysqld/mysqld.pid
... skip-bdb
... """
>>> config = ConfigParser.RawConfigParser(allow_no_value=True)
>>> config.readfp(StringIO.StringIO(sample_config))
>>> config.get('mysqld', 'user')
'mysql'
>>> print config.get('mysqld', 'skip-bdb')
None
>>> print config.get('mysqld', 'unknown')
Traceback (most recent call last):
...
NoOptionError: No option 'unknown' in section: 'mysqld'
(Contributed by Mats Kindahl; bpo-7005(9).)
* Deprecated function: ‘contextlib.nested()’, which allows handling
more than one context manager with a single *note with: 5ce.
statement, has been deprecated, because the ‘with’ statement now
supports multiple context managers.
* The *note cookielib: 70. module now ignores cookies that have an
invalid version field, one that doesn’t contain an integer value.
(Fixed by John J. Lee; bpo-3924(10).)
* The *note copy: 25. module’s *note deepcopy(): b6e. function will
now correctly copy bound instance methods. (Implemented by Robert
Collins; bpo-1515(11).)
* The *note ctypes: 2a. module now always converts ‘None’ to a C
‘NULL’ pointer for arguments declared as pointers. (Changed by
Thomas Heller; bpo-4606(12).) The underlying libffi library(13)
has been updated to version 3.0.9, containing various fixes for
different platforms. (Updated by Matthias Klose; bpo-8142(14).)
* New method: the *note datetime: 30. module’s *note timedelta: 9cf.
class gained a *note total_seconds(): 12f9. method that returns the
number of seconds in the duration. (Contributed by Brian Quinlan;
bpo-5788(15).)
* New method: the *note Decimal: 29f. class gained a *note
from_float(): 11f6. class method that performs an exact conversion
of a floating-point number to a ‘Decimal’. This exact conversion
strives for the closest decimal approximation to the floating-point
representation’s value; the resulting decimal value will therefore
still include the inaccuracy, if any. For example,
‘Decimal.from_float(0.1)’ returns
‘Decimal('0.1000000000000000055511151231257827021181583404541015625')’.
(Implemented by Raymond Hettinger; bpo-4796(16).)
Comparing instances of *note Decimal: 29f. with floating-point
numbers now produces sensible results based on the numeric values
of the operands. Previously such comparisons would fall back to
Python’s default rules for comparing objects, which produced
arbitrary results based on their type. Note that you still cannot
combine ‘Decimal’ and floating point in other operations such as
addition, since you should be explicitly choosing how to convert
between float and ‘Decimal’. (Fixed by Mark Dickinson;
bpo-2531(17).)
The constructor for *note Decimal: 29f. now accepts floating-point
numbers (added by Raymond Hettinger; bpo-8257(18)) and non-European
Unicode characters such as Arabic-Indic digits (contributed by Mark
Dickinson; bpo-6595(19)).
Most of the methods of the *note Context: 10a5. class now accept
integers as well as *note Decimal: 29f. instances; the only
exceptions are the *note canonical(): 12fa. and *note
is_canonical(): 12fb. methods. (Patch by Juan José Conti;
bpo-7633(20).)
When using *note Decimal: 29f. instances with a string’s *note
format(): 61d. method, the default alignment was previously
left-alignment. This has been changed to right-alignment, which is
more sensible for numeric types. (Changed by Mark Dickinson;
bpo-6857(21).)
Comparisons involving a signaling NaN value (or ‘sNAN’) now signal
*note InvalidOperation: 109e. instead of silently returning a true
or false value depending on the comparison operator. Quiet NaN
values (or ‘NaN’) are now hashable. (Fixed by Mark Dickinson;
bpo-7279(22).)
* The *note difflib: 37. module now produces output that is more
compatible with modern ‘diff’/‘patch’ tools through one small
change, using a tab character instead of spaces as a separator in
the header giving the filename. (Fixed by Anatoly Techtonik;
bpo-7585(23).)
* The Distutils ‘sdist’ command now always regenerates the ‘MANIFEST’
file, since even if the ‘MANIFEST.in’ or ‘setup.py’ files haven’t
been modified, the user might have created some new files that
should be included. (Fixed by Tarek Ziadé; bpo-8688(24).)
* The *note doctest: 3a. module’s *note IGNORE_EXCEPTION_DETAIL:
12fc. flag will now ignore the name of the module containing the
exception being tested. (Patch by Lennart Regebro; bpo-7490(25).)
* The *note email: 3b. module’s *note Message: 27e. class will now
accept a Unicode-valued payload, automatically converting the
payload to the encoding specified by ‘output_charset’. (Added by
R. David Murray; bpo-1368247(26).)
* The *note Fraction: 1e9. class now accepts a single float or *note
Decimal: 29f. instance, or two rational numbers, as arguments to
its constructor. (Implemented by Mark Dickinson; rationals added
in bpo-5812(27), and float/decimal in bpo-8294(28).)
Ordering comparisons (‘<’, ‘<=’, ‘>’, ‘>=’) between fractions and
complex numbers now raise a *note TypeError: 534. This fixes an
oversight, making the *note Fraction: 1e9. match the other numeric
types.
* New class: *note FTP_TLS: 8fa. in the *note ftplib: 5e. module
provides secure FTP connections using TLS encapsulation of
authentication as well as subsequent control and data transfers.
(Contributed by Giampaolo Rodola; bpo-2054(29).)
The *note storbinary(): 12fd. method for binary uploads can now
restart uploads thanks to an added 'rest' parameter (patch by Pablo
Mouzo; bpo-6845(30).)
* New class decorator: *note total_ordering(): f3d. in the *note
functools: 5f. module takes a class that defines an *note __eq__():
afa. method and one of *note __lt__(): 1292, *note __le__(): 12fe,
*note __gt__(): 12ff, or *note __ge__(): 1300, and generates the
missing comparison methods. Since the ‘__cmp__()’ method is being
deprecated in Python 3.x, this decorator makes it easier to define
ordered classes. (Added by Raymond Hettinger; bpo-5479(31).)
New function: *note cmp_to_key(): 11cc. will take an old-style
comparison function that expects two arguments and return a new
callable that can be used as the 'key' parameter to functions such
as *note sorted(): bce, *note min(): f03. and *note max(): f04,
etc. The primary intended use is to help with making code
compatible with Python 3.x. (Added by Raymond Hettinger.)
* New function: the *note gc: 60. module’s *note is_tracked(): 1301.
returns true if a given instance is tracked by the garbage
collector, false otherwise. (Contributed by Antoine Pitrou;
bpo-4688(32).)
* The *note gzip: 67. module’s *note GzipFile: 416. now supports the
context management protocol, so you can write ‘with
gzip.GzipFile(...) as f:’ (contributed by Hagen Fürstenau;
bpo-3860(33)), and it now implements the *note io.BufferedIOBase:
690. ABC, so you can wrap it with *note io.BufferedReader: 1200.
for faster processing (contributed by Nir Aides; bpo-7471(34)).
It’s also now possible to override the modification time recorded
in a gzipped file by providing an optional timestamp to the
constructor. (Contributed by Jacques Frechet; bpo-4272(35).)
Files in gzip format can be padded with trailing zero bytes; the
*note gzip: 67. module will now consume these trailing bytes.
(Fixed by Tadek Pietraszek and Brian Curtin; bpo-2846(36).)
* New attribute: the *note hashlib: 68. module now has an
‘algorithms’ attribute containing a tuple naming the supported
algorithms. In Python 2.7, ‘hashlib.algorithms’ contains ‘('md5',
'sha1', 'sha224', 'sha256', 'sha384', 'sha512')’. (Contributed by
Carl Chenet; bpo-7418(37).)
* The default *note HTTPResponse: 10c4. class used by the *note
httplib: 6e. module now supports buffering, resulting in much
faster reading of HTTP responses. (Contributed by Kristján Valur
Jónsson; bpo-4879(38).)
The *note HTTPConnection: b40. and *note HTTPSConnection: b41.
classes now support a 'source_address' parameter, a ‘(host, port)’
2-tuple giving the source address that will be used for the
connection. (Contributed by Eldon Ziegler; bpo-3972(39).)
* The ‘ihooks’ module now supports relative imports. Note that
‘ihooks’ is an older module for customizing imports, superseded by
the ‘imputil’ module added in Python 2.0. (Relative import support
added by Neil Schemenauer.)
* The *note imaplib: 74. module now supports IPv6 addresses.
(Contributed by Derek Morr; bpo-1655(40).)
* New function: the *note inspect: 7e. module’s *note getcallargs():
eb7. takes a callable and its positional and keyword arguments, and
figures out which of the callable’s parameters will receive each
argument, returning a dictionary mapping argument names to their
values. For example:
>>> from inspect import getcallargs
>>> def f(a, b=1, *pos, **named):
... pass
...
>>> getcallargs(f, 1, 2, 3)
{'a': 1, 'b': 2, 'pos': (3,), 'named': {}}
>>> getcallargs(f, a=2, x=4)
{'a': 2, 'b': 1, 'pos': (), 'named': {'x': 4}}
>>> getcallargs(f)
Traceback (most recent call last):
...
TypeError: f() takes at least 1 argument (0 given)
Contributed by George Sakkis; bpo-3135(41).
* Updated module: The *note io: 7f. library has been upgraded to the
version shipped with Python 3.1. For 3.1, the I/O library was
entirely rewritten in C and is 2 to 20 times faster depending on
the task being performed. The original Python version was renamed
to the ‘_pyio’ module.
One minor resulting change: the *note io.TextIOBase: 691. class now
has an *note errors: 1302. attribute giving the error setting used
for encoding and decoding errors (one of ‘'strict'’, ‘'replace'’,
‘'ignore'’).
The *note io.FileIO: 1303. class now raises an *note OSError: 413.
when passed an invalid file descriptor. (Implemented by Benjamin
Peterson; bpo-4991(42).) The *note truncate(): 1304. method now
preserves the file position; previously it would change the file
position to the end of the new file. (Fixed by Pascal Chambon;
bpo-6939(43).)
* New function: ‘itertools.compress(data, selectors)’ takes two
iterators. Elements of 'data' are returned if the corresponding
value in 'selectors' is true:
itertools.compress('ABCDEF', [1,0,1,0,1,1]) =>
A, C, E, F
New function: ‘itertools.combinations_with_replacement(iter, r)’
returns all the possible 'r'-length combinations of elements from
the iterable 'iter'. Unlike *note combinations(): 1305, individual
elements can be repeated in the generated combinations:
itertools.combinations_with_replacement('abc', 2) =>
('a', 'a'), ('a', 'b'), ('a', 'c'),
('b', 'b'), ('b', 'c'), ('c', 'c')
Note that elements are treated as unique depending on their
position in the input, not their actual values.
The *note itertools.count(): 1276. function now has a 'step'
argument that allows incrementing by values other than 1. *note
count(): 1276. also now allows keyword arguments, and using
non-integer values such as floats or *note Decimal: 29f. instances.
(Implemented by Raymond Hettinger; bpo-5032(44).)
*note itertools.combinations(): 1305. and *note
itertools.product(): 1306. previously raised *note ValueError: 204.
for values of 'r' larger than the input iterable. This was deemed
a specification error, so they now return an empty iterator.
(Fixed by Raymond Hettinger; bpo-4816(45).)
* Updated module: The *note json: 82. module was upgraded to version
2.0.9 of the simplejson package, which includes a C extension that
makes encoding and decoding faster. (Contributed by Bob Ippolito;
bpo-4136(46).)
To support the new *note collections.OrderedDict: 5d7. type, *note
json.load(): cb4. now has an optional 'object_pairs_hook' parameter
that will be called with any object literal that decodes to a list
of pairs. (Contributed by Raymond Hettinger; bpo-5381(47).)
* The *note mailbox: 8b. module’s *note Maildir: 418. class now
records the timestamp on the directories it reads, and only
re-reads them if the modification time has subsequently changed.
This improves performance by avoiding unneeded directory scans.
(Fixed by A.M. Kuchling and Antoine Pitrou; bpo-1607951(48),
bpo-6896(49).)
* New functions: the *note math: 8e. module gained *note erf(): bd8.
and *note erfc(): bd9. for the error function and the complementary
error function, *note expm1(): 11de. which computes ‘e**x - 1’ with
more precision than using *note exp(): 1307. and subtracting 1,
*note gamma(): 11df. for the Gamma function, and *note lgamma():
11e0. for the natural log of the Gamma function. (Contributed by
Mark Dickinson and nirinA raseliarison; bpo-3366(50).)
* The *note multiprocessing: 94. module’s ‘Manager*’ classes can now
be passed a callable that will be called whenever a subprocess is
started, along with a set of arguments that will be passed to the
callable. (Contributed by lekma; bpo-5585(51).)
The *note Pool: f66. class, which controls a pool of worker
processes, now has an optional 'maxtasksperchild' parameter.
Worker processes will perform the specified number of tasks and
then exit, causing the ‘Pool’ to start a new worker. This is
useful if tasks may leak memory or other resources, or if some
tasks will cause the worker to become very large. (Contributed by
Charles Cazabon; bpo-6963(52).)
* The ‘nntplib’ module now supports IPv6 addresses. (Contributed by
Derek Morr; bpo-1664(53).)
* New functions: the *note os: a1. module wraps the following POSIX
system calls: *note getresgid(): 1308. and *note getresuid(): 1309,
which return the real, effective, and saved GIDs and UIDs; *note
setresgid(): 130a. and *note setresuid(): 130b, which set real,
effective, and saved GIDs and UIDs to new values; *note
initgroups(): 130c, which initialize the group access list for the
current process. (GID/UID functions contributed by Travis H.;
bpo-6508(54). Support for initgroups added by Jean-Paul Calderone;
bpo-7333(55).)
The *note os.fork(): 197. function now re-initializes the import
lock in the child process; this fixes problems on Solaris when
*note fork(): 197. is called from a thread. (Fixed by Zsolt
Cserna; bpo-7242(56).)
* In the *note os.path: a2. module, the *note normpath(): 130d. and
*note abspath(): 130e. functions now preserve Unicode; if their
input path is a Unicode string, the return value is also a Unicode
string. (*note normpath(): 130d. fixed by Matt Giuca in
bpo-5827(57); *note abspath(): 130e. fixed by Ezio Melotti in
bpo-3426(58).)
* The *note pydoc: b5. module now has help for the various symbols
that Python uses. You can now do ‘help('<<')’ or ‘help('@')’, for
example. (Contributed by David Laban; bpo-4739(59).)
* The *note re: b9. module’s *note split(): 2a4, *note sub(): 2a5,
and *note subn(): 2a6. now accept an optional 'flags' argument, for
consistency with the other functions in the module. (Added by
Gregory P. Smith.)
* New function: *note run_path(): 181. in the *note runpy: be. module
will execute the code at a provided 'path' argument. 'path' can be
the path of a Python source file (‘example.py’), a compiled
bytecode file (‘example.pyc’), a directory (‘./package/’), or a zip
archive (‘example.zip’). If a directory or zip path is provided,
it will be added to the front of ‘sys.path’ and the module *note
__main__: 1. will be imported. It’s expected that the directory or
zip contains a ‘__main__.py’; if it doesn’t, some other
‘__main__.py’ might be imported from a location later in
‘sys.path’. This makes more of the machinery of *note runpy: be.
available to scripts that want to mimic the way Python’s command
line processes an explicit path name. (Added by Nick Coghlan;
bpo-6816(60).)
* New function: in the *note shutil: c5. module, *note
make_archive(): 4af. takes a filename, archive type (zip or
tar-format), and a directory path, and creates an archive
containing the directory’s contents. (Added by Tarek Ziadé.)
*note shutil: c5.’s *note copyfile(): a58. and *note copytree():
a28. functions now raise a ‘SpecialFileError’ exception when asked
to copy a named pipe. Previously the code would treat named pipes
like a regular file by opening them for reading, and this would
block indefinitely. (Fixed by Antoine Pitrou; bpo-3002(61).)
* The *note signal: c6. module no longer re-installs the signal
handler unless this is truly necessary, which fixes a bug that
could make it impossible to catch the EINTR signal robustly.
(Fixed by Charles-Francois Natali; bpo-8354(62).)
* New functions: in the *note site: c7. module, three new functions
return various site- and user-specific paths. *note
getsitepackages(): 1231. returns a list containing all global
site-packages directories, *note getusersitepackages(): 1233.
returns the path of the user’s site-packages directory, and *note
getuserbase(): 1232. returns the value of the *note USER_BASE:
130f. environment variable, giving the path to a directory that can
be used to store data. (Contributed by Tarek Ziadé; bpo-6693(63).)
The *note site: c7. module now reports exceptions occurring when
the *note sitecustomize: c8. module is imported, and will no longer
catch and swallow the *note KeyboardInterrupt: 9d1. exception.
(Fixed by Victor Stinner; bpo-3137(64).)
* The *note create_connection(): 66e. function gained a
'source_address' parameter, a ‘(host, port)’ 2-tuple giving the
source address that will be used for the connection. (Contributed
by Eldon Ziegler; bpo-3972(65).)
The *note recv_into(): 1310. and *note recvfrom_into(): 1311.
methods will now write into objects that support the buffer API,
most usefully the *note bytearray: 53a. and *note memoryview: 464.
objects. (Implemented by Antoine Pitrou; bpo-8104(66).)
* The *note SocketServer: cd. module’s *note TCPServer: 1312. class
now supports socket timeouts and disabling the Nagle algorithm.
The ‘disable_nagle_algorithm’ class attribute defaults to ‘False’;
if overridden to be true, new request connections will have the
TCP_NODELAY option set to prevent buffering many small sends into a
single TCP packet. The *note timeout: 1313. class attribute can
hold a timeout in seconds that will be applied to the request
socket; if no request is received within that time, *note
handle_timeout(): 1314. will be called and *note handle_request():
1315. will return. (Contributed by Kristján Valur Jónsson;
bpo-6192(67) and bpo-6267(68).)
* Updated module: the *note sqlite3: cf. module has been updated to
version 2.6.0 of the pysqlite package(69). Version 2.6.0 includes
a number of bugfixes, and adds the ability to load SQLite
extensions from shared libraries. Call the
‘enable_load_extension(True)’ method to enable extensions, and then
call *note load_extension(): 4b4. to load a particular shared
library. (Updated by Gerhard Häring.)
* The *note ssl: d0. module’s *note SSLSocket: 8e9. objects now
support the buffer API, which fixed a test suite failure (fix by
Antoine Pitrou; bpo-7133(70)) and automatically set OpenSSL’s
‘SSL_MODE_AUTO_RETRY’, which will prevent an error code being
returned from ‘recv()’ operations that trigger an SSL renegotiation
(fix by Antoine Pitrou; bpo-8222(71)).
The *note wrap_socket(): 520. constructor function now takes a
'ciphers' argument that’s a string listing the encryption
algorithms to be allowed; the format of the string is described in
the OpenSSL documentation(72). (Added by Antoine Pitrou;
bpo-8322(73).)
Another change makes the extension load all of OpenSSL’s ciphers
and digest algorithms so that they’re all available. Some SSL
certificates couldn’t be verified, reporting an “unknown algorithm”
error. (Reported by Beda Kosata, and fixed by Antoine Pitrou;
bpo-8484(74).)
The version of OpenSSL being used is now available as the module
attributes *note ssl.OPENSSL_VERSION: 1211. (a string), *note
ssl.OPENSSL_VERSION_INFO: 1212. (a 5-tuple), and *note
ssl.OPENSSL_VERSION_NUMBER: 1213. (an integer). (Added by Antoine
Pitrou; bpo-8321(75).)
* The *note struct: d5. module will no longer silently ignore
overflow errors when a value is too large for a particular integer
format code (one of ‘bBhHiIlLqQ’); it now always raises a *note
struct.error: 1316. exception. (Changed by Mark Dickinson;
bpo-1523(76).) The *note pack(): 126a. function will also attempt
to use *note __index__(): 718. to convert and pack non-integers
before trying the *note __int__(): 717. method or reporting an
error. (Changed by Mark Dickinson; bpo-8300(77).)
* New function: the *note subprocess: d6. module’s *note
check_output(): fbc. runs a command with a specified set of
arguments and returns the command’s output as a string when the
command runs without error, or raises a *note CalledProcessError:
1317. exception otherwise.
>>> subprocess.check_output(['df', '-h', '.'])
'Filesystem Size Used Avail Capacity Mounted on\n
/dev/disk0s2 52G 49G 3.0G 94% /\n'
>>> subprocess.check_output(['df', '-h', '/bogus'])
...
subprocess.CalledProcessError: Command '['df', '-h', '/bogus']' returned non-zero exit status 1
(Contributed by Gregory P. Smith.)
The *note subprocess: d6. module will now retry its internal system
calls on receiving an *note EINTR: d86. signal. (Reported by
several people; final patch by Gregory P. Smith in
bpo-1068268(78).)
* New function: *note is_declared_global(): 1318. in the *note
symtable: d8. module returns true for variables that are explicitly
declared to be global, false for ones that are implicitly global.
(Contributed by Jeremy Hylton.)
* The *note syslog: dc. module will now use the value of
‘sys.argv[0]’ as the identifier instead of the previous default
value of ‘'python'’. (Changed by Sean Reifschneider;
bpo-8451(79).)
* The *note sys.version_info: 69c. value is now a named tuple, with
attributes named ‘major’, ‘minor’, ‘micro’, ‘releaselevel’, and
‘serial’. (Contributed by Ross Light; bpo-4285(80).)
*note sys.getwindowsversion(): ce7. also returns a named tuple,
with attributes named ‘major’, ‘minor’, ‘build’, ‘platform’,
‘service_pack’, ‘service_pack_major’, ‘service_pack_minor’,
‘suite_mask’, and ‘product_type’. (Contributed by Brian Curtin;
bpo-7766(81).)
* The *note tarfile: de. module’s default error handling has changed,
to no longer suppress fatal errors. The default error level was
previously 0, which meant that errors would only result in a
message being written to the debug log, but because the debug log
is not activated by default, these errors go unnoticed. The
default error level is now 1, which raises an exception if there’s
an error. (Changed by Lars Gustäbel; bpo-7357(82).)
*note tarfile: de. now supports filtering the *note TarInfo: 1203.
objects being added to a tar file. When you call *note add(): bf3,
you may supply an optional 'filter' argument that’s a callable.
The 'filter' callable will be passed the *note TarInfo: 1203. for
every file being added, and can modify and return it. If the
callable returns ‘None’, the file will be excluded from the
resulting archive. This is more powerful than the existing
'exclude' argument, which has therefore been deprecated. (Added by
Lars Gustäbel; bpo-6856(83).) The *note TarFile: 1202. class also
now supports the context management protocol. (Added by Lars
Gustäbel; bpo-7232(84).)
* The *note wait(): 1319. method of the *note threading.Event: 114c.
class now returns the internal flag on exit. This means the method
will usually return true because *note wait(): 1319. is supposed to
block until the internal flag becomes true. The return value will
only be false if a timeout was provided and the operation timed
out. (Contributed by Tim Lesher; bpo-1674032(85).)
* The Unicode database provided by the *note unicodedata: 105. module
is now used internally to determine which characters are numeric,
whitespace, or represent line breaks. The database also includes
information from the ‘Unihan.txt’ data file (patch by Anders
Chrigström and Amaury Forgeot d’Arc; bpo-1571184(86)) and has been
updated to version 5.2.0 (updated by Florent Xicluna;
bpo-8024(87)).
* The *note urlparse: 10a. module’s *note urlsplit(): d3f. now
handles unknown URL schemes in a fashion compliant with RFC
3986(88): if the URL is of the form ‘"://..."’, the text
before the ‘://’ is treated as the scheme, even if it’s a made-up
scheme that the module doesn’t know about. This change may break
code that worked around the old behaviour. For example, Python
2.6.4 or 2.5 will return the following:
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', '', '//host/filename?query', '', '')
Python 2.7 (and Python 2.6.5) will return:
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', 'host', '/filename?query', '', '')
(Python 2.7 actually produces slightly different output, since it
returns a named tuple instead of a standard tuple.)
The *note urlparse: 10a. module also supports IPv6 literal
addresses as defined by RFC 2732(89) (contributed by Senthil
Kumaran; bpo-2987(90)).
>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo')
ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]',
path='/foo', params='', query='', fragment='')
* New class: the *note WeakSet: 5d9. class in the *note weakref: 114.
module is a set that only holds weak references to its elements;
elements will be removed once there are no references pointing to
them. (Originally implemented in Python 3.x by Raymond Hettinger,
and backported to 2.7 by Michael Foord.)
* The *note xml.etree.ElementTree: 125. library, no longer escapes
ampersands and angle brackets when outputting an XML processing
instruction (which looks like ‘’)
or comment (which looks like ‘’). (Patch by Neil
Muller; bpo-2746(91).)
* The XML-RPC client and server, provided by the *note xmlrpclib:
12e. and *note SimpleXMLRPCServer: 12f. modules, have improved
performance by supporting HTTP/1.1 keep-alive and by optionally
using gzip encoding to compress the XML being exchanged. The gzip
compression is controlled by the ‘encode_threshold’ attribute of
*note SimpleXMLRPCRequestHandler: 131a, which contains a size in
bytes; responses larger than this will be compressed. (Contributed
by Kristján Valur Jónsson; bpo-6267(92).)
* The *note zipfile: 131. module’s *note ZipFile: 6c0. now supports
the context management protocol, so you can write ‘with
zipfile.ZipFile(...) as f:’. (Contributed by Brian Curtin;
bpo-5511(93).)
*note zipfile: 131. now also supports archiving empty directories
and extracts them correctly. (Fixed by Kuba Wieczorek;
bpo-4710(94).) Reading files out of an archive is faster, and
interleaving *note read(): 131b. and *note readline(): 131c. now
works correctly. (Contributed by Nir Aides; bpo-7610(95).)
The *note is_zipfile(): 131d. function now accepts a file object,
in addition to the path names accepted in earlier versions.
(Contributed by Gabriel Genellina; bpo-4756(96).)
The *note writestr(): 131e. method now has an optional
'compress_type' parameter that lets you override the default
compression method specified in the *note ZipFile: 6c0.
constructor. (Contributed by Ronald Oussoren; bpo-6003(97).)
* Menu:
* New module; importlib: New module importlib.
* New module; sysconfig: New module sysconfig.
* ttk; Themed Widgets for Tk: ttk Themed Widgets for Tk.
* Updated module; unittest: Updated module unittest.
* Updated module; ElementTree 1.3: Updated module ElementTree 1 3.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5142
(2) https://bugs.python.org/issue?@action=redirect&bpo=7703
(3) https://www.jcea.es/programacion/pybsddb.htm
(4) https://bugs.python.org/issue?@action=redirect&bpo=8156
(5) https://bugs.python.org/issue?@action=redirect&bpo=3860
(6) https://bugs.python.org/issue?@action=redirect&bpo=1696199
(7) https://bugs.python.org/issue?@action=redirect&bpo=1818
(8) https://bugs.python.org/issue?@action=redirect&bpo=8729
(9) https://bugs.python.org/issue?@action=redirect&bpo=7005
(10) https://bugs.python.org/issue?@action=redirect&bpo=3924
(11) https://bugs.python.org/issue?@action=redirect&bpo=1515
(12) https://bugs.python.org/issue?@action=redirect&bpo=4606
(13) https://sourceware.org/libffi/
(14) https://bugs.python.org/issue?@action=redirect&bpo=8142
(15) https://bugs.python.org/issue?@action=redirect&bpo=5788
(16) https://bugs.python.org/issue?@action=redirect&bpo=4796
(17) https://bugs.python.org/issue?@action=redirect&bpo=2531
(18) https://bugs.python.org/issue?@action=redirect&bpo=8257
(19) https://bugs.python.org/issue?@action=redirect&bpo=6595
(20) https://bugs.python.org/issue?@action=redirect&bpo=7633
(21) https://bugs.python.org/issue?@action=redirect&bpo=6857
(22) https://bugs.python.org/issue?@action=redirect&bpo=7279
(23) https://bugs.python.org/issue?@action=redirect&bpo=7585
(24) https://bugs.python.org/issue?@action=redirect&bpo=8688
(25) https://bugs.python.org/issue?@action=redirect&bpo=7490
(26) https://bugs.python.org/issue?@action=redirect&bpo=1368247
(27) https://bugs.python.org/issue?@action=redirect&bpo=5812
(28) https://bugs.python.org/issue?@action=redirect&bpo=8294
(29) https://bugs.python.org/issue?@action=redirect&bpo=2054
(30) https://bugs.python.org/issue?@action=redirect&bpo=6845
(31) https://bugs.python.org/issue?@action=redirect&bpo=5479
(32) https://bugs.python.org/issue?@action=redirect&bpo=4688
(33) https://bugs.python.org/issue?@action=redirect&bpo=3860
(34) https://bugs.python.org/issue?@action=redirect&bpo=7471
(35) https://bugs.python.org/issue?@action=redirect&bpo=4272
(36) https://bugs.python.org/issue?@action=redirect&bpo=2846
(37) https://bugs.python.org/issue?@action=redirect&bpo=7418
(38) https://bugs.python.org/issue?@action=redirect&bpo=4879
(39) https://bugs.python.org/issue?@action=redirect&bpo=3972
(40) https://bugs.python.org/issue?@action=redirect&bpo=1655
(41) https://bugs.python.org/issue?@action=redirect&bpo=3135
(42) https://bugs.python.org/issue?@action=redirect&bpo=4991
(43) https://bugs.python.org/issue?@action=redirect&bpo=6939
(44) https://bugs.python.org/issue?@action=redirect&bpo=5032
(45) https://bugs.python.org/issue?@action=redirect&bpo=4816
(46) https://bugs.python.org/issue?@action=redirect&bpo=4136
(47) https://bugs.python.org/issue?@action=redirect&bpo=5381
(48) https://bugs.python.org/issue?@action=redirect&bpo=1607951
(49) https://bugs.python.org/issue?@action=redirect&bpo=6896
(50) https://bugs.python.org/issue?@action=redirect&bpo=3366
(51) https://bugs.python.org/issue?@action=redirect&bpo=5585
(52) https://bugs.python.org/issue?@action=redirect&bpo=6963
(53) https://bugs.python.org/issue?@action=redirect&bpo=1664
(54) https://bugs.python.org/issue?@action=redirect&bpo=6508
(55) https://bugs.python.org/issue?@action=redirect&bpo=7333
(56) https://bugs.python.org/issue?@action=redirect&bpo=7242
(57) https://bugs.python.org/issue?@action=redirect&bpo=5827
(58) https://bugs.python.org/issue?@action=redirect&bpo=3426
(59) https://bugs.python.org/issue?@action=redirect&bpo=4739
(60) https://bugs.python.org/issue?@action=redirect&bpo=6816
(61) https://bugs.python.org/issue?@action=redirect&bpo=3002
(62) https://bugs.python.org/issue?@action=redirect&bpo=8354
(63) https://bugs.python.org/issue?@action=redirect&bpo=6693
(64) https://bugs.python.org/issue?@action=redirect&bpo=3137
(65) https://bugs.python.org/issue?@action=redirect&bpo=3972
(66) https://bugs.python.org/issue?@action=redirect&bpo=8104
(67) https://bugs.python.org/issue?@action=redirect&bpo=6192
(68) https://bugs.python.org/issue?@action=redirect&bpo=6267
(69) https://github.com/ghaering/pysqlite
(70) https://bugs.python.org/issue?@action=redirect&bpo=7133
(71) https://bugs.python.org/issue?@action=redirect&bpo=8222
(72) https://docs.openssl.org/1.0.2/man1/ciphers/
(73) https://bugs.python.org/issue?@action=redirect&bpo=8322
(74) https://bugs.python.org/issue?@action=redirect&bpo=8484
(75) https://bugs.python.org/issue?@action=redirect&bpo=8321
(76) https://bugs.python.org/issue?@action=redirect&bpo=1523
(77) https://bugs.python.org/issue?@action=redirect&bpo=8300
(78) https://bugs.python.org/issue?@action=redirect&bpo=1068268
(79) https://bugs.python.org/issue?@action=redirect&bpo=8451
(80) https://bugs.python.org/issue?@action=redirect&bpo=4285
(81) https://bugs.python.org/issue?@action=redirect&bpo=7766
(82) https://bugs.python.org/issue?@action=redirect&bpo=7357
(83) https://bugs.python.org/issue?@action=redirect&bpo=6856
(84) https://bugs.python.org/issue?@action=redirect&bpo=7232
(85) https://bugs.python.org/issue?@action=redirect&bpo=1674032
(86) https://bugs.python.org/issue?@action=redirect&bpo=1571184
(87) https://bugs.python.org/issue?@action=redirect&bpo=8024
(88) https://datatracker.ietf.org/doc/html/rfc3986.html
(89) https://datatracker.ietf.org/doc/html/rfc2732.html
(90) https://bugs.python.org/issue?@action=redirect&bpo=2987
(91) https://bugs.python.org/issue?@action=redirect&bpo=2746
(92) https://bugs.python.org/issue?@action=redirect&bpo=6267
(93) https://bugs.python.org/issue?@action=redirect&bpo=5511
(94) https://bugs.python.org/issue?@action=redirect&bpo=4710
(95) https://bugs.python.org/issue?@action=redirect&bpo=7610
(96) https://bugs.python.org/issue?@action=redirect&bpo=4756
(97) https://bugs.python.org/issue?@action=redirect&bpo=6003
File: python.info, Node: New module importlib, Next: New module sysconfig, Up: New and Improved Modules
1.15.11.1 New module: importlib
...............................
Python 3.1 includes the *note importlib: 77. package, a
re-implementation of the logic underlying Python’s *note import: 5de.
statement. *note importlib: 77. is useful for implementers of Python
interpreters and to users who wish to write new importers that can
participate in the import process. Python 2.7 doesn’t contain the
complete *note importlib: 77. package, but instead has a tiny subset
that contains a single function, *note import_module(): 513.
‘import_module(name, package=None)’ imports a module. 'name' is a
string containing the module or package’s name. It’s possible to do
relative imports by providing a string that begins with a ‘.’ character,
such as ‘..utils.errors’. For relative imports, the 'package' argument
must be provided and is the name of the package that will be used as the
anchor for the relative import. *note import_module(): 513. both
inserts the imported module into ‘sys.modules’ and returns the module
object.
Here are some examples:
>>> from importlib import import_module
>>> anydbm = import_module('anydbm') # Standard absolute import
>>> anydbm
>>> # Relative import
>>> file_util = import_module('..file_util', 'distutils.command')
>>> file_util
*note importlib: 77. was implemented by Brett Cannon and introduced in
Python 3.1.
File: python.info, Node: New module sysconfig, Next: ttk Themed Widgets for Tk, Prev: New module importlib, Up: New and Improved Modules
1.15.11.2 New module: sysconfig
...............................
The *note sysconfig: db. module has been pulled out of the Distutils
package, becoming a new top-level module in its own right. *note
sysconfig: db. provides functions for getting information about Python’s
build process: compiler switches, installation paths, the platform name,
and whether Python is running from its source directory.
Some of the functions in the module are:
* *note get_config_var(): 1024. returns variables from Python’s
Makefile and the ‘pyconfig.h’ file.
* *note get_config_vars(): 1025. returns a dictionary containing all
of the configuration variables.
* *note get_path(): 1321. returns the configured path for a
particular type of module: the standard library, site-specific
modules, platform-specific modules, etc.
* *note is_python_build(): 2e1. returns true if you’re running a
binary from a Python source tree, and false otherwise.
Consult the *note sysconfig: db. documentation for more details and for
a complete list of functions.
The Distutils package and *note sysconfig: db. are now maintained by
Tarek Ziadé, who has also started a Distutils2 package (source
repository at ‘https://hg.python.org/distutils2/’) for developing a
next-generation version of Distutils.
File: python.info, Node: ttk Themed Widgets for Tk, Next: Updated module unittest, Prev: New module sysconfig, Up: New and Improved Modules
1.15.11.3 ttk: Themed Widgets for Tk
....................................
Tcl/Tk 8.5 includes a set of themed widgets that re-implement basic Tk
widgets but have a more customizable appearance and can therefore more
closely resemble the native platform’s widgets. This widget set was
originally called Tile, but was renamed to Ttk (for “themed Tk”) on
being added to Tcl/Tck release 8.5.
To learn more, read the *note ttk: f9. module documentation. You may
also wish to read the Tcl/Tk manual page describing the Ttk theme
engine, available at
‘https://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.html’. Some screenshots
of the Python/Ttk code in use are at
‘https://code.google.com/archive/p/python-ttk/wikis/Screenshots.wiki’.
The *note tkinter.ttk: f9. module was written by Guilherme Polo and
added in bpo-2983(1). An alternate version called ‘Tile.py’, written by
Martin Franklin and maintained by Kevin Walzer, was proposed for
inclusion in bpo-2618(2), but the authors argued that Guilherme Polo’s
work was more comprehensive.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2983
(2) https://bugs.python.org/issue?@action=redirect&bpo=2618
File: python.info, Node: Updated module unittest, Next: Updated module ElementTree 1 3, Prev: ttk Themed Widgets for Tk, Up: New and Improved Modules
1.15.11.4 Updated module: unittest
..................................
The *note unittest: 106. module was greatly enhanced; many new features
were added. Most of these features were implemented by Michael Foord,
unless otherwise noted. The enhanced version of the module is
downloadable separately for use with Python versions 2.4 to 2.6,
packaged as the ‘unittest2’ package, from unittest2(1).
When used from the command line, the module can automatically discover
tests. It’s not as fancy as py.test(2) or nose(3), but provides a
simple way to run tests kept within a set of package directories. For
example, the following command will search the ‘test/’ subdirectory for
any importable test files named ‘test*.py’:
python -m unittest discover -s test
Consult the *note unittest: 106. module documentation for more details.
(Developed in bpo-6001(4).)
The *note main(): fde. function supports some other new options:
* *note -b: 1325. or ‘--buffer’ will buffer the standard output and
standard error streams during each test. If the test passes, any
resulting output will be discarded; on failure, the buffered output
will be displayed.
* *note -c: 1326. or ‘--catch’ will cause the control-C interrupt to
be handled more gracefully. Instead of interrupting the test
process immediately, the currently running test will be completed
and then the partial results up to the interruption will be
reported. If you’re impatient, a second press of control-C will
cause an immediate interruption.
This control-C handler tries to avoid causing problems when the
code being tested or the tests being run have defined a signal
handler of their own, by noticing that a signal handler was already
set and calling it. If this doesn’t work for you, there’s a *note
removeHandler(): 1327. decorator that can be used to mark tests
that should have the control-C handling disabled.
* *note -f: 1328. or ‘--failfast’ makes test execution stop
immediately when a test fails instead of continuing to execute
further tests. (Suggested by Cliff Dyer and implemented by Michael
Foord; bpo-8074(5).)
The progress messages now show ‘x’ for expected failures and ‘u’ for
unexpected successes when run in verbose mode. (Contributed by Benjamin
Peterson.)
Test cases can raise the *note SkipTest: fdf. exception to skip a test
(bpo-1034053(6)).
The error messages for *note assertEqual(): 524, *note assertTrue():
522, and *note assertFalse(): 523. failures now provide more
information. If you set the *note longMessage: 1329. attribute of your
*note TestCase: 449. classes to true, both the standard error message
and any additional message you provide will be printed for failures.
(Added by Michael Foord; bpo-5663(7).)
The *note assertRaises(): 528. method now returns a context handler when
called without providing a callable object to run. For example, you can
write this:
with self.assertRaises(KeyError):
{}['foo']
(Implemented by Antoine Pitrou; bpo-4444(8).)
Module- and class-level setup and teardown fixtures are now supported.
Modules can contain ‘setUpModule()’ and ‘tearDownModule()’ functions.
Classes can have *note setUpClass(): a4e. and *note tearDownClass():
132a. methods that must be defined as class methods (using
‘@classmethod’ or equivalent). These functions and methods are invoked
when the test runner switches to a test case in a different module or
class.
The methods *note addCleanup(): a81. and *note doCleanups(): 132b. were
added. *note addCleanup(): a81. lets you add cleanup functions that
will be called unconditionally (after *note setUp(): 132c. if *note
setUp(): 132c. fails, otherwise after *note tearDown(): 132d.). This
allows for much simpler resource allocation and deallocation during
tests (bpo-5679(9)).
A number of new methods were added that provide more specialized tests.
Many of these methods were written by Google engineers for use in their
test suites; Gregory P. Smith, Michael Foord, and GvR worked on merging
them into Python’s version of *note unittest: 106.
* *note assertIsNone(): 127d. and *note assertIsNotNone(): 127e. take
one expression and verify that the result is or is not ‘None’.
* *note assertIs(): 132e. and *note assertIsNot(): 132f. take two
values and check whether the two values evaluate to the same object
or not. (Added by Michael Foord; bpo-2578(10).)
* *note assertIsInstance(): 1330. and *note assertNotIsInstance():
1331. check whether the resulting object is an instance of a
particular class, or of one of a tuple of classes. (Added by Georg
Brandl; bpo-7031(11).)
* *note assertGreater(): 1332, *note assertGreaterEqual(): 1333,
*note assertLess(): 1334, and *note assertLessEqual(): 1335.
compare two quantities.
* *note assertMultiLineEqual(): 1336. compares two strings, and if
they’re not equal, displays a helpful comparison that highlights
the differences in the two strings. This comparison is now used by
default when Unicode strings are compared with *note assertEqual():
524.
* *note assertRegexpMatches(): 529. and *note
assertNotRegexpMatches(): 52b. checks whether the first argument is
a string matching or not matching the regular expression provided
as the second argument (bpo-8038(12)).
* *note assertRaisesRegexp(): 52a. checks whether a particular
exception is raised, and then also checks that the string
representation of the exception matches the provided regular
expression.
* *note assertIn(): 1337. and *note assertNotIn(): 1338. tests
whether 'first' is or is not in 'second'.
* *note assertItemsEqual(): 121d. tests whether two provided
sequences contain the same elements.
* *note assertSetEqual(): 1278. compares whether two sets are equal,
and only reports the differences between the sets in case of error.
* Similarly, *note assertListEqual(): 127a. and *note
assertTupleEqual(): 127b. compare the specified types and explain
any differences without necessarily printing their full values;
these methods are now used by default when comparing lists and
tuples using *note assertEqual(): 524. More generally, *note
assertSequenceEqual(): 127c. compares two sequences and can
optionally check whether both sequences are of a particular type.
* *note assertDictEqual(): 1279. compares two dictionaries and
reports the differences; it’s now used by default when you compare
two dictionaries using *note assertEqual(): 524.
‘assertDictContainsSubset()’ checks whether all of the key/value
pairs in 'first' are found in 'second'.
* *note assertAlmostEqual(): 526. and *note assertNotAlmostEqual():
527. test whether 'first' and 'second' are approximately equal.
This method can either round their difference to an optionally
specified number of 'places' (the default is 7) and compare it to
zero, or require the difference to be smaller than a supplied
'delta' value.
* *note loadTestsFromName(): 1339. properly honors the *note
suiteClass: 133a. attribute of the *note TestLoader: 290. (Fixed
by Mark Roddy; bpo-6866(13).)
* A new hook lets you extend the *note assertEqual(): 524. method to
handle new data types. The *note addTypeEqualityFunc(): 133b.
method takes a type object and a function. The function will be
used when both of the objects being compared are of the specified
type. This function should compare the two objects and raise an
exception if they don’t match; it’s a good idea for the function to
provide additional information about why the two objects aren’t
matching, much as the new sequence comparison methods do.
*note unittest.main(): fde. now takes an optional ‘exit’ argument. If
false, *note main(): fde. doesn’t call *note sys.exit(): 133c, allowing
*note main(): fde. to be used from the interactive interpreter.
(Contributed by J. Pablo Fernández; bpo-3379(14).)
*note TestResult: 115b. has new *note startTestRun(): 133d. and *note
stopTestRun(): 133e. methods that are called immediately before and
after a test run. (Contributed by Robert Collins; bpo-5728(15).)
With all these changes, the ‘unittest.py’ was becoming awkwardly large,
so the module was turned into a package and the code split into several
files (by Benjamin Peterson). This doesn’t affect how the module is
imported or used.
See also
........
‘https://web.archive.org/web/20210619163128/http://www.voidspace.org.uk/python/articles/unittest2.shtml’
Describes the new features, how to use them, and the rationale for
various design decisions. (By Michael Foord.)
---------- Footnotes ----------
(1) https://pypi.org/project/unittest2/
(2) https://pytest.org
(3) https://nose.readthedocs.io/
(4) https://bugs.python.org/issue?@action=redirect&bpo=6001
(5) https://bugs.python.org/issue?@action=redirect&bpo=8074
(6) https://bugs.python.org/issue?@action=redirect&bpo=1034053
(7) https://bugs.python.org/issue?@action=redirect&bpo=5663
(8) https://bugs.python.org/issue?@action=redirect&bpo=4444
(9) https://bugs.python.org/issue?@action=redirect&bpo=5679
(10) https://bugs.python.org/issue?@action=redirect&bpo=2578
(11) https://bugs.python.org/issue?@action=redirect&bpo=7031
(12) https://bugs.python.org/issue?@action=redirect&bpo=8038
(13) https://bugs.python.org/issue?@action=redirect&bpo=6866
(14) https://bugs.python.org/issue?@action=redirect&bpo=3379
(15) https://bugs.python.org/issue?@action=redirect&bpo=5728
File: python.info, Node: Updated module ElementTree 1 3, Prev: Updated module unittest, Up: New and Improved Modules
1.15.11.5 Updated module: ElementTree 1.3
.........................................
The version of the ElementTree library included with Python was updated
to version 1.3. Some of the new features are:
* The various parsing functions now take a 'parser' keyword argument
giving an *note XMLParser: a53. instance that will be used. This
makes it possible to override the file’s internal encoding:
p = ET.XMLParser(encoding='utf-8')
t = ET.XML(""" """, parser=p)
Errors in parsing XML now raise a *note ParseError: 126b.
exception, whose instances have a ‘position’ attribute containing a
('line', 'column') tuple giving the location of the problem.
* ElementTree’s code for converting trees to a string has been
significantly reworked, making it roughly twice as fast in many
cases. The *note ElementTree.write(): ff0. and ‘Element.write()’
methods now have a 'method' parameter that can be “xml” (the
default), “html”, or “text”. HTML mode will output empty elements
as ‘ ’ instead of ‘ ’, and text mode will skip
over elements and only output the text chunks. If you set the
*note tag: 1341. attribute of an element to ‘None’ but leave its
children in place, the element will be omitted when the tree is
written out, so you don’t need to do more extensive rearrangement
to remove a single element.
Namespace handling has also been improved. All ‘xmlns:’
declarations are now output on the root element, not scattered
throughout the resulting XML. You can set the default namespace for
a tree by setting the ‘default_namespace’ attribute and can
register new prefixes with *note register_namespace(): 11c4. In
XML mode, you can use the true/false 'xml_declaration' parameter to
suppress the XML declaration.
* New *note Element: 30a. method: *note extend(): 11c5. appends the
items from a sequence to the element’s children. Elements
themselves behave like sequences, so it’s easy to move children
from one element to another:
from xml.etree import ElementTree as ET
t = ET.XML("""
- 1
- 2
- 3
""")
new = ET.XML(' ')
new.extend(t)
# Outputs - 1
...
print ET.tostring(new)
* New *note Element: 30a. method: *note iter(): 1342. yields the
children of the element as a generator. It’s also possible to
write ‘for child in elem:’ to loop over an element’s children. The
existing method ‘getiterator()’ is now deprecated, as is
‘getchildren()’ which constructs and returns a list of children.
* New *note Element: 30a. method: *note itertext(): 11c7. yields all
chunks of text that are descendants of the element. For example:
t = ET.XML("""
- 1
- 2
- 3
""")
# Outputs ['\n ', '1', ' ', '2', ' ', '3', '\n']
print list(t.itertext())
* Deprecated: using an element as a Boolean (i.e., ‘if elem:’) would
return true if the element had any children, or false if there were
no children. This behaviour is confusing – ‘None’ is false, but so
is a childless element? – so it will now trigger a *note
FutureWarning: 411. In your code, you should be explicit: write
‘len(elem) != 0’ if you’re interested in the number of children, or
‘elem is not None’.
Fredrik Lundh develops ElementTree and produced the 1.3 version; you can
read his article describing 1.3 at
‘https://web.archive.org/web/20200703234532/http://effbot.org/zone/elementtree-13-intro.htm’.
Florent Xicluna updated the version included with Python, after
discussions on python-dev and in bpo-6472(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=6472
File: python.info, Node: Build and C API Changes<8>, Next: Other Changes and Fixes, Prev: New and Improved Modules, Up: What’s New in Python 2 7
1.15.12 Build and C API Changes
-------------------------------
Changes to Python’s build process and to the C API include:
* The latest release of the GNU Debugger, GDB 7, can be scripted
using Python(1). When you begin debugging an executable program P,
GDB will look for a file named ‘P-gdb.py’ and automatically read
it. Dave Malcolm contributed a ‘python-gdb.py’ that adds a number
of commands useful when debugging Python itself. For example,
‘py-up’ and ‘py-down’ go up or down one Python stack frame, which
usually corresponds to several C stack frames. ‘py-print’ prints
the value of a Python variable, and ‘py-bt’ prints the Python stack
trace. (Added as a result of bpo-8032(2).)
* If you use the ‘.gdbinit’ file provided with Python, the “pyo”
macro in the 2.7 version now works correctly when the thread being
debugged doesn’t hold the GIL; the macro now acquires it before
printing. (Contributed by Victor Stinner; bpo-3632(3).)
* *note Py_AddPendingCall(): 981. is now thread-safe, letting any
worker thread submit notifications to the main Python thread. This
is particularly useful for asynchronous IO operations.
(Contributed by Kristján Valur Jónsson; bpo-4293(4).)
* New function: *note PyCode_NewEmpty(): 1344. creates an empty code
object; only the filename, function name, and first line number are
required. This is useful for extension modules that are attempting
to construct a more useful traceback stack. Previously such
extensions needed to call ‘PyCode_New()’, which had many more
arguments. (Added by Jeffrey Yasskin.)
* New function: *note PyErr_NewExceptionWithDoc(): 125c. creates a
new exception class, just as the existing *note
PyErr_NewException(): 125d. does, but takes an extra ‘char *’
argument containing the docstring for the new exception class.
(Added by ‘lekma’ on the Python bug tracker; bpo-7033(5).)
* New function: *note PyFrame_GetLineNumber(): 786. takes a frame
object and returns the line number that the frame is currently
executing. Previously code would need to get the index of the
bytecode instruction currently executing, and then look up the line
number corresponding to that address. (Added by Jeffrey Yasskin.)
* New functions: *note PyLong_AsLongAndOverflow(): 125a. and *note
PyLong_AsLongLongAndOverflow(): 1259. approximates a Python long
integer as a C long or long long. If the number is too large to
fit into the output type, an 'overflow' flag is set and returned to
the caller. (Contributed by Case Van Horsen; bpo-7528(6) and
bpo-7767(7).)
* New function: stemming from the rewrite of string-to-float
conversion, a new *note PyOS_string_to_double(): 1288. function was
added. The old ‘PyOS_ascii_strtod()’ and ‘PyOS_ascii_atof()’
functions are now deprecated.
* New function: ‘PySys_SetArgvEx()’ sets the value of ‘sys.argv’ and
can optionally update ‘sys.path’ to include the directory
containing the script named by ‘sys.argv[0]’ depending on the value
of an 'updatepath' parameter.
This function was added to close a security hole for applications
that embed Python. The old function, ‘PySys_SetArgv()’, would
always update ‘sys.path’, and sometimes it would add the current
directory. This meant that, if you ran an application embedding
Python in a directory controlled by someone else, attackers could
put a Trojan-horse module in the directory (say, a file named
‘os.py’) that your application would then import and run.
If you maintain a C/C++ application that embeds Python, check
whether you’re calling ‘PySys_SetArgv()’ and carefully consider
whether the application should be using ‘PySys_SetArgvEx()’ with
'updatepath' set to false.
Security issue reported as CVE 2008-5983(8); discussed in
bpo-5753(9), and fixed by Antoine Pitrou.
* New macros: the Python header files now define the following
macros: ‘Py_ISALNUM’, ‘Py_ISALPHA’, ‘Py_ISDIGIT’, ‘Py_ISLOWER’,
‘Py_ISSPACE’, ‘Py_ISUPPER’, ‘Py_ISXDIGIT’, ‘Py_TOLOWER’, and
‘Py_TOUPPER’. All of these functions are analogous to the C
standard macros for classifying characters, but ignore the current
locale setting, because in several places Python needs to analyze
characters in a locale-independent way. (Added by Eric Smith;
bpo-5793(10).)
* Removed function: ‘PyEval_CallObject()’ is now only available as a
macro. A function version was being kept around to preserve ABI
linking compatibility, but that was in 1997; it can certainly be
deleted by now. (Removed by Antoine Pitrou; bpo-8276(11).)
* New format codes: the ‘PyString_FromFormat()’,
‘PyString_FromFormatV()’, and *note PyErr_Format(): eaa. functions
now accept ‘%lld’ and ‘%llu’ format codes for displaying C’s long
long types. (Contributed by Mark Dickinson; bpo-7228(12).)
* The complicated interaction between threads and process forking has
been changed. Previously, the child process created by *note
os.fork(): 197. might fail because the child is created with only a
single thread running, the thread performing the *note os.fork():
197. If other threads were holding a lock, such as Python’s import
lock, when the fork was performed, the lock would still be marked
as “held” in the new process. But in the child process nothing
would ever release the lock, since the other threads weren’t
replicated, and the child process would no longer be able to
perform imports.
Python 2.7 acquires the import lock before performing an *note
os.fork(): 197, and will also clean up any locks created using the
*note threading: ed. module. C extension modules that have
internal locks, or that call ‘fork()’ themselves, will not benefit
from this clean-up.
(Fixed by Thomas Wouters; bpo-1590864(13).)
* The *note Py_Finalize(): 1345. function now calls the internal
‘threading._shutdown()’ function; this prevents some exceptions
from being raised when an interpreter shuts down. (Patch by Adam
Olsen; bpo-1722344(14).)
* When using the *note PyMemberDef: 54c. structure to define
attributes of a type, Python will no longer let you try to delete
or set a ‘T_STRING_INPLACE’ attribute.
* Global symbols defined by the *note ctypes: 2a. module are now
prefixed with ‘Py’, or with ‘_ctypes’. (Implemented by Thomas
Heller; bpo-3102(15).)
* New configure option: the ‘--with-system-expat’ switch allows
building the *note pyexpat: 126. module to use the system Expat
library. (Contributed by Arfrever Frehtes Taifersar Arahesis;
bpo-7609(16).)
* New configure option: the ‘--with-valgrind’ option will now disable
the pymalloc allocator, which is difficult for the Valgrind
memory-error detector to analyze correctly. Valgrind will
therefore be better at detecting memory leaks and overruns.
(Contributed by James Henstridge; bpo-2422(17).)
* New configure option: you can now supply an empty string to
‘--with-dbmliborder=’ in order to disable all of the various DBM
modules. (Added by Arfrever Frehtes Taifersar Arahesis;
bpo-6491(18).)
* The ‘configure’ script now checks for floating-point rounding bugs
on certain 32-bit Intel chips and defines a ‘X87_DOUBLE_ROUNDING’
preprocessor definition. No code currently uses this definition,
but it’s available if anyone wishes to use it. (Added by Mark
Dickinson; bpo-2937(19).)
‘configure’ also now sets a ‘LDCXXSHARED’ Makefile variable for
supporting C++ linking. (Contributed by Arfrever Frehtes Taifersar
Arahesis; bpo-1222585(20).)
* The build process now creates the necessary files for pkg-config
support. (Contributed by Clinton Roy; bpo-3585(21).)
* The build process now supports Subversion 1.7. (Contributed by
Arfrever Frehtes Taifersar Arahesis; bpo-6094(22).)
* Menu:
* Capsules::
* Port-Specific Changes; Windows: Port-Specific Changes Windows.
* Port-Specific Changes; Mac OS X: Port-Specific Changes Mac OS X.
* Port-Specific Changes; FreeBSD: Port-Specific Changes FreeBSD.
---------- Footnotes ----------
(1)
https://web.archive.org/web/20110715084810/http://sourceware.org/gdb/current/onlinedocs/gdb/Python.html
(2) https://bugs.python.org/issue?@action=redirect&bpo=8032
(3) https://bugs.python.org/issue?@action=redirect&bpo=3632
(4) https://bugs.python.org/issue?@action=redirect&bpo=4293
(5) https://bugs.python.org/issue?@action=redirect&bpo=7033
(6) https://bugs.python.org/issue?@action=redirect&bpo=7528
(7) https://bugs.python.org/issue?@action=redirect&bpo=7767
(8) https://www.cve.org/CVERecord?id=CVE-2008-5983
(9) https://bugs.python.org/issue?@action=redirect&bpo=5753
(10) https://bugs.python.org/issue?@action=redirect&bpo=5793
(11) https://bugs.python.org/issue?@action=redirect&bpo=8276
(12) https://bugs.python.org/issue?@action=redirect&bpo=7228
(13) https://bugs.python.org/issue?@action=redirect&bpo=1590864
(14) https://bugs.python.org/issue?@action=redirect&bpo=1722344
(15) https://bugs.python.org/issue?@action=redirect&bpo=3102
(16) https://bugs.python.org/issue?@action=redirect&bpo=7609
(17) https://bugs.python.org/issue?@action=redirect&bpo=2422
(18) https://bugs.python.org/issue?@action=redirect&bpo=6491
(19) https://bugs.python.org/issue?@action=redirect&bpo=2937
(20) https://bugs.python.org/issue?@action=redirect&bpo=1222585
(21) https://bugs.python.org/issue?@action=redirect&bpo=3585
(22) https://bugs.python.org/issue?@action=redirect&bpo=6094
File: python.info, Node: Capsules, Next: Port-Specific Changes Windows, Up: Build and C API Changes<8>
1.15.12.1 Capsules
..................
Python 3.1 adds a new C datatype, *note PyCapsule: 1266, for providing a
C API to an extension module. A capsule is essentially the holder of a
C ‘void *’ pointer, and is made available as a module attribute; for
example, the *note socket: cc. module’s API is exposed as ‘socket.CAPI’,
and *note unicodedata: 105. exposes ‘ucnhash_CAPI’. Other extensions
can import the module, access its dictionary to get the capsule object,
and then get the ‘void *’ pointer, which will usually point to an array
of pointers to the module’s various API functions.
There is an existing data type already used for this, ‘PyCObject’, but
it doesn’t provide type safety. Evil code written in pure Python could
cause a segmentation fault by taking a ‘PyCObject’ from module A and
somehow substituting it for the ‘PyCObject’ in module B. Capsules know
their own name, and getting the pointer requires providing the name:
void *vtable;
if (!PyCapsule_IsValid(capsule, "mymodule.CAPI") {
PyErr_SetString(PyExc_ValueError, "argument type invalid");
return NULL;
}
vtable = PyCapsule_GetPointer(capsule, "mymodule.CAPI");
You are assured that ‘vtable’ points to whatever you’re expecting. If a
different capsule was passed in, *note PyCapsule_IsValid(): 1348. would
detect the mismatched name and return false. Refer to *note Providing a
C API for an Extension Module: 1349. for more information on using these
objects.
Python 2.7 now uses capsules internally to provide various
extension-module APIs, but the ‘PyCObject_AsVoidPtr()’ was modified to
handle capsules, preserving compile-time compatibility with the
‘PyCObject’ interface. Use of ‘PyCObject_AsVoidPtr()’ will signal a
*note PendingDeprecationWarning: 8c7, which is silent by default.
Implemented in Python 3.1 and backported to 2.7 by Larry Hastings;
discussed in bpo-5630(1).
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5630
File: python.info, Node: Port-Specific Changes Windows, Next: Port-Specific Changes Mac OS X, Prev: Capsules, Up: Build and C API Changes<8>
1.15.12.2 Port-Specific Changes: Windows
........................................
* The *note msvcrt: 93. module now contains some constants from the
‘crtassem.h’ header file: *note CRT_ASSEMBLY_VERSION: 134b, *note
VC_ASSEMBLY_PUBLICKEYTOKEN: 134c, and *note
LIBRARIES_ASSEMBLY_NAME_PREFIX: 134d. (Contributed by David
Cournapeau; bpo-4365(1).)
* The *note _winreg: 116. module for accessing the registry now
implements the *note CreateKeyEx(): 134e. and *note DeleteKeyEx():
134f. functions, extended versions of previously supported
functions that take several extra arguments. The *note
DisableReflectionKey(): 1350, *note EnableReflectionKey(): 1351,
and *note QueryReflectionKey(): 1352. were also tested and
documented. (Implemented by Brian Curtin: bpo-7347(2).)
* The new ‘_beginthreadex()’ API is used to start threads, and the
native thread-local storage functions are now used. (Contributed
by Kristján Valur Jónsson; bpo-3582(3).)
* The *note os.kill(): 1353. function now works on Windows. The
signal value can be the constants *note CTRL_C_EVENT: 1354, *note
CTRL_BREAK_EVENT: 1355, or any integer. The first two constants
will send ‘Control’-‘C’ and ‘Control’-‘Break’ keystroke events to
subprocesses; any other value will use the ‘TerminateProcess()’
API. (Contributed by Miki Tebeka; bpo-1220212(4).)
* The *note os.listdir(): 10ee. function now correctly fails for an
empty path. (Fixed by Hirokazu Yamamoto; bpo-5913(5).)
* The *note mimetypes: 8f. module will now read the MIME database
from the Windows registry when initializing. (Patch by Gabriel
Genellina; bpo-4969(6).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4365
(2) https://bugs.python.org/issue?@action=redirect&bpo=7347
(3) https://bugs.python.org/issue?@action=redirect&bpo=3582
(4) https://bugs.python.org/issue?@action=redirect&bpo=1220212
(5) https://bugs.python.org/issue?@action=redirect&bpo=5913
(6) https://bugs.python.org/issue?@action=redirect&bpo=4969
File: python.info, Node: Port-Specific Changes Mac OS X, Next: Port-Specific Changes FreeBSD, Prev: Port-Specific Changes Windows, Up: Build and C API Changes<8>
1.15.12.3 Port-Specific Changes: Mac OS X
.........................................
* The path ‘/Library/Python/2.7/site-packages’ is now appended to
‘sys.path’, in order to share added packages between the system
installation and a user-installed copy of the same version.
(Changed by Ronald Oussoren; bpo-4865(1).)
Changed in version 2.7.13: As of 2.7.13, this change was removed.
‘/Library/Python/2.7/site-packages’, the site-packages directory
used by the Apple-supplied system Python 2.7 is no longer appended
to ‘sys.path’ for user-installed Pythons such as from the
python.org installers. As of macOS 10.12, Apple changed how the
system site-packages directory is configured, which could cause
installation of pip components, like setuptools, to fail. Packages
installed for the system Python will no longer be shared with
user-installed Pythons. (bpo-28440(2))
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=4865
(2) https://bugs.python.org/issue?@action=redirect&bpo=28440
File: python.info, Node: Port-Specific Changes FreeBSD, Prev: Port-Specific Changes Mac OS X, Up: Build and C API Changes<8>
1.15.12.4 Port-Specific Changes: FreeBSD
........................................
* FreeBSD 7.1’s ‘SO_SETFIB’ constant, used with the *note socket():
da0. methods *note getsockopt(): cd5./*note setsockopt(): cd6. to
select an alternate routing table, is now available in the *note
socket: cc. module. (Added by Kyle VanderBeek; bpo-8235(1).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=8235
File: python.info, Node: Other Changes and Fixes, Next: Porting to Python 2 7, Prev: Build and C API Changes<8>, Up: What’s New in Python 2 7
1.15.13 Other Changes and Fixes
-------------------------------
* Two benchmark scripts, ‘iobench’ and ‘ccbench’, were added to the
‘Tools’ directory. ‘iobench’ measures the speed of the built-in
file I/O objects returned by *note open(): 517. while performing
various operations, and ‘ccbench’ is a concurrency benchmark that
tries to measure computing throughput, thread switching latency,
and IO processing bandwidth when performing several tasks using a
varying number of threads.
* The ‘Tools/i18n/msgfmt.py’ script now understands plural forms in
‘.po’ files. (Fixed by Martin von Löwis; bpo-5464(1).)
* When importing a module from a ‘.pyc’ or ‘.pyo’ file with an
existing ‘.py’ counterpart, the *note co_filename: 1359. attributes
of the resulting code objects are overwritten when the original
filename is obsolete. This can happen if the file has been
renamed, moved, or is accessed through different paths. (Patch by
Ziga Seilnacht and Jean-Paul Calderone; bpo-1180193(2).)
* The ‘regrtest.py’ script now takes a ‘--randseed=’ switch that
takes an integer that will be used as the random seed for the ‘-r’
option that executes tests in random order. The ‘-r’ option also
reports the seed that was used (Added by Collin Winter.)
* Another ‘regrtest.py’ switch is ‘-j’, which takes an integer
specifying how many tests run in parallel. This allows reducing
the total runtime on multi-core machines. This option is
compatible with several other options, including the ‘-R’ switch
which is known to produce long runtimes. (Added by Antoine Pitrou,
bpo-6152(3).) This can also be used with a new ‘-F’ switch that
runs selected tests in a loop until they fail. (Added by Antoine
Pitrou; bpo-7312(4).)
* When executed as a script, the ‘py_compile.py’ module now accepts
‘'-'’ as an argument, which will read standard input for the list
of filenames to be compiled. (Contributed by Piotr Ożarowski;
bpo-8233(5).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=5464
(2) https://bugs.python.org/issue?@action=redirect&bpo=1180193
(3) https://bugs.python.org/issue?@action=redirect&bpo=6152
(4) https://bugs.python.org/issue?@action=redirect&bpo=7312
(5) https://bugs.python.org/issue?@action=redirect&bpo=8233
File: python.info, Node: Porting to Python 2 7, Next: New Features Added to Python 2 7 Maintenance Releases, Prev: Other Changes and Fixes, Up: What’s New in Python 2 7
1.15.14 Porting to Python 2.7
-----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code:
* The *note range(): 943. function processes its arguments more
consistently; it will now call *note __int__(): 717. on non-float,
non-integer arguments that are supplied to it. (Fixed by Alexander
Belopolsky; bpo-1533(1).)
* The string *note format(): 61b. method changed the default
precision used for floating-point and complex numbers from 6
decimal places to 12, which matches the precision used by *note
str(): 447. (Changed by Eric Smith; bpo-5920(2).)
* Because of an optimization for the *note with: 5ce. statement, the
special methods *note __enter__(): 5c4. and *note __exit__(): 12f3.
must belong to the object’s type, and cannot be directly attached
to the object’s instance. This affects new-style classes (derived
from *note object: a8c.) and C extension types. (bpo-6101(3).)
* Due to a bug in Python 2.6, the 'exc_value' parameter to *note
__exit__(): 12f3. methods was often the string representation of
the exception, not an instance. This was fixed in 2.7, so
'exc_value' will be an instance as expected. (Fixed by Florent
Xicluna; bpo-7853(4).)
* When a restricted set of attributes were set using ‘__slots__’,
deleting an unset attribute would not raise *note AttributeError:
348. as you would expect. Fixed by Benjamin Peterson;
bpo-7604(5).)
In the standard library:
* Operations with *note datetime: 1cc. instances that resulted in a
year falling outside the supported range didn’t always raise *note
OverflowError: 87f. Such errors are now checked more carefully and
will now raise the exception. (Reported by Mark Leander, patch by
Anand B. Pillai and Alexander Belopolsky; bpo-7150(6).)
* When using *note Decimal: 29f. instances with a string’s *note
format(): 61b. method, the default alignment was previously
left-alignment. This has been changed to right-alignment, which
might change the output of your programs. (Changed by Mark
Dickinson; bpo-6857(7).)
Comparisons involving a signaling NaN value (or ‘sNAN’) now signal
*note InvalidOperation: 109e. instead of silently returning a true
or false value depending on the comparison operator. Quiet NaN
values (or ‘NaN’) are now hashable. (Fixed by Mark Dickinson;
bpo-7279(8).)
* The *note xml.etree.ElementTree: 125. library no longer escapes
ampersands and angle brackets when outputting an XML processing
instruction (which looks like ‘’)
or comment (which looks like ‘’). (Patch by Neil
Muller; bpo-2746(9).)
* The ‘readline()’ method of *note StringIO: f22. objects now does
nothing when a negative length is requested, as other file-like
objects do. (bpo-7348(10)).
* The *note syslog: dc. module will now use the value of
‘sys.argv[0]’ as the identifier instead of the previous default
value of ‘'python'’. (Changed by Sean Reifschneider;
bpo-8451(11).)
* The *note tarfile: de. module’s default error handling has changed,
to no longer suppress fatal errors. The default error level was
previously 0, which meant that errors would only result in a
message being written to the debug log, but because the debug log
is not activated by default, these errors go unnoticed. The
default error level is now 1, which raises an exception if there’s
an error. (Changed by Lars Gustäbel; bpo-7357(12).)
* The *note urlparse: 10a. module’s *note urlsplit(): d3f. now
handles unknown URL schemes in a fashion compliant with RFC
3986(13): if the URL is of the form ‘"://..."’, the text
before the ‘://’ is treated as the scheme, even if it’s a made-up
scheme that the module doesn’t know about. This change may break
code that worked around the old behaviour. For example, Python
2.6.4 or 2.5 will return the following:
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', '', '//host/filename?query', '', '')
Python 2.7 (and Python 2.6.5) will return:
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', 'host', '/filename?query', '', '')
(Python 2.7 actually produces slightly different output, since it
returns a named tuple instead of a standard tuple.)
For C extensions:
* C extensions that use integer format codes with the ‘PyArg_Parse*’
family of functions will now raise a *note TypeError: 534.
exception instead of triggering a *note DeprecationWarning: 1a5.
(bpo-5080(14)).
* Use the new *note PyOS_string_to_double(): 1288. function instead
of the old ‘PyOS_ascii_strtod()’ and ‘PyOS_ascii_atof()’ functions,
which are now deprecated.
For applications that embed Python:
* The ‘PySys_SetArgvEx()’ function was added, letting applications
close a security hole when the existing ‘PySys_SetArgv()’ function
was used. Check whether you’re calling ‘PySys_SetArgv()’ and
carefully consider whether the application should be using
‘PySys_SetArgvEx()’ with 'updatepath' set to false.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1533
(2) https://bugs.python.org/issue?@action=redirect&bpo=5920
(3) https://bugs.python.org/issue?@action=redirect&bpo=6101
(4) https://bugs.python.org/issue?@action=redirect&bpo=7853
(5) https://bugs.python.org/issue?@action=redirect&bpo=7604
(6) https://bugs.python.org/issue?@action=redirect&bpo=7150
(7) https://bugs.python.org/issue?@action=redirect&bpo=6857
(8) https://bugs.python.org/issue?@action=redirect&bpo=7279
(9) https://bugs.python.org/issue?@action=redirect&bpo=2746
(10) https://bugs.python.org/issue?@action=redirect&bpo=7348
(11) https://bugs.python.org/issue?@action=redirect&bpo=8451
(12) https://bugs.python.org/issue?@action=redirect&bpo=7357
(13) https://datatracker.ietf.org/doc/html/rfc3986.html
(14) https://bugs.python.org/issue?@action=redirect&bpo=5080
File: python.info, Node: New Features Added to Python 2 7 Maintenance Releases, Next: Acknowledgements, Prev: Porting to Python 2 7, Up: What’s New in Python 2 7
1.15.15 New Features Added to Python 2.7 Maintenance Releases
-------------------------------------------------------------
New features may be added to Python 2.7 maintenance releases when the
situation genuinely calls for it. Any such additions must go through
the Python Enhancement Proposal process, and make a compelling case for
why they can’t be adequately addressed by either adding the new feature
solely to Python 3, or else by publishing it on the Python Package
Index.
In addition to the specific proposals listed below, there is a general
exemption allowing new ‘-3’ warnings to be added in any Python 2.7
maintenance release.
* Menu:
* Two new environment variables for debug mode::
* PEP 434; IDLE Enhancement Exception for All Branches: PEP 434 IDLE Enhancement Exception for All Branches.
* PEP 466; Network Security Enhancements for Python 2.7: PEP 466 Network Security Enhancements for Python 2 7.
* PEP 477; Backport ensurepip (PEP 453) to Python 2.7: PEP 477 Backport ensurepip PEP 453 to Python 2 7.
* PEP 476; Enabling certificate verification by default for stdlib http clients: PEP 476 Enabling certificate verification by default for stdlib http clients<2>.
* PEP 493; HTTPS verification migration tools for Python 2.7: PEP 493 HTTPS verification migration tools for Python 2 7.
* New make regen-all build target: New make regen-all build target<3>.
* Removal of make touch build target: Removal of make touch build target<3>.
File: python.info, Node: Two new environment variables for debug mode, Next: PEP 434 IDLE Enhancement Exception for All Branches, Up: New Features Added to Python 2 7 Maintenance Releases
1.15.15.1 Two new environment variables for debug mode
......................................................
In debug mode, the ‘[xxx refs]’ statistic is not written by default, the
‘PYTHONSHOWREFCOUNT’ environment variable now must also be set.
(Contributed by Victor Stinner; bpo-31733(1).)
When Python is compiled with ‘COUNT_ALLOC’ defined, allocation counts
are no longer dumped by default anymore: the ‘PYTHONSHOWALLOCCOUNT’
environment variable must now also be set. Moreover, allocation counts
are now dumped into stderr, rather than stdout. (Contributed by Victor
Stinner; bpo-31692(2).)
Added in version 2.7.15.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=31733
(2) https://bugs.python.org/issue?@action=redirect&bpo=31692
File: python.info, Node: PEP 434 IDLE Enhancement Exception for All Branches, Next: PEP 466 Network Security Enhancements for Python 2 7, Prev: Two new environment variables for debug mode, Up: New Features Added to Python 2 7 Maintenance Releases
1.15.15.2 PEP 434: IDLE Enhancement Exception for All Branches
..............................................................
PEP 434(1) describes a general exemption for changes made to the IDLE
development environment shipped along with Python. This exemption makes
it possible for the IDLE developers to provide a more consistent user
experience across all supported versions of Python 2 and 3.
For details of any IDLE changes, refer to the NEWS file for the specific
release.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0434/
File: python.info, Node: PEP 466 Network Security Enhancements for Python 2 7, Next: PEP 477 Backport ensurepip PEP 453 to Python 2 7, Prev: PEP 434 IDLE Enhancement Exception for All Branches, Up: New Features Added to Python 2 7 Maintenance Releases
1.15.15.3 PEP 466: Network Security Enhancements for Python 2.7
...............................................................
PEP 466(1) describes a number of network security enhancement proposals
that have been approved for inclusion in Python 2.7 maintenance
releases, with the first of those changes appearing in the Python 2.7.7
release.
PEP 466(2) related features added in Python 2.7.7:
* *note hmac.compare_digest(): 10bf. was backported from Python 3 to
make a timing attack resistant comparison operation available to
Python 2 applications. (Contributed by Alex Gaynor; bpo-21306(3).)
* OpenSSL 1.0.1g was upgraded in the official Windows installers
published on python.org. (Contributed by Zachary Ware;
bpo-21462(4).)
PEP 466(5) related features added in Python 2.7.8:
* *note hashlib.pbkdf2_hmac(): 50a. was backported from Python 3 to
make a hashing algorithm suitable for secure password storage
broadly available to Python 2 applications. (Contributed by Alex
Gaynor; bpo-21304(6).)
* OpenSSL 1.0.1h was upgraded for the official Windows installers
published on python.org. (Contributed by Zachary Ware in
bpo-21671(7) for CVE 2014-0224(8).)
PEP 466(9) related features added in Python 2.7.9:
* Most of Python 3.4’s *note ssl: d0. module was backported. This
means *note ssl: d0. now supports Server Name Indication, TLS1.x
settings, access to the platform certificate store, the *note
SSLContext: 296. class, and other features. (Contributed by Alex
Gaynor and David Reid; bpo-21308(10).)
Refer to the “Version added: 2.7.9” notes in the module
documentation for specific details.
* *note os.urandom(): 51e. was changed to cache a file descriptor to
‘/dev/urandom’ instead of reopening ‘/dev/urandom’ on every call.
(Contributed by Alex Gaynor; bpo-21305(11).)
* *note hashlib.algorithms_guaranteed: 135f. and *note
hashlib.algorithms_available: 1360. were backported from Python 3
to make it easier for Python 2 applications to select the strongest
available hash algorithm. (Contributed by Alex Gaynor in
bpo-21307(12))
---------- Footnotes ----------
(1) https://peps.python.org/pep-0466/
(2) https://peps.python.org/pep-0466/
(3) https://bugs.python.org/issue?@action=redirect&bpo=21306
(4) https://bugs.python.org/issue?@action=redirect&bpo=21462
(5) https://peps.python.org/pep-0466/
(6) https://bugs.python.org/issue?@action=redirect&bpo=21304
(7) https://bugs.python.org/issue?@action=redirect&bpo=21671
(8) https://www.cve.org/CVERecord?id=CVE-2014-0224
(9) https://peps.python.org/pep-0466/
(10) https://bugs.python.org/issue?@action=redirect&bpo=21308
(11) https://bugs.python.org/issue?@action=redirect&bpo=21305
(12) https://bugs.python.org/issue?@action=redirect&bpo=21307
File: python.info, Node: PEP 477 Backport ensurepip PEP 453 to Python 2 7, Next: PEP 476 Enabling certificate verification by default for stdlib http clients<2>, Prev: PEP 466 Network Security Enhancements for Python 2 7, Up: New Features Added to Python 2 7 Maintenance Releases
1.15.15.4 PEP 477: Backport ensurepip (PEP 453) to Python 2.7
.............................................................
PEP 477(1) approves the inclusion of the PEP 453(2) ensurepip module and
the improved documentation that was enabled by it in the Python 2.7
maintenance releases, appearing first in the Python 2.7.9 release.
* Menu:
* Bootstrapping pip By Default: Bootstrapping pip By Default<2>.
* Documentation Changes: Documentation Changes<2>.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0477/
(2) https://peps.python.org/pep-0453/
File: python.info, Node: Bootstrapping pip By Default<2>, Next: Documentation Changes<2>, Up: PEP 477 Backport ensurepip PEP 453 to Python 2 7
1.15.15.5 Bootstrapping pip By Default
......................................
The new *note ensurepip: 55. module (defined in PEP 453(1)) provides a
standard cross-platform mechanism to bootstrap the pip installer into
Python installations. The version of ‘pip’ included with Python 2.7.9
is ‘pip’ 1.5.6, and future 2.7.x maintenance releases will update the
bundled version to the latest version of ‘pip’ that is available at the
time of creating the release candidate.
By default, the commands ‘pip’, ‘pipX’ and ‘pipX.Y’ will be installed on
all platforms (where X.Y stands for the version of the Python
installation), along with the ‘pip’ Python package and its dependencies.
For CPython *note source builds on POSIX systems: eee, the ‘make
install’ and ‘make altinstall’ commands do not bootstrap ‘pip’ by
default. This behaviour can be controlled through configure options,
and overridden through Makefile options.
On Windows and Mac OS X, the CPython installers now default to
installing ‘pip’ along with CPython itself (users may opt out of
installing it during the installation process). Window users will need
to opt in to the automatic ‘PATH’ modifications to have ‘pip’ available
from the command line by default, otherwise it can still be accessed
through the Python launcher for Windows as ‘py -m pip’.
As discussed in the PEP(2), platform packagers may choose not to install
these commands by default, as long as, when invoked, they provide clear
and simple directions on how to install them on that platform (usually
using the system package manager).
---------- Footnotes ----------
(1) https://peps.python.org/pep-0453/
(2)
https://peps.python.org/pep-0477/#disabling-ensurepip-by-downstream-distributors
File: python.info, Node: Documentation Changes<2>, Prev: Bootstrapping pip By Default<2>, Up: PEP 477 Backport ensurepip PEP 453 to Python 2 7
1.15.15.6 Documentation Changes
...............................
As part of this change, the *note Installing Python Modules: ef0. and
distributing-index sections of the documentation have been completely
redesigned as short getting started and FAQ documents. Most packaging
documentation has now been moved out to the Python Packaging Authority
maintained Python Packaging User Guide(1) and the documentation of the
individual projects.
However, as this migration is currently still incomplete, the legacy
versions of those guides remaining available as *note Building C and C++
Extensions with setuptools: ef1. and *note Building C and C++ Extensions
with setuptools: ef2.
See also
........
PEP 453(2) – Explicit bootstrapping of pip in Python installations
PEP written by Donald Stufft and Nick Coghlan, implemented by
Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily.
---------- Footnotes ----------
(1) https://packaging.python.org
(2) https://peps.python.org/pep-0453/
File: python.info, Node: PEP 476 Enabling certificate verification by default for stdlib http clients<2>, Next: PEP 493 HTTPS verification migration tools for Python 2 7, Prev: PEP 477 Backport ensurepip PEP 453 to Python 2 7, Up: New Features Added to Python 2 7 Maintenance Releases
1.15.15.7 PEP 476: Enabling certificate verification by default for stdlib http clients
.......................................................................................
PEP 476(1) updated *note httplib: 6e. and modules which use it, such as
*note urllib2: 10b. and *note xmlrpclib: 12e, to now verify that the
server presents a certificate which is signed by a Certificate Authority
in the platform trust store and whose hostname matches the hostname
being requested by default, significantly improving security for many
applications. This change was made in the Python 2.7.9 release.
For applications which require the old previous behavior, they can pass
an alternate context:
import urllib2
import ssl
# This disables all verification
context = ssl._create_unverified_context()
# This allows using a specific certificate for the host, which doesn't need
# to be in the trust store
context = ssl.create_default_context(cafile="/path/to/file.crt")
urllib2.urlopen("https://invalid-cert", context=context)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0476/
File: python.info, Node: PEP 493 HTTPS verification migration tools for Python 2 7, Next: New make regen-all build target<3>, Prev: PEP 476 Enabling certificate verification by default for stdlib http clients<2>, Up: New Features Added to Python 2 7 Maintenance Releases
1.15.15.8 PEP 493: HTTPS verification migration tools for Python 2.7
....................................................................
PEP 493(1) provides additional migration tools to support a more
incremental infrastructure upgrade process for environments containing
applications and services relying on the historically permissive
processing of server certificates when establishing client HTTPS
connections. These additions were made in the Python 2.7.12 release.
These tools are intended for use in cases where affected applications
and services can’t be modified to explicitly pass a more permissive SSL
context when establishing the connection.
For applications and services which can’t be modified at all, the new
‘PYTHONHTTPSVERIFY’ environment variable may be set to ‘0’ to revert an
entire Python process back to the default permissive behaviour of Python
2.7.8 and earlier.
For cases where the connection establishment code can’t be modified, but
the overall application can be, the new
‘ssl._https_verify_certificates()’ function can be used to adjust the
default behaviour at runtime.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0493/
File: python.info, Node: New make regen-all build target<3>, Next: Removal of make touch build target<3>, Prev: PEP 493 HTTPS verification migration tools for Python 2 7, Up: New Features Added to Python 2 7 Maintenance Releases
1.15.15.9 New ‘make regen-all’ build target
...........................................
To simplify cross-compilation, and to ensure that CPython can reliably
be compiled without requiring an existing version of Python to already
be available, the autotools-based build system no longer attempts to
implicitly recompile generated files based on file modification times.
Instead, a new ‘make regen-all’ command has been added to force
regeneration of these files when desired (e.g. after an initial version
of Python has already been built based on the pregenerated versions).
More selective regeneration targets are also defined - see
Makefile.pre.in(1) for details.
(Contributed by Victor Stinner in bpo-23404(2).)
Added in version 2.7.14.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Makefile.pre.in
(2) https://bugs.python.org/issue?@action=redirect&bpo=23404
File: python.info, Node: Removal of make touch build target<3>, Prev: New make regen-all build target<3>, Up: New Features Added to Python 2 7 Maintenance Releases
1.15.15.10 Removal of ‘make touch’ build target
...............................................
The ‘make touch’ build target previously used to request implicit
regeneration of generated files by updating their modification times has
been removed.
It has been replaced by the new ‘make regen-all’ target.
(Contributed by Victor Stinner in bpo-23404(1).)
Changed in version 2.7.14.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=23404
File: python.info, Node: Acknowledgements, Prev: New Features Added to Python 2 7 Maintenance Releases, Up: What’s New in Python 2 7
1.15.16 Acknowledgements
------------------------
The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Nick Coghlan, Philip Jenvey, Ryan Lovett, R. David Murray, Hugh
Secker-Walker.
File: python.info, Node: What’s New in Python 2 6, Next: What’s New in Python 2 5, Prev: What’s New in Python 2 7, Up: What’s New in Python
1.16 What’s New in Python 2.6
=============================
Author: A.M. Kuchling (amk at amk.ca)
This article explains the new features in Python 2.6, released on
October 1, 2008. The release schedule is described in PEP 361(1).
The major theme of Python 2.6 is preparing the migration path to Python
3.0, a major redesign of the language. Whenever possible, Python 2.6
incorporates new features and syntax from 3.0 while remaining compatible
with existing code by not removing older features or syntax. When it’s
not possible to do that, Python 2.6 tries to do what it can, adding
compatibility functions in a ‘future_builtins’ module and a ‘-3’ switch
to warn about usages that will become unsupported in 3.0.
Some significant new packages have been added to the standard library,
such as the *note multiprocessing: 94. and *note json: 82. modules, but
there aren’t many new features that aren’t related to Python 3.0 in some
way.
Python 2.6 also sees a number of improvements and bugfixes throughout
the source. A search through the change logs finds there were 259
patches applied and 612 bugs fixed between Python 2.5 and 2.6. Both
figures are likely to be underestimates.
This article doesn’t attempt to provide a complete specification of the
new features, but instead provides a convenient overview. For full
details, you should refer to the documentation for Python 2.6. If you
want to understand the rationale for the design and implementation,
refer to the PEP for a particular new feature. Whenever possible,
“What’s New in Python” links to the bug/patch item for each change.
* Menu:
* Python 3.0: Python 3 0.
* Changes to the Development Process::
* PEP 343; The ‘with’ statement: PEP 343 The ‘with’ statement.
* PEP 366; Explicit Relative Imports From a Main Module: PEP 366 Explicit Relative Imports From a Main Module.
* PEP 370; Per-user site-packages Directory: PEP 370 Per-user site-packages Directory.
* PEP 371; The multiprocessing Package: PEP 371 The multiprocessing Package.
* PEP 3101; Advanced String Formatting: PEP 3101 Advanced String Formatting.
* PEP 3105; print As a Function: PEP 3105 print As a Function.
* PEP 3110; Exception-Handling Changes: PEP 3110 Exception-Handling Changes.
* PEP 3112; Byte Literals: PEP 3112 Byte Literals.
* PEP 3116; New I/O Library: PEP 3116 New I/O Library.
* PEP 3118; Revised Buffer Protocol: PEP 3118 Revised Buffer Protocol.
* PEP 3119; Abstract Base Classes: PEP 3119 Abstract Base Classes.
* PEP 3127; Integer Literal Support and Syntax: PEP 3127 Integer Literal Support and Syntax.
* PEP 3129; Class Decorators: PEP 3129 Class Decorators.
* PEP 3141; A Type Hierarchy for Numbers: PEP 3141 A Type Hierarchy for Numbers.
* Other Language Changes: Other Language Changes<15>.
* New and Improved Modules: New and Improved Modules<2>.
* Deprecations and Removals::
* Build and C API Changes: Build and C API Changes<9>.
* Porting to Python 2.6: Porting to Python 2 6.
* Acknowledgements: Acknowledgements<2>.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0361/
File: python.info, Node: Python 3 0, Next: Changes to the Development Process, Up: What’s New in Python 2 6
1.16.1 Python 3.0
-----------------
The development cycle for Python versions 2.6 and 3.0 was synchronized,
with the alpha and beta releases for both versions being made on the
same days. The development of 3.0 has influenced many features in 2.6.
Python 3.0 is a far-ranging redesign of Python that breaks compatibility
with the 2.x series. This means that existing Python code will need
some conversion in order to run on Python 3.0. However, not all the
changes in 3.0 necessarily break compatibility. In cases where new
features won’t cause existing code to break, they’ve been backported to
2.6 and are described in this document in the appropriate place. Some
of the 3.0-derived features are:
* A ‘__complex__()’ method for converting objects to a complex
number.
* Alternate syntax for catching exceptions: ‘except TypeError as
exc’.
* The addition of *note functools.reduce(): 12cf. as a synonym for
the built-in ‘reduce()’ function.
Python 3.0 adds several new built-in functions and changes the semantics
of some existing builtins. Functions that are new in 3.0 such as *note
bin(): 129b. have simply been added to Python 2.6, but existing builtins
haven’t been changed; instead, the ‘future_builtins’ module has versions
with the new 3.0 semantics. Code written to be compatible with 3.0 can
do ‘from future_builtins import hex, map’ as necessary.
A new command-line switch, ‘-3’, enables warnings about features that
will be removed in Python 3.0. You can run code with this switch to see
how much work will be necessary to port code to 3.0. The value of this
switch is available to Python code as the boolean variable
‘sys.py3kwarning’, and to C extension code as ‘Py_Py3kWarningFlag’.
See also
........
The 3'xxx' series of PEPs, which contains proposals for Python 3.0. PEP
3000(1) describes the development process for Python 3.0. Start with
PEP 3100(2) that describes the general goals for Python 3.0, and then
explore the higher-numbered PEPs that propose specific features.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3000/
(2) https://peps.python.org/pep-3100/
File: python.info, Node: Changes to the Development Process, Next: PEP 343 The ‘with’ statement, Prev: Python 3 0, Up: What’s New in Python 2 6
1.16.2 Changes to the Development Process
-----------------------------------------
While 2.6 was being developed, the Python development process underwent
two significant changes: we switched from SourceForge’s issue tracker to
a customized Roundup installation, and the documentation was converted
from LaTeX to reStructuredText.
* Menu:
* New Issue Tracker; Roundup: New Issue Tracker Roundup.
* New Documentation Format; reStructuredText Using Sphinx: New Documentation Format reStructuredText Using Sphinx.
File: python.info, Node: New Issue Tracker Roundup, Next: New Documentation Format reStructuredText Using Sphinx, Up: Changes to the Development Process
1.16.2.1 New Issue Tracker: Roundup
...................................
For a long time, the Python developers had been growing increasingly
annoyed by SourceForge’s bug tracker. SourceForge’s hosted solution
doesn’t permit much customization; for example, it wasn’t possible to
customize the life cycle of issues.
The infrastructure committee of the Python Software Foundation therefore
posted a call for issue trackers, asking volunteers to set up different
products and import some of the bugs and patches from SourceForge. Four
different trackers were examined: Jira(1), Launchpad(2), Roundup(3), and
Trac(4). The committee eventually settled on Jira and Roundup as the
two candidates. Jira is a commercial product that offers no-cost hosted
instances to free-software projects; Roundup is an open-source project
that requires volunteers to administer it and a server to host it.
After posting a call for volunteers, a new Roundup installation was set
up at ‘https://bugs.python.org’. One installation of Roundup can host
multiple trackers, and this server now also hosts issue trackers for
Jython and for the Python web site. It will surely find other uses in
the future. Where possible, this edition of “What’s New in Python”
links to the bug/patch item for each change.
Hosting of the Python bug tracker is kindly provided by Upfront
Systems(5) of Stellenbosch, South Africa. Martin von Löwis put a lot of
effort into importing existing bugs and patches from SourceForge; his
scripts for this import operation are at
‘https://svn.python.org/view/tracker/importer/’ and may be useful to
other projects wishing to move from SourceForge to Roundup.
See also
........
‘https://bugs.python.org’
The Python bug tracker.
‘https://bugs.jython.org’:
The Jython bug tracker.
‘https://roundup.sourceforge.io/’
Roundup downloads and documentation.
‘https://svn.python.org/view/tracker/importer/’
Martin von Löwis’s conversion scripts.
---------- Footnotes ----------
(1) https://www.atlassian.com/software/jira/
(2) https://launchpad.net/
(3) https://roundup.sourceforge.io/
(4) https://trac.edgewall.org/
(5) https://upfrontsoftware.co.za
File: python.info, Node: New Documentation Format reStructuredText Using Sphinx, Prev: New Issue Tracker Roundup, Up: Changes to the Development Process
1.16.2.2 New Documentation Format: reStructuredText Using Sphinx
................................................................
The Python documentation was written using LaTeX since the project
started around 1989. In the 1980s and early 1990s, most documentation
was printed out for later study, not viewed online. LaTeX was widely
used because it provided attractive printed output while remaining
straightforward to write once the basic rules of the markup were
learned.
Today LaTeX is still used for writing publications destined for
printing, but the landscape for programming tools has shifted. We no
longer print out reams of documentation; instead, we browse through it
online and HTML has become the most important format to support.
Unfortunately, converting LaTeX to HTML is fairly complicated and Fred
L. Drake Jr., the long-time Python documentation editor, spent a lot of
time maintaining the conversion process. Occasionally people would
suggest converting the documentation into SGML and later XML, but
performing a good conversion is a major task and no one ever committed
the time required to finish the job.
During the 2.6 development cycle, Georg Brandl put a lot of effort into
building a new toolchain for processing the documentation. The
resulting package is called Sphinx, and is available from
‘https://www.sphinx-doc.org/’.
Sphinx concentrates on HTML output, producing attractively styled and
modern HTML; printed output is still supported through conversion to
LaTeX. The input format is reStructuredText, a markup syntax supporting
custom extensions and directives that is commonly used in the Python
community.
Sphinx is a standalone package that can be used for writing, and almost
two dozen other projects (listed on the Sphinx web site(1)) have adopted
Sphinx as their documentation tool.
See also
........
Documenting Python(2)
Describes how to write for Python’s documentation.
Sphinx(3)
Documentation and code for the Sphinx toolchain.
Docutils(4)
The underlying reStructuredText parser and toolset.
---------- Footnotes ----------
(1) https://www.sphinx-doc.org/en/master/examples.html
(2) https://devguide.python.org/documenting/
(3) https://www.sphinx-doc.org/
(4) https://docutils.sourceforge.io
File: python.info, Node: PEP 343 The ‘with’ statement, Next: PEP 366 Explicit Relative Imports From a Main Module, Prev: Changes to the Development Process, Up: What’s New in Python 2 6
1.16.3 PEP 343: The ‘with’ statement
------------------------------------
The previous version, Python 2.5, added the ‘*note with: 5ce.’ statement
as an optional feature, to be enabled by a ‘from __future__ import
with_statement’ directive. In 2.6 the statement no longer needs to be
specially enabled; this means that ‘with’ is now always a keyword. The
rest of this section is a copy of the corresponding section from the
“What’s New in Python 2.5” document; if you’re familiar with the
‘‘with’’ statement from Python 2.5, you can skip this section.
The ‘*note with: 5ce.’ statement clarifies code that previously would
use ‘try...finally’ blocks to ensure that clean-up code is executed. In
this section, I’ll discuss the statement as it will commonly be used.
In the next section, I’ll examine the implementation details and show
how to write objects for use with this statement.
The ‘*note with: 5ce.’ statement is a control-flow structure whose basic
structure is:
with expression [as variable]:
with-block
The expression is evaluated, and it should result in an object that
supports the context management protocol (that is, has *note
__enter__(): 5c4. and *note __exit__(): 12f3. methods).
The object’s *note __enter__(): 5c4. is called before 'with-block' is
executed and therefore can run set-up code. It also may return a value
that is bound to the name 'variable', if given. (Note carefully that
'variable' is 'not' assigned the result of 'expression'.)
After execution of the 'with-block' is finished, the object’s *note
__exit__(): 12f3. method is called, even if the block raised an
exception, and can therefore run clean-up code.
Some standard Python objects now support the context management protocol
and can be used with the ‘*note with: 5ce.’ statement. File objects are
one example:
with open('/etc/passwd', 'r') as f:
for line in f:
print line
... more processing code ...
After this statement has executed, the file object in 'f' will have been
automatically closed, even if the *note for: 2ec. loop raised an
exception part-way through the block.
Note: In this case, 'f' is the same object created by *note open():
517, because *note __enter__(): 5c4. returns 'self'.
The *note threading: ed. module’s locks and condition variables also
support the ‘*note with: 5ce.’ statement:
lock = threading.Lock()
with lock:
# Critical section of code
...
The lock is acquired before the block is executed and always released
once the block is complete.
The ‘localcontext()’ function in the *note decimal: 36. module makes it
easy to save and restore the current decimal context, which encapsulates
the desired precision and rounding characteristics for computations:
from decimal import Decimal, Context, localcontext
# Displays with default precision of 28 digits
v = Decimal('578')
print v.sqrt()
with localcontext(Context(prec=16)):
# All code in this block uses a precision of 16 digits.
# The original context is restored on exiting the block.
print v.sqrt()
* Menu:
* Writing Context Managers::
* The contextlib module::
File: python.info, Node: Writing Context Managers, Next: The contextlib module, Up: PEP 343 The ‘with’ statement
1.16.3.1 Writing Context Managers
.................................
Under the hood, the ‘*note with: 5ce.’ statement is fairly complicated.
Most people will only use ‘‘with’’ in company with existing objects and
don’t need to know these details, so you can skip the rest of this
section if you like. Authors of new objects will need to understand the
details of the underlying implementation and should keep reading.
A high-level explanation of the context management protocol is:
* The expression is evaluated and should result in an object called a
“context manager”. The context manager must have *note
__enter__(): 5c4. and *note __exit__(): 12f3. methods.
* The context manager’s *note __enter__(): 5c4. method is called.
The value returned is assigned to 'VAR'. If no ‘as VAR’ clause is
present, the value is simply discarded.
* The code in 'BLOCK' is executed.
* If 'BLOCK' raises an exception, the context manager’s *note
__exit__(): 12f3. method is called with three arguments, the
exception details (‘type, value, traceback’, the same values
returned by *note sys.exc_info(): 686, which can also be ‘None’ if
no exception occurred). The method’s return value controls whether
an exception is re-raised: any false value re-raises the exception,
and ‘True’ will result in suppressing it. You’ll only rarely want
to suppress the exception, because if you do the author of the code
containing the ‘*note with: 5ce.’ statement will never realize
anything went wrong.
* If 'BLOCK' didn’t raise an exception, the *note __exit__(): 12f3.
method is still called, but 'type', 'value', and 'traceback' are
all ‘None’.
Let’s think through an example. I won’t present detailed code but will
only sketch the methods necessary for a database that supports
transactions.
(For people unfamiliar with database terminology: a set of changes to
the database are grouped into a transaction. Transactions can be either
committed, meaning that all the changes are written into the database,
or rolled back, meaning that the changes are all discarded and the
database is unchanged. See any database textbook for more information.)
Let’s assume there’s an object representing a database connection. Our
goal will be to let the user write code like this:
db_connection = DatabaseConnection()
with db_connection as cursor:
cursor.execute('insert into ...')
cursor.execute('delete from ...')
# ... more operations ...
The transaction should be committed if the code in the block runs
flawlessly or rolled back if there’s an exception. Here’s the basic
interface for ‘DatabaseConnection’ that I’ll assume:
class DatabaseConnection:
# Database interface
def cursor(self):
"Returns a cursor object and starts a new transaction"
def commit(self):
"Commits current transaction"
def rollback(self):
"Rolls back current transaction"
The *note __enter__(): 5c4. method is pretty easy, having only to start
a new transaction. For this application the resulting cursor object
would be a useful result, so the method will return it. The user can
then add ‘as cursor’ to their ‘*note with: 5ce.’ statement to bind the
cursor to a variable name.
class DatabaseConnection:
...
def __enter__(self):
# Code to start a new transaction
cursor = self.cursor()
return cursor
The *note __exit__(): 12f3. method is the most complicated because it’s
where most of the work has to be done. The method has to check if an
exception occurred. If there was no exception, the transaction is
committed. The transaction is rolled back if there was an exception.
In the code below, execution will just fall off the end of the function,
returning the default value of ‘None’. ‘None’ is false, so the
exception will be re-raised automatically. If you wished, you could be
more explicit and add a *note return: 9ce. statement at the marked
location.
class DatabaseConnection:
...
def __exit__(self, type, value, tb):
if tb is None:
# No exception, so commit
self.commit()
else:
# Exception occurred, so rollback.
self.rollback()
# return False
File: python.info, Node: The contextlib module, Prev: Writing Context Managers, Up: PEP 343 The ‘with’ statement
1.16.3.2 The contextlib module
..............................
The *note contextlib: 23. module provides some functions and a decorator
that are useful when writing objects for use with the ‘*note with: 5ce.’
statement.
The decorator is called ‘contextmanager()’, and lets you write a single
generator function instead of defining a new class. The generator
should yield exactly one value. The code up to the *note yield: 9cd.
will be executed as the *note __enter__(): 5c4. method, and the value
yielded will be the method’s return value that will get bound to the
variable in the ‘*note with: 5ce.’ statement’s ‘as’ clause, if any. The
code after the ‘yield’ will be executed in the *note __exit__(): 12f3.
method. Any exception raised in the block will be raised by the ‘yield’
statement.
Using this decorator, our database example from the previous section
could be written as:
from contextlib import contextmanager
@contextmanager
def db_transaction(connection):
cursor = connection.cursor()
try:
yield cursor
except:
connection.rollback()
raise
else:
connection.commit()
db = DatabaseConnection()
with db_transaction(db) as cursor:
...
The *note contextlib: 23. module also has a ‘nested(mgr1, mgr2, ...)’
function that combines a number of context managers so you don’t need to
write nested ‘*note with: 5ce.’ statements. In this example, the single
‘‘with’’ statement both starts a database transaction and acquires a
thread lock:
lock = threading.Lock()
with nested (db_transaction(db), lock) as (cursor, locked):
...
Finally, the ‘closing()’ function returns its argument so that it can be
bound to a variable, and calls the argument’s ‘.close()’ method at the
end of the block.
import urllib, sys
from contextlib import closing
with closing(urllib.urlopen('http://www.yahoo.com')) as f:
for line in f:
sys.stdout.write(line)
See also
........
PEP 343(1) - The “with” statement
PEP written by Guido van Rossum and Nick Coghlan; implemented by
Mike Bland, Guido van Rossum, and Neal Norwitz. The PEP shows the
code generated for a ‘*note with: 5ce.’ statement, which can be
helpful in learning how the statement works.
The documentation for the *note contextlib: 23. module.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0343/
File: python.info, Node: PEP 366 Explicit Relative Imports From a Main Module, Next: PEP 370 Per-user site-packages Directory, Prev: PEP 343 The ‘with’ statement, Up: What’s New in Python 2 6
1.16.4 PEP 366: Explicit Relative Imports From a Main Module
------------------------------------------------------------
Python’s *note -m: 5dd. switch allows running a module as a script.
When you ran a module that was located inside a package, relative
imports didn’t work correctly.
The fix for Python 2.6 adds a *note module.__package__: 2db. attribute.
When this attribute is present, relative imports will be relative to the
value of this attribute instead of the *note __name__: 1374. attribute.
PEP 302-style importers can then set *note __package__: 2db. as
necessary. The *note runpy: be. module that implements the *note -m:
5dd. switch now does this, so relative imports will now work correctly
in scripts running from inside a package.
File: python.info, Node: PEP 370 Per-user site-packages Directory, Next: PEP 371 The multiprocessing Package, Prev: PEP 366 Explicit Relative Imports From a Main Module, Up: What’s New in Python 2 6
1.16.5 PEP 370: Per-user ‘site-packages’ Directory
--------------------------------------------------
When you run Python, the module search path ‘sys.path’ usually includes
a directory whose path ends in ‘"site-packages"’. This directory is
intended to hold locally installed packages available to all users using
a machine or a particular site installation.
Python 2.6 introduces a convention for user-specific site directories.
The directory varies depending on the platform:
* Unix and Mac OS X: ‘~/.local/’
* Windows: ‘%APPDATA%/Python’
Within this directory, there will be version-specific subdirectories,
such as ‘lib/python2.6/site-packages’ on Unix/Mac OS and
‘Python26/site-packages’ on Windows.
If you don’t like the default directory, it can be overridden by an
environment variable. *note PYTHONUSERBASE: 1376. sets the root
directory used for all Python versions supporting this feature. On
Windows, the directory for application-specific data can be changed by
setting the ‘APPDATA’ environment variable. You can also modify the
‘site.py’ file for your Python installation.
The feature can be disabled entirely by running Python with the *note
-s: 1377. option or setting the *note PYTHONNOUSERSITE: 1378.
environment variable.
See also
........
PEP 370(1) - Per-user ‘site-packages’ Directory
PEP written and implemented by Christian Heimes.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0370/
File: python.info, Node: PEP 371 The multiprocessing Package, Next: PEP 3101 Advanced String Formatting, Prev: PEP 370 Per-user site-packages Directory, Up: What’s New in Python 2 6
1.16.6 PEP 371: The ‘multiprocessing’ Package
---------------------------------------------
The new *note multiprocessing: 94. package lets Python programs create
new processes that will perform a computation and return a result to the
parent. The parent and child processes can communicate using queues and
pipes, synchronize their operations using locks and semaphores, and can
share simple arrays of data.
The *note multiprocessing: 94. module started out as an exact emulation
of the *note threading: ed. module using processes instead of threads.
That goal was discarded along the path to Python 2.6, but the general
approach of the module is still similar. The fundamental class is the
‘Process’, which is passed a callable object and a collection of
arguments. The ‘start()’ method sets the callable running in a
subprocess, after which you can call the ‘is_alive()’ method to check
whether the subprocess is still running and the ‘join()’ method to wait
for the process to exit.
Here’s a simple example where the subprocess will calculate a factorial.
The function doing the calculation is written strangely so that it takes
significantly longer when the input argument is a multiple of 4.
import time
from multiprocessing import Process, Queue
def factorial(queue, N):
"Compute a factorial."
# If N is a multiple of 4, this function will take much longer.
if (N % 4) == 0:
time.sleep(.05 * N/4)
# Calculate the result
fact = 1L
for i in range(1, N+1):
fact = fact * i
# Put the result on the queue
queue.put(fact)
if __name__ == '__main__':
queue = Queue()
N = 5
p = Process(target=factorial, args=(queue, N))
p.start()
p.join()
result = queue.get()
print 'Factorial', N, '=', result
A *note Queue: 137a. is used to communicate the result of the factorial.
The *note Queue: 137a. object is stored in a global variable. The child
process will use the value of the variable when the child was created;
because it’s a *note Queue: 137a, parent and child can use the object to
communicate. (If the parent were to change the value of the global
variable, the child’s value would be unaffected, and vice versa.)
Two other classes, ‘Pool’ and ‘Manager’, provide higher-level
interfaces. ‘Pool’ will create a fixed number of worker processes, and
requests can then be distributed to the workers by calling ‘apply()’ or
‘apply_async()’ to add a single request, and *note map(): 860. or
‘map_async()’ to add a number of requests. The following code uses a
‘Pool’ to spread requests across 5 worker processes and retrieve a list
of results:
from multiprocessing import Pool
def factorial(N, dictionary):
"Compute a factorial."
...
p = Pool(5)
result = p.map(factorial, range(1, 1000, 10))
for v in result:
print v
This produces the following output:
1
39916800
51090942171709440000
8222838654177922817725562880000000
33452526613163807108170062053440751665152000000000
...
The other high-level interface, the ‘Manager’ class, creates a separate
server process that can hold master copies of Python data structures.
Other processes can then access and modify these data structures using
proxy objects. The following example creates a shared dictionary by
calling the *note dict(): 258. method; the worker processes then insert
values into the dictionary. (Locking is not done for you automatically,
which doesn’t matter in this example. ‘Manager’’s methods also include
‘Lock()’, ‘RLock()’, and ‘Semaphore()’ to create shared locks.)
import time
from multiprocessing import Pool, Manager
def factorial(N, dictionary):
"Compute a factorial."
# Calculate the result
fact = 1L
for i in range(1, N+1):
fact = fact * i
# Store result in dictionary
dictionary[N] = fact
if __name__ == '__main__':
p = Pool(5)
mgr = Manager()
d = mgr.dict() # Create shared dictionary
# Run tasks using the pool
for N in range(1, 1000, 10):
p.apply_async(factorial, (N, d))
# Mark pool as closed -- no more tasks can be added.
p.close()
# Wait for tasks to exit
p.join()
# Output results
for k, v in sorted(d.items()):
print k, v
This will produce the output:
1 1
11 39916800
21 51090942171709440000
31 8222838654177922817725562880000000
41 33452526613163807108170062053440751665152000000000
51 15511187532873822802242430164693032110632597200169861120000...
See also
........
The documentation for the *note multiprocessing: 94. module.
PEP 371(1) - Addition of the multiprocessing package
PEP written by Jesse Noller and Richard Oudkerk; implemented by
Richard Oudkerk and Jesse Noller.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0371/
File: python.info, Node: PEP 3101 Advanced String Formatting, Next: PEP 3105 print As a Function, Prev: PEP 371 The multiprocessing Package, Up: What’s New in Python 2 6
1.16.7 PEP 3101: Advanced String Formatting
-------------------------------------------
In Python 3.0, the ‘%’ operator is supplemented by a more powerful
string formatting method, *note format(): 61b. Support for the *note
str.format(): 61d. method has been backported to Python 2.6.
In 2.6, both 8-bit and Unicode strings have a ‘.format()’ method that
treats the string as a template and takes the arguments to be formatted.
The formatting template uses curly brackets (‘{’, ‘}’) as special
characters:
>>> # Substitute positional argument 0 into the string.
>>> "User ID: {0}".format("root")
'User ID: root'
>>> # Use the named keyword arguments
>>> "User ID: {uid} Last seen: {last_login}".format(
... uid="root",
... last_login = "5 Mar 2008 07:20")
'User ID: root Last seen: 5 Mar 2008 07:20'
Curly brackets can be escaped by doubling them:
>>> "Empty dict: {{}}".format()
"Empty dict: {}"
Field names can be integers indicating positional arguments, such as
‘{0}’, ‘{1}’, etc. or names of keyword arguments. You can also supply
compound field names that read attributes or access dictionary keys:
>>> import sys
>>> print 'Platform: {0.platform}\nPython version: {0.version}'.format(sys)
Platform: darwin
Python version: 2.6a1+ (trunk:61261M, Mar 5 2008, 20:29:41)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)]'
>>> import mimetypes
>>> 'Content-type: {0[.mp4]}'.format(mimetypes.types_map)
'Content-type: video/mp4'
Note that when using dictionary-style notation such as ‘[.mp4]’, you
don’t need to put any quotation marks around the string; it will look up
the value using ‘.mp4’ as the key. Strings beginning with a number will
be converted to an integer. You can’t write more complicated
expressions inside a format string.
So far we’ve shown how to specify which field to substitute into the
resulting string. The precise formatting used is also controllable by
adding a colon followed by a format specifier. For example:
>>> # Field 0: left justify, pad to 15 characters
>>> # Field 1: right justify, pad to 6 characters
>>> fmt = '{0:15} ${1:>6}'
>>> fmt.format('Registration', 35)
'Registration $ 35'
>>> fmt.format('Tutorial', 50)
'Tutorial $ 50'
>>> fmt.format('Banquet', 125)
'Banquet $ 125'
Format specifiers can reference other fields through nesting:
>>> fmt = '{0:{1}}'
>>> width = 15
>>> fmt.format('Invoice #1234', width)
'Invoice #1234 '
>>> width = 35
>>> fmt.format('Invoice #1234', width)
'Invoice #1234 '
The alignment of a field within the desired width can be specified:
Character Effect
----------------------------------------------------------------------
< (default) Left-align
> Right-align
^ Center
= (For numeric types only) Pad after the sign.
Format specifiers can also include a presentation type, which controls
how the value is formatted. For example, floating-point numbers can be
formatted as a general number or in exponential notation:
>>> '{0:g}'.format(3.75)
'3.75'
>>> '{0:e}'.format(3.75)
'3.750000e+00'
A variety of presentation types are available. Consult the 2.6
documentation for a *note complete list: 137c.; here’s a sample:
‘b’ Binary. Outputs the number in base 2.
‘c’ Character. Converts the integer to the corresponding Unicode character
before printing.
‘d’ Decimal Integer. Outputs the number in base 10.
‘o’ Octal format. Outputs the number in base 8.
‘x’ Hex format. Outputs the number in base 16, using lower-case letters for
the digits above 9.
‘e’ Exponent notation. Prints the number in scientific notation using the
letter ‘e’ to indicate the exponent.
‘g’ General format. This prints the number as a fixed-point number, unless
the number is too large, in which case it switches to ‘e’ exponent
notation.
‘n’ Number. This is the same as ‘g’ (for floats) or ‘d’ (for integers),
except that it uses the current locale setting to insert the appropriate
number separator characters.
‘%’ Percentage. Multiplies the number by 100 and displays in fixed (‘f’)
format, followed by a percent sign.
Classes and types can define a ‘__format__()’ method to control how
they’re formatted. It receives a single argument, the format specifier:
def __format__(self, format_spec):
if isinstance(format_spec, unicode):
return unicode(str(self))
else:
return str(self)
There’s also a *note format(): 61b. builtin that will format a single
value. It calls the type’s ‘__format__()’ method with the provided
specifier:
>>> format(75.6564, '.2f')
'75.66'
See also
........
*note Format String Syntax: 137c.
The reference documentation for format fields.
PEP 3101(1) - Advanced String Formatting
PEP written by Talin. Implemented by Eric Smith.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3101/
File: python.info, Node: PEP 3105 print As a Function, Next: PEP 3110 Exception-Handling Changes, Prev: PEP 3101 Advanced String Formatting, Up: What’s New in Python 2 6
1.16.8 PEP 3105: ‘print’ As a Function
--------------------------------------
The ‘print’ statement becomes the *note print(): f70. function in Python
3.0. Making *note print(): f70. a function makes it possible to replace
the function by doing ‘def print(...)’ or importing a new function from
somewhere else.
Python 2.6 has a ‘__future__’ import that removes ‘print’ as language
syntax, letting you use the functional form instead. For example:
>>> from __future__ import print_function
>>> print('# of entries', len(dictionary), file=sys.stderr)
The signature of the new function is:
def print(*args, sep=' ', end='\n', file=None)
The parameters are:
* 'args': positional arguments whose values will be printed out.
* 'sep': the separator, which will be printed between arguments.
* 'end': the ending text, which will be printed after all of the
arguments have been output.
* 'file': the file object to which the output will be sent.
See also
........
PEP 3105(1) - Make print a function
PEP written by Georg Brandl.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3105/
File: python.info, Node: PEP 3110 Exception-Handling Changes, Next: PEP 3112 Byte Literals, Prev: PEP 3105 print As a Function, Up: What’s New in Python 2 6
1.16.9 PEP 3110: Exception-Handling Changes
-------------------------------------------
One error that Python programmers occasionally make is writing the
following code:
try:
...
except TypeError, ValueError: # Wrong!
...
The author is probably trying to catch both *note TypeError: 534. and
*note ValueError: 204. exceptions, but this code actually does something
different: it will catch *note TypeError: 534. and bind the resulting
exception object to the local name ‘"ValueError"’. The *note
ValueError: 204. exception will not be caught at all. The correct code
specifies a tuple of exceptions:
try:
...
except (TypeError, ValueError):
...
This error happens because the use of the comma here is ambiguous: does
it indicate two different nodes in the parse tree, or a single node
that’s a tuple?
Python 3.0 makes this unambiguous by replacing the comma with the word
“as”. To catch an exception and store the exception object in the
variable ‘exc’, you must write:
try:
...
except TypeError as exc:
...
Python 3.0 will only support the use of “as”, and therefore interprets
the first example as catching two different exceptions. Python 2.6
supports both the comma and “as”, so existing code will continue to
work. We therefore suggest using “as” when writing new Python code that
will only be executed with 2.6.
See also
........
PEP 3110(1) - Catching Exceptions in Python 3000
PEP written and implemented by Collin Winter.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3110/
File: python.info, Node: PEP 3112 Byte Literals, Next: PEP 3116 New I/O Library, Prev: PEP 3110 Exception-Handling Changes, Up: What’s New in Python 2 6
1.16.10 PEP 3112: Byte Literals
-------------------------------
Python 3.0 adopts Unicode as the language’s fundamental string type and
denotes 8-bit literals differently, either as ‘b'string'’ or using a
*note bytes: 1c2. constructor. For future compatibility, Python 2.6
adds *note bytes: 1c2. as a synonym for the *note str: 447. type, and it
also supports the ‘b''’ notation.
The 2.6 *note str: 447. differs from 3.0’s *note bytes: 1c2. type in
various ways; most notably, the constructor is completely different. In
3.0, ‘bytes([65, 66, 67])’ is 3 elements long, containing the bytes
representing ‘ABC’; in 2.6, ‘bytes([65, 66, 67])’ returns the 12-byte
string representing the *note str(): 447. of the list.
The primary use of *note bytes: 1c2. in 2.6 will be to write tests of
object type such as ‘isinstance(x, bytes)’. This will help the 2to3
converter, which can’t tell whether 2.x code intends strings to contain
either characters or 8-bit bytes; you can now use either *note bytes:
1c2. or *note str: 447. to represent your intention exactly, and the
resulting code will also be correct in Python 3.0.
There’s also a ‘__future__’ import that causes all string literals to
become Unicode strings. This means that ‘\u’ escape sequences can be
used to include Unicode characters:
from __future__ import unicode_literals
s = ('\u751f\u3080\u304e\u3000\u751f\u3054'
'\u3081\u3000\u751f\u305f\u307e\u3054')
print len(s) # 12 Unicode characters
At the C level, Python 3.0 will rename the existing 8-bit string type,
called ‘PyStringObject’ in Python 2.x, to *note PyBytesObject: 78d.
Python 2.6 uses ‘#define’ to support using the names *note
PyBytesObject(): 78d, *note PyBytes_Check(): 1380, *note
PyBytes_FromStringAndSize(): 1381, and all the other functions and
macros used with strings.
Instances of the *note bytes: 1c2. type are immutable just as strings
are. A new *note bytearray: 53a. type stores a mutable sequence of
bytes:
>>> bytearray([65, 66, 67])
bytearray(b'ABC')
>>> b = bytearray(u'\u21ef\u3244', 'utf-8')
>>> b
bytearray(b'\xe2\x87\xaf\xe3\x89\x84')
>>> b[0] = '\xe3'
>>> b
bytearray(b'\xe3\x87\xaf\xe3\x89\x84')
>>> unicode(str(b), 'utf-8')
u'\u31ef \u3244'
Byte arrays support most of the methods of string types, such as
‘startswith()’/‘endswith()’, ‘find()’/‘rfind()’, and some of the methods
of lists, such as ‘append()’, ‘pop()’, and ‘reverse()’.
>>> b = bytearray('ABC')
>>> b.append('d')
>>> b.append(ord('e'))
>>> b
bytearray(b'ABCde')
There’s also a corresponding C API, with *note PyByteArray_FromObject():
1382, *note PyByteArray_FromStringAndSize(): 1383, and various other
functions.
See also
........
PEP 3112(1) - Bytes literals in Python 3000
PEP written by Jason Orendorff; backported to 2.6 by Christian
Heimes.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3112/
File: python.info, Node: PEP 3116 New I/O Library, Next: PEP 3118 Revised Buffer Protocol, Prev: PEP 3112 Byte Literals, Up: What’s New in Python 2 6
1.16.11 PEP 3116: New I/O Library
---------------------------------
Python’s built-in file objects support a number of methods, but
file-like objects don’t necessarily support all of them. Objects that
imitate files usually support ‘read()’ and ‘write()’, but they may not
support *note readline(): ba, for example. Python 3.0 introduces a
layered I/O library in the *note io: 7f. module that separates buffering
and text-handling features from the fundamental read and write
operations.
There are three levels of abstract base classes provided by the *note
io: 7f. module:
* ‘RawIOBase’ defines raw I/O operations: ‘read()’, ‘readinto()’,
‘write()’, ‘seek()’, ‘tell()’, ‘truncate()’, and ‘close()’. Most
of the methods of this class will often map to a single system
call. There are also ‘readable()’, ‘writable()’, and ‘seekable()’
methods for determining what operations a given object will allow.
Python 3.0 has concrete implementations of this class for files and
sockets, but Python 2.6 hasn’t restructured its file and socket
objects in this way.
* ‘BufferedIOBase’ is an abstract base class that buffers data in
memory to reduce the number of system calls used, making I/O
processing more efficient. It supports all of the methods of
‘RawIOBase’, and adds a ‘raw’ attribute holding the underlying raw
object.
There are five concrete classes implementing this ABC.
‘BufferedWriter’ and ‘BufferedReader’ are for objects that support
write-only or read-only usage that have a ‘seek()’ method for
random access. ‘BufferedRandom’ objects support read and write
access upon the same underlying stream, and ‘BufferedRWPair’ is for
objects such as TTYs that have both read and write operations
acting upon unconnected streams of data. The ‘BytesIO’ class
supports reading, writing, and seeking over an in-memory buffer.
* ‘TextIOBase’: Provides functions for reading and writing strings
(remember, strings will be Unicode in Python 3.0), and supporting
*note universal newlines: d3a. ‘TextIOBase’ defines the *note
readline(): ba. method and supports iteration upon objects.
There are two concrete implementations. ‘TextIOWrapper’ wraps a
buffered I/O object, supporting all of the methods for text I/O and
adding a ‘buffer’ attribute for access to the underlying object.
‘StringIO’ simply buffers everything in memory without ever writing
anything to disk.
(In Python 2.6, *note io.StringIO: f22. is implemented in pure
Python, so it’s pretty slow. You should therefore stick with the
existing ‘StringIO’ module or ‘cStringIO’ for now. At some point
Python 3.0’s *note io: 7f. module will be rewritten into C for
speed, and perhaps the C implementation will be backported to the
2.x releases.)
In Python 2.6, the underlying implementations haven’t been restructured
to build on top of the *note io: 7f. module’s classes. The module is
being provided to make it easier to write code that’s forward-compatible
with 3.0, and to save developers the effort of writing their own
implementations of buffering and text I/O.
See also
........
PEP 3116(1) - New I/O
PEP written by Daniel Stutzbach, Mike Verdone, and Guido van
Rossum. Code by Guido van Rossum, Georg Brandl, Walter Doerwald,
Jeremy Hylton, Martin von Löwis, Tony Lownds, and others.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3116/
File: python.info, Node: PEP 3118 Revised Buffer Protocol, Next: PEP 3119 Abstract Base Classes, Prev: PEP 3116 New I/O Library, Up: What’s New in Python 2 6
1.16.12 PEP 3118: Revised Buffer Protocol
-----------------------------------------
The buffer protocol is a C-level API that lets Python types exchange
pointers into their internal representations. A memory-mapped file can
be viewed as a buffer of characters, for example, and this lets another
module such as *note re: b9. treat memory-mapped files as a string of
characters to be searched.
The primary users of the buffer protocol are numeric-processing packages
such as NumPy, which expose the internal representation of arrays so
that callers can write data directly into an array instead of going
through a slower API. This PEP updates the buffer protocol in light of
experience from NumPy development, adding a number of new features such
as indicating the shape of an array or locking a memory region.
The most important new C API function is ‘PyObject_GetBuffer(PyObject
*obj, Py_buffer *view, int flags)’, which takes an object and a set of
flags, and fills in the ‘Py_buffer’ structure with information about the
object’s memory representation. Objects can use this operation to lock
memory in place while an external caller could be modifying the
contents, so there’s a corresponding ‘PyBuffer_Release(Py_buffer *view)’
to indicate that the external caller is done.
The 'flags' argument to *note PyObject_GetBuffer(): 395. specifies
constraints upon the memory returned. Some examples are:
* *note PyBUF_WRITABLE: 1386. indicates that the memory must be
writable.
* ‘PyBUF_LOCK’ requests a read-only or exclusive lock on the memory.
* *note PyBUF_C_CONTIGUOUS: 1387. and *note PyBUF_F_CONTIGUOUS: 1388.
requests a C-contiguous (last dimension varies the fastest) or
Fortran-contiguous (first dimension varies the fastest) array
layout.
Two new argument codes for *note PyArg_ParseTuple(): 56e, ‘s*’ and ‘z*’,
return locked buffer objects for a parameter.
See also
........
PEP 3118(1) - Revising the buffer protocol
PEP written by Travis Oliphant and Carl Banks; implemented by
Travis Oliphant.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3118/
File: python.info, Node: PEP 3119 Abstract Base Classes, Next: PEP 3127 Integer Literal Support and Syntax, Prev: PEP 3118 Revised Buffer Protocol, Up: What’s New in Python 2 6
1.16.13 PEP 3119: Abstract Base Classes
---------------------------------------
Some object-oriented languages such as Java support interfaces,
declaring that a class has a given set of methods or supports a given
access protocol. Abstract Base Classes (or ABCs) are an equivalent
feature for Python. The ABC support consists of an *note abc: 4. module
containing a metaclass called ‘ABCMeta’, special handling of this
metaclass by the *note isinstance(): 43d. and *note issubclass(): 7bc.
builtins, and a collection of basic ABCs that the Python developers
think will be widely useful. Future versions of Python will probably
add more ABCs.
Let’s say you have a particular class and wish to know whether it
supports dictionary-style access. The phrase “dictionary-style” is
vague, however. It probably means that accessing items with ‘obj[1]’
works. Does it imply that setting items with ‘obj[2] = value’ works?
Or that the object will have ‘keys()’, ‘values()’, and ‘items()’
methods? What about the iterative variants such as ‘iterkeys()’? *note
copy(): 25. and ‘update()’? Iterating over the object with *note
iter(): 7d2.?
The Python 2.6 *note collections: 1d. module includes a number of
different ABCs that represent these distinctions. ‘Iterable’ indicates
that a class defines ‘__iter__()’, and ‘Container’ means the class
defines a ‘__contains__()’ method and therefore supports ‘x in y’
expressions. The basic dictionary interface of getting items, setting
items, and ‘keys()’, ‘values()’, and ‘items()’, is defined by the
‘MutableMapping’ ABC.
You can derive your own classes from a particular ABC to indicate they
support that ABC’s interface:
import collections
class Storage(collections.MutableMapping):
...
Alternatively, you could write the class without deriving from the
desired ABC and instead register the class by calling the ABC’s
‘register()’ method:
import collections
class Storage:
...
collections.MutableMapping.register(Storage)
For classes that you write, deriving from the ABC is probably clearer.
The ‘register()’ method is useful when you’ve written a new ABC that can
describe an existing type or class, or if you want to declare that some
third-party class implements an ABC. For example, if you defined a
‘PrintableType’ ABC, it’s legal to do:
# Register Python's types
PrintableType.register(int)
PrintableType.register(float)
PrintableType.register(str)
Classes should obey the semantics specified by an ABC, but Python can’t
check this; it’s up to the class author to understand the ABC’s
requirements and to implement the code accordingly.
To check whether an object supports a particular interface, you can now
write:
def func(d):
if not isinstance(d, collections.MutableMapping):
raise ValueError("Mapping object expected, not %r" % d)
Don’t feel that you must now begin writing lots of checks as in the
above example. Python has a strong tradition of duck-typing, where
explicit type-checking is never done and code simply calls methods on an
object, trusting that those methods will be there and raising an
exception if they aren’t. Be judicious in checking for ABCs and only do
it where it’s absolutely necessary.
You can write your own ABCs by using ‘abc.ABCMeta’ as the metaclass in a
class definition:
from abc import ABCMeta, abstractmethod
class Drawable():
__metaclass__ = ABCMeta
@abstractmethod
def draw(self, x, y, scale=1.0):
pass
def draw_doubled(self, x, y):
self.draw(x, y, scale=2.0)
class Square(Drawable):
def draw(self, x, y, scale):
...
In the ‘Drawable’ ABC above, the ‘draw_doubled()’ method renders the
object at twice its size and can be implemented in terms of other
methods described in ‘Drawable’. Classes implementing this ABC
therefore don’t need to provide their own implementation of
‘draw_doubled()’, though they can do so. An implementation of ‘draw()’
is necessary, though; the ABC can’t provide a useful generic
implementation.
You can apply the ‘@abstractmethod’ decorator to methods such as
‘draw()’ that must be implemented; Python will then raise an exception
for classes that don’t define the method. Note that the exception is
only raised when you actually try to create an instance of a subclass
lacking the method:
>>> class Circle(Drawable):
... pass
...
>>> c = Circle()
Traceback (most recent call last):
File "", line 1, in
TypeError: Can't instantiate abstract class Circle with abstract methods draw
>>>
Abstract data attributes can be declared using the ‘@abstractproperty’
decorator:
from abc import abstractproperty
...
@abstractproperty
def readonly(self):
return self._x
Subclasses must then define a ‘readonly()’ property.
See also
........
PEP 3119(1) - Introducing Abstract Base Classes
PEP written by Guido van Rossum and Talin. Implemented by Guido
van Rossum. Backported to 2.6 by Benjamin Aranguren, with Alex
Martelli.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3119/
File: python.info, Node: PEP 3127 Integer Literal Support and Syntax, Next: PEP 3129 Class Decorators, Prev: PEP 3119 Abstract Base Classes, Up: What’s New in Python 2 6
1.16.14 PEP 3127: Integer Literal Support and Syntax
----------------------------------------------------
Python 3.0 changes the syntax for octal (base-8) integer literals,
prefixing them with “0o” or “0O” instead of a leading zero, and adds
support for binary (base-2) integer literals, signalled by a “0b” or
“0B” prefix.
Python 2.6 doesn’t drop support for a leading 0 signalling an octal
number, but it does add support for “0o” and “0b”:
>>> 0o21, 2*8 + 1
(17, 17)
>>> 0b101111
47
The *note oct(): 12c1. builtin still returns numbers prefixed with a
leading zero, and a new *note bin(): 129b. builtin returns the binary
representation for a number:
>>> oct(42)
'052'
>>> future_builtins.oct(42)
'0o52'
>>> bin(173)
'0b10101101'
The *note int(): 259. and ‘long()’ builtins will now accept the “0o” and
“0b” prefixes when base-8 or base-2 are requested, or when the 'base'
argument is zero (signalling that the base used should be determined
from the string):
>>> int ('0o52', 0)
42
>>> int('1101', 2)
13
>>> int('0b1101', 2)
13
>>> int('0b1101', 0)
13
See also
........
PEP 3127(1) - Integer Literal Support and Syntax
PEP written by Patrick Maupin; backported to 2.6 by Eric Smith.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3127/
File: python.info, Node: PEP 3129 Class Decorators, Next: PEP 3141 A Type Hierarchy for Numbers, Prev: PEP 3127 Integer Literal Support and Syntax, Up: What’s New in Python 2 6
1.16.15 PEP 3129: Class Decorators
----------------------------------
Decorators have been extended from functions to classes. It’s now legal
to write:
@foo
@bar
class A:
pass
This is equivalent to:
class A:
pass
A = foo(bar(A))
See also
........
PEP 3129(1) - Class Decorators
PEP written by Collin Winter.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3129/
File: python.info, Node: PEP 3141 A Type Hierarchy for Numbers, Next: Other Language Changes<15>, Prev: PEP 3129 Class Decorators, Up: What’s New in Python 2 6
1.16.16 PEP 3141: A Type Hierarchy for Numbers
----------------------------------------------
Python 3.0 adds several abstract base classes for numeric types inspired
by Scheme’s numeric tower. These classes were backported to 2.6 as the
*note numbers: 9e. module.
The most general ABC is ‘Number’. It defines no operations at all, and
only exists to allow checking if an object is a number by doing
‘isinstance(obj, Number)’.
‘Complex’ is a subclass of ‘Number’. Complex numbers can undergo the
basic operations of addition, subtraction, multiplication, division, and
exponentiation, and you can retrieve the real and imaginary parts and
obtain a number’s conjugate. Python’s built-in complex type is an
implementation of ‘Complex’.
‘Real’ further derives from ‘Complex’, and adds operations that only
work on real numbers: ‘floor()’, ‘trunc()’, rounding, taking the
remainder mod N, floor division, and comparisons.
‘Rational’ numbers derive from ‘Real’, have ‘numerator’ and
‘denominator’ properties, and can be converted to floats. Python 2.6
adds a simple rational-number class, ‘Fraction’, in the *note fractions:
5d. module. (It’s called ‘Fraction’ instead of ‘Rational’ to avoid a
name clash with *note numbers.Rational: 12b1.)
‘Integral’ numbers derive from ‘Rational’, and can be shifted left and
right with ‘<<’ and ‘>>’, combined using bitwise operations such as ‘&’
and ‘|’, and can be used as array indexes and slice boundaries.
In Python 3.0, the PEP slightly redefines the existing builtins *note
round(): 12cd, *note math.floor(): 138d, *note math.ceil(): 138e, and
adds a new one, *note math.trunc(): 138f, that’s been backported to
Python 2.6. *note math.trunc(): 138f. rounds toward zero, returning the
closest ‘Integral’ that’s between the function’s argument and zero.
See also
........
PEP 3141(1) - A Type Hierarchy for Numbers
PEP written by Jeffrey Yasskin.
Scheme's numerical tower(2), from the Guile manual.
Scheme's number datatypes(3) from the R5RS Scheme specification.
* Menu:
* The fractions Module::
---------- Footnotes ----------
(1) https://peps.python.org/pep-3141/
(2)
https://www.gnu.org/software/guile/manual/html_node/Numerical-Tower.html#Numerical-Tower
(3)
https://conservatory.scheme.org/schemers/Documents/Standards/R5RS/HTML/r5rs-Z-H-9.html#%_sec_6.2
File: python.info, Node: The fractions Module, Up: PEP 3141 A Type Hierarchy for Numbers
1.16.16.1 The ‘fractions’ Module
................................
To fill out the hierarchy of numeric types, the *note fractions: 5d.
module provides a rational-number class. Rational numbers store their
values as a numerator and denominator forming a fraction, and can
exactly represent numbers such as ‘2/3’ that floating-point numbers can
only approximate.
The ‘Fraction’ constructor takes two ‘Integral’ values that will be the
numerator and denominator of the resulting fraction.
>>> from fractions import Fraction
>>> a = Fraction(2, 3)
>>> b = Fraction(2, 5)
>>> float(a), float(b)
(0.66666666666666663, 0.40000000000000002)
>>> a+b
Fraction(16, 15)
>>> a/b
Fraction(5, 3)
For converting floating-point numbers to rationals, the float type now
has an ‘as_integer_ratio()’ method that returns the numerator and
denominator for a fraction that evaluates to the same floating-point
value:
>>> (2.5) .as_integer_ratio()
(5, 2)
>>> (3.1415) .as_integer_ratio()
(7074029114692207L, 2251799813685248L)
>>> (1./3) .as_integer_ratio()
(6004799503160661L, 18014398509481984L)
Note that values that can only be approximated by floating-point
numbers, such as 1./3, are not simplified to the number being
approximated; the fraction attempts to match the floating-point value
'exactly'.
The *note fractions: 5d. module is based upon an implementation by
Sjoerd Mullender that was in Python’s ‘Demo/classes/’ directory for a
long time. This implementation was significantly updated by Jeffrey
Yasskin.
File: python.info, Node: Other Language Changes<15>, Next: New and Improved Modules<2>, Prev: PEP 3141 A Type Hierarchy for Numbers, Up: What’s New in Python 2 6
1.16.17 Other Language Changes
------------------------------
Some smaller changes made to the core Python language are:
* Directories and zip archives containing a ‘__main__.py’ file can
now be executed directly by passing their name to the interpreter.
The directory or zip archive is automatically inserted as the first
entry in sys.path. (Suggestion and initial patch by Andy Chu,
subsequently revised by Phillip J. Eby and Nick Coghlan;
bpo-1739468(1).)
* The *note hasattr(): 4ce. function was catching and ignoring all
errors, under the assumption that they meant a ‘__getattr__()’
method was failing somehow and the return value of *note hasattr():
4ce. would therefore be ‘False’. This logic shouldn’t be applied
to *note KeyboardInterrupt: 9d1. and *note SystemExit: d40,
however; Python 2.6 will no longer discard such exceptions when
*note hasattr(): 4ce. encounters them. (Fixed by Benjamin
Peterson; bpo-2196(2).)
* When calling a function using the ‘**’ syntax to provide keyword
arguments, you are no longer required to use a Python dictionary;
any mapping will now work:
>>> def f(**kw):
... print sorted(kw)
...
>>> ud=UserDict.UserDict()
>>> ud['a'] = 1
>>> ud['b'] = 'string'
>>> f(**ud)
['a', 'b']
(Contributed by Alexander Belopolsky; bpo-1686487(3).)
It’s also become legal to provide keyword arguments after a ‘*args’
argument to a function call.
>>> def f(*args, **kw):
... print args, kw
...
>>> f(1,2,3, *(4,5,6), keyword=13)
(1, 2, 3, 4, 5, 6) {'keyword': 13}
Previously this would have been a syntax error. (Contributed by
Amaury Forgeot d’Arc; bpo-3473(4).)
* A new builtin, ‘next(iterator, [default])’ returns the next item
from the specified iterator. If the 'default' argument is
supplied, it will be returned if 'iterator' has been exhausted;
otherwise, the *note StopIteration: bfa. exception will be raised.
(Backported in bpo-2719(5).)
* Tuples now have ‘index()’ and ‘count()’ methods matching the list
type’s ‘index()’ and ‘count()’ methods:
>>> t = (0,1,2,3,4,0,1,2)
>>> t.index(3)
3
>>> t.count(0)
2
(Contributed by Raymond Hettinger)
* The built-in types now have improved support for extended slicing
syntax, accepting various combinations of ‘(start, stop, step)’.
Previously, the support was partial and certain corner cases
wouldn’t work. (Implemented by Thomas Wouters.)
* Properties now have three attributes, ‘getter’, ‘setter’ and
‘deleter’, that are decorators providing useful shortcuts for
adding a getter, setter or deleter function to an existing
property. You would use them like this:
class C(object):
@property
def x(self):
return self._x
@x.setter
def x(self, value):
self._x = value
@x.deleter
def x(self):
del self._x
class D(C):
@C.x.getter
def x(self):
return self._x * 2
@x.setter
def x(self, value):
self._x = value / 2
* Several methods of the built-in set types now accept multiple
iterables: ‘intersection()’, ‘intersection_update()’, ‘union()’,
‘update()’, ‘difference()’ and ‘difference_update()’.
>>> s=set('1234567890')
>>> s.intersection('abc123', 'cdf246') # Intersection between all inputs
set(['2'])
>>> s.difference('246', '789')
set(['1', '0', '3', '5'])
(Contributed by Raymond Hettinger.)
* Many floating-point features were added. The *note float(): 2f1.
function will now turn the string ‘nan’ into an IEEE 754 Not A
Number value, and ‘+inf’ and ‘-inf’ into positive or negative
infinity. This works on any platform with IEEE 754 semantics.
(Contributed by Christian Heimes; bpo-1635(6).)
Other functions in the *note math: 8e. module, ‘isinf()’ and
‘isnan()’, return true if their floating-point argument is infinite
or Not A Number. (bpo-1640(7))
Conversion functions were added to convert floating-point numbers
into hexadecimal strings (bpo-3008(8)). These functions convert
floats to and from a string representation without introducing
rounding errors from the conversion between decimal and binary.
Floats have a *note hex(): 12c2. method that returns a string
representation, and the ‘float.fromhex()’ method converts a string
back into a number:
>>> a = 3.75
>>> a.hex()
'0x1.e000000000000p+1'
>>> float.fromhex('0x1.e000000000000p+1')
3.75
>>> b=1./3
>>> b.hex()
'0x1.5555555555555p-2'
* A numerical nicety: when creating a complex number from two floats
on systems that support signed zeros (-0 and +0), the *note
complex(): 2f2. constructor will now preserve the sign of the zero.
(Fixed by Mark T. Dickinson; bpo-1507(9).)
* Classes that inherit a ‘__hash__()’ method from a parent class can
set ‘__hash__ = None’ to indicate that the class isn’t hashable.
This will make ‘hash(obj)’ raise a *note TypeError: 534. and the
class will not be indicated as implementing the ‘Hashable’ ABC.
You should do this when you’ve defined a ‘__cmp__()’ or ‘__eq__()’
method that compares objects by their value rather than by
identity. All objects have a default hash method that uses
‘id(obj)’ as the hash value. There’s no tidy way to remove the
‘__hash__()’ method inherited from a parent class, so assigning
‘None’ was implemented as an override. At the C level, extensions
can set ‘tp_hash’ to *note PyObject_HashNotImplemented(): 1392.
(Fixed by Nick Coghlan and Amaury Forgeot d’Arc; bpo-2235(10).)
* The *note GeneratorExit: 1393. exception now subclasses *note
BaseException: 5b7. instead of *note Exception: 9d9. This means
that an exception handler that does ‘except Exception:’ will not
inadvertently catch *note GeneratorExit: 1393. (Contributed by
Chad Austin; bpo-1537(11).)
* Generator objects now have a ‘gi_code’ attribute that refers to the
original code object backing the generator. (Contributed by Collin
Winter; bpo-1473257(12).)
* The *note compile(): 192. built-in function now accepts keyword
arguments as well as positional parameters. (Contributed by Thomas
Wouters; bpo-1444529(13).)
* The *note complex(): 2f2. constructor now accepts strings
containing parenthesized complex numbers, meaning that
‘complex(repr(cplx))’ will now round-trip values. For example,
‘complex('(3+4j)')’ now returns the value (3+4j).
(bpo-1491866(14))
* The string ‘translate()’ method now accepts ‘None’ as the
translation table parameter, which is treated as the identity
transformation. This makes it easier to carry out operations that
only delete characters. (Contributed by Bengt Richter and
implemented by Raymond Hettinger; bpo-1193128(15).)
* The built-in *note dir(): 62e. function now checks for a
‘__dir__()’ method on the objects it receives. This method must
return a list of strings containing the names of valid attributes
for the object, and lets the object control the value that *note
dir(): 62e. produces. Objects that have ‘__getattr__()’ or
‘__getattribute__()’ methods can use this to advertise
pseudo-attributes they will honor. (bpo-1591665(16))
* Instance method objects have new attributes for the object and
function comprising the method; the new synonym for ‘im_self’ is
*note __self__: 1394, and ‘im_func’ is also available as *note
__func__: 12ef. The old names are still supported in Python 2.6,
but are gone in 3.0.
* An obscure change: when you use the *note locals(): 141. function
inside a *note class: 12ca. statement, the resulting dictionary no
longer returns free variables. (Free variables, in this case, are
variables referenced in the ‘class’ statement that aren’t
attributes of the class.)
* Menu:
* Optimizations: Optimizations<14>.
* Interpreter Changes: Interpreter Changes<2>.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1739468
(2) https://bugs.python.org/issue?@action=redirect&bpo=2196
(3) https://bugs.python.org/issue?@action=redirect&bpo=1686487
(4) https://bugs.python.org/issue?@action=redirect&bpo=3473
(5) https://bugs.python.org/issue?@action=redirect&bpo=2719
(6) https://bugs.python.org/issue?@action=redirect&bpo=1635
(7) https://bugs.python.org/issue?@action=redirect&bpo=1640
(8) https://bugs.python.org/issue?@action=redirect&bpo=3008
(9) https://bugs.python.org/issue?@action=redirect&bpo=1507
(10) https://bugs.python.org/issue?@action=redirect&bpo=2235
(11) https://bugs.python.org/issue?@action=redirect&bpo=1537
(12) https://bugs.python.org/issue?@action=redirect&bpo=1473257
(13) https://bugs.python.org/issue?@action=redirect&bpo=1444529
(14) https://bugs.python.org/issue?@action=redirect&bpo=1491866
(15) https://bugs.python.org/issue?@action=redirect&bpo=1193128
(16) https://bugs.python.org/issue?@action=redirect&bpo=1591665
File: python.info, Node: Optimizations<14>, Next: Interpreter Changes<2>, Up: Other Language Changes<15>
1.16.17.1 Optimizations
.......................
* The *note warnings: 112. module has been rewritten in C. This makes
it possible to invoke warnings from the parser, and may also make
the interpreter’s startup faster. (Contributed by Neal Norwitz and
Brett Cannon; bpo-1631171(1).)
* Type objects now have a cache of methods that can reduce the work
required to find the correct method implementation for a particular
class; once cached, the interpreter doesn’t need to traverse base
classes to figure out the right method to call. The cache is
cleared if a base class or the class itself is modified, so the
cache should remain correct even in the face of Python’s dynamic
nature. (Original optimization implemented by Armin Rigo, updated
for Python 2.6 by Kevin Jacobs; bpo-1700288(2).)
By default, this change is only applied to types that are included
with the Python core. Extension modules may not necessarily be
compatible with this cache, so they must explicitly add
‘Py_TPFLAGS_HAVE_VERSION_TAG’ to the module’s ‘tp_flags’ field to
enable the method cache. (To be compatible with the method cache,
the extension module’s code must not directly access and modify the
‘tp_dict’ member of any of the types it implements. Most modules
don’t do this, but it’s impossible for the Python interpreter to
determine that. See bpo-1878(3) for some discussion.)
* Function calls that use keyword arguments are significantly faster
by doing a quick pointer comparison, usually saving the time of a
full string comparison. (Contributed by Raymond Hettinger, after
an initial implementation by Antoine Pitrou; bpo-1819(4).)
* All of the functions in the *note struct: d5. module have been
rewritten in C, thanks to work at the Need For Speed sprint.
(Contributed by Raymond Hettinger.)
* Some of the standard built-in types now set a bit in their type
objects. This speeds up checking whether an object is a subclass
of one of these types. (Contributed by Neal Norwitz.)
* Unicode strings now use faster code for detecting whitespace and
line breaks; this speeds up the ‘split()’ method by about 25% and
‘splitlines()’ by 35%. (Contributed by Antoine Pitrou.) Memory
usage is reduced by using pymalloc for the Unicode string’s data.
* The ‘with’ statement now stores the *note __exit__(): 12f3. method
on the stack, producing a small speedup. (Implemented by Jeffrey
Yasskin.)
* To reduce memory usage, the garbage collector will now clear
internal free lists when garbage-collecting the highest generation
of objects. This may return memory to the operating system sooner.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1631171
(2) https://bugs.python.org/issue?@action=redirect&bpo=1700288
(3) https://bugs.python.org/issue?@action=redirect&bpo=1878
(4) https://bugs.python.org/issue?@action=redirect&bpo=1819
File: python.info, Node: Interpreter Changes<2>, Prev: Optimizations<14>, Up: Other Language Changes<15>
1.16.17.2 Interpreter Changes
.............................
Two command-line options have been reserved for use by other Python
implementations. The *note -J: 1398. switch has been reserved for use
by Jython for Jython-specific options, such as switches that are passed
to the underlying JVM. *note -X: 176. has been reserved for options
specific to a particular implementation of Python such as CPython,
Jython, or IronPython. If either option is used with Python 2.6, the
interpreter will report that the option isn’t currently used.
Python can now be prevented from writing ‘.pyc’ or ‘.pyo’ files by
supplying the *note -B: 1399. switch to the Python interpreter, or by
setting the *note PYTHONDONTWRITEBYTECODE: 139a. environment variable
before running the interpreter. This setting is available to Python
programs as the ‘sys.dont_write_bytecode’ variable, and Python code can
change the value to modify the interpreter’s behaviour. (Contributed by
Neal Norwitz and Georg Brandl.)
The encoding used for standard input, output, and standard error can be
specified by setting the *note PYTHONIOENCODING: 1003. environment
variable before running the interpreter. The value should be a string
in the form ‘’ or ‘:’. The 'encoding'
part specifies the encoding’s name, e.g. ‘utf-8’ or ‘latin-1’; the
optional 'errorhandler' part specifies what to do with characters that
can’t be handled by the encoding, and should be one of “error”,
“ignore”, or “replace”. (Contributed by Martin von Löwis.)
File: python.info, Node: New and Improved Modules<2>, Next: Deprecations and Removals, Prev: Other Language Changes<15>, Up: What’s New in Python 2 6
1.16.18 New and Improved Modules
--------------------------------
As in every release, Python’s standard library received a number of
enhancements and bug fixes. Here’s a partial list of the most notable
changes, sorted alphabetically by module name. Consult the ‘Misc/NEWS’
file in the source tree for a more complete list of changes, or look
through the Subversion logs for all the details.
* The ‘asyncore’ and ‘asynchat’ modules are being actively maintained
again, and a number of patches and bugfixes were applied.
(Maintained by Josiah Carlson; see bpo-1736190(1) for one patch.)
* The ‘bsddb’ module also has a new maintainer, Jesús Cea Avión, and
the package is now available as a standalone package. The web page
for the package is www.jcea.es/programacion/pybsddb.htm(2). The
plan is to remove the package from the standard library in Python
3.0, because its pace of releases is much more frequent than
Python’s.
The ‘bsddb.dbshelve’ module now uses the highest pickling protocol
available, instead of restricting itself to protocol 1.
(Contributed by W. Barnes.)
* The ‘cgi’ module will now read variables from the query string of
an HTTP POST request. This makes it possible to use form actions
with URLs that include query strings such as
“/cgi-bin/add.py?category=1”. (Contributed by Alexandre Fiori and
Nubis; bpo-1817(3).)
The ‘parse_qs()’ and ‘parse_qsl()’ functions have been relocated
from the ‘cgi’ module to the *note urlparse: 10a. module. The
versions still available in the ‘cgi’ module will trigger *note
PendingDeprecationWarning: 8c7. messages in 2.6 (bpo-600362(4)).
* The *note cmath: 18. module underwent extensive revision,
contributed by Mark Dickinson and Christian Heimes. Five new
functions were added:
* ‘polar()’ converts a complex number to polar form, returning
the modulus and argument of the complex number.
* ‘rect()’ does the opposite, turning a modulus, argument pair
back into the corresponding complex number.
* ‘phase()’ returns the argument (also called the angle) of a
complex number.
* ‘isnan()’ returns True if either the real or imaginary part of
its argument is a NaN.
* ‘isinf()’ returns True if either the real or imaginary part of
its argument is infinite.
The revisions also improved the numerical soundness of the *note
cmath: 18. module. For all functions, the real and imaginary parts
of the results are accurate to within a few units of least
precision (ulps) whenever possible. See bpo-1381(5) for the
details. The branch cuts for ‘asinh()’, ‘atanh()’: and ‘atan()’
have also been corrected.
The tests for the module have been greatly expanded; nearly 2000
new test cases exercise the algebraic functions.
On IEEE 754 platforms, the *note cmath: 18. module now handles IEEE
754 special values and floating-point exceptions in a manner
consistent with Annex ‘G’ of the C99 standard.
* A new data type in the *note collections: 1d. module:
‘namedtuple(typename, fieldnames)’ is a factory function that
creates subclasses of the standard tuple whose fields are
accessible by name as well as index. For example:
>>> var_type = collections.namedtuple('variable',
... 'id name type size')
>>> # Names are separated by spaces or commas.
>>> # 'id, name, type, size' would also work.
>>> var_type._fields
('id', 'name', 'type', 'size')
>>> var = var_type(1, 'frequency', 'int', 4)
>>> print var[0], var.id # Equivalent
1 1
>>> print var[2], var.type # Equivalent
int int
>>> var._asdict()
{'size': 4, 'type': 'int', 'id': 1, 'name': 'frequency'}
>>> v2 = var._replace(name='amplitude')
>>> v2
variable(id=1, name='amplitude', type='int', size=4)
Several places in the standard library that returned tuples have
been modified to return ‘namedtuple()’ instances. For example, the
‘Decimal.as_tuple()’ method now returns a named tuple with ‘sign’,
‘digits’, and ‘exponent’ fields.
(Contributed by Raymond Hettinger.)
* Another change to the *note collections: 1d. module is that the
‘deque’ type now supports an optional 'maxlen' parameter; if
supplied, the deque’s size will be restricted to no more than
'maxlen' items. Adding more items to a full deque causes old items
to be discarded.
>>> from collections import deque
>>> dq=deque(maxlen=3)
>>> dq
deque([], maxlen=3)
>>> dq.append(1); dq.append(2); dq.append(3)
>>> dq
deque([1, 2, 3], maxlen=3)
>>> dq.append(4)
>>> dq
deque([2, 3, 4], maxlen=3)
(Contributed by Raymond Hettinger.)
* The *note Cookie: 71. module’s *note Morsel: c04. objects now
support an *note httponly: 139c. attribute. In some browsers.
cookies with this attribute set cannot be accessed or manipulated
by JavaScript code. (Contributed by Arvin Schnell;
bpo-1638033(6).)
* A new window method in the *note curses: 2b. module, ‘chgat()’,
changes the display attributes for a certain number of characters
on a single line. (Contributed by Fabian Kreutz.)
# Boldface text starting at y=0,x=21
# and affecting the rest of the line.
stdscr.chgat(0, 21, curses.A_BOLD)
The ‘Textbox’ class in the *note curses.textpad: 2e. module now
supports editing in insert mode as well as overwrite mode. Insert
mode is enabled by supplying a true value for the 'insert_mode'
parameter when creating the ‘Textbox’ instance.
* The *note datetime: 30. module’s ‘strftime()’ methods now support a
‘%f’ format code that expands to the number of microseconds in the
object, zero-padded on the left to six places. (Contributed by
Skip Montanaro; bpo-1158(7).)
* The *note decimal: 36. module was updated to version 1.66 of the
General Decimal Specification(8). New features include some
methods for some basic mathematical functions such as ‘exp()’ and
‘log10()’:
>>> Decimal(1).exp()
Decimal("2.718281828459045235360287471")
>>> Decimal("2.7182818").ln()
Decimal("0.9999999895305022877376682436")
>>> Decimal(1000).log10()
Decimal("3")
The ‘as_tuple()’ method of ‘Decimal’ objects now returns a named
tuple with ‘sign’, ‘digits’, and ‘exponent’ fields.
(Implemented by Facundo Batista and Mark Dickinson. Named tuple
support added by Raymond Hettinger.)
* The *note difflib: 37. module’s ‘SequenceMatcher’ class now returns
named tuples representing matches, with ‘a’, ‘b’, and ‘size’
attributes. (Contributed by Raymond Hettinger.)
* An optional ‘timeout’ parameter, specifying a timeout measured in
seconds, was added to the *note ftplib.FTP: 8f9. class constructor
as well as the ‘connect()’ method. (Added by Facundo Batista.)
Also, the ‘FTP’ class’s ‘storbinary()’ and ‘storlines()’ now take
an optional 'callback' parameter that will be called with each
block of data after the data has been sent. (Contributed by Phil
Schwartz; bpo-1221598(9).)
* The ‘reduce()’ built-in function is also available in the *note
functools: 5f. module. In Python 3.0, the builtin has been dropped
and ‘reduce()’ is only available from *note functools: 5f.;
currently there are no plans to drop the builtin in the 2.x series.
(Patched by Christian Heimes; bpo-1739906(10).)
* When possible, the *note getpass: 62. module will now use
‘/dev/tty’ to print a prompt message and read the password, falling
back to standard error and standard input. If the password may be
echoed to the terminal, a warning is printed before the prompt is
displayed. (Contributed by Gregory P. Smith.)
* The *note glob.glob(): 2a1. function can now return Unicode
filenames if a Unicode path was used and Unicode filenames are
matched within the directory. (bpo-1001604(11))
* A new function in the *note heapq: 69. module, ‘merge(iter1, iter2,
...)’, takes any number of iterables returning data in sorted
order, and returns a new generator that returns the contents of all
the iterators, also in sorted order. For example:
>>> list(heapq.merge([1, 3, 5, 9], [2, 8, 16]))
[1, 2, 3, 5, 8, 9, 16]
Another new function, ‘heappushpop(heap, item)’, pushes 'item' onto
'heap', then pops off and returns the smallest item. This is more
efficient than making a call to ‘heappush()’ and then ‘heappop()’.
*note heapq: 69. is now implemented to only use less-than
comparison, instead of the less-than-or-equal comparison it
previously used. This makes *note heapq: 69.’s usage of a type
match the *note list.sort(): bcf. method. (Contributed by Raymond
Hettinger.)
* An optional ‘timeout’ parameter, specifying a timeout measured in
seconds, was added to the *note httplib.HTTPConnection: b40. and
*note HTTPSConnection: b41. class constructors. (Added by Facundo
Batista.)
* Most of the *note inspect: 7e. module’s functions, such as
‘getmoduleinfo()’ and ‘getargs()’, now return named tuples. In
addition to behaving like tuples, the elements of the return value
can also be accessed as attributes. (Contributed by Raymond
Hettinger.)
Some new functions in the module include ‘isgenerator()’,
‘isgeneratorfunction()’, and ‘isabstract()’.
* The *note itertools: 81. module gained several new functions.
‘izip_longest(iter1, iter2, ...[, fillvalue])’ makes tuples from
each of the elements; if some of the iterables are shorter than
others, the missing values are set to 'fillvalue'. For example:
>>> tuple(itertools.izip_longest([1,2,3], [1,2,3,4,5]))
((1, 1), (2, 2), (3, 3), (None, 4), (None, 5))
‘product(iter1, iter2, ..., [repeat=N])’ returns the Cartesian
product of the supplied iterables, a set of tuples containing every
possible combination of the elements returned from each iterable.
>>> list(itertools.product([1,2,3], [4,5,6]))
[(1, 4), (1, 5), (1, 6),
(2, 4), (2, 5), (2, 6),
(3, 4), (3, 5), (3, 6)]
The optional 'repeat' keyword argument is used for taking the
product of an iterable or a set of iterables with themselves,
repeated 'N' times. With a single iterable argument, 'N'-tuples
are returned:
>>> list(itertools.product([1,2], repeat=3))
[(1, 1, 1), (1, 1, 2), (1, 2, 1), (1, 2, 2),
(2, 1, 1), (2, 1, 2), (2, 2, 1), (2, 2, 2)]
With two iterables, '2N'-tuples are returned.
>>> list(itertools.product([1,2], [3,4], repeat=2))
[(1, 3, 1, 3), (1, 3, 1, 4), (1, 3, 2, 3), (1, 3, 2, 4),
(1, 4, 1, 3), (1, 4, 1, 4), (1, 4, 2, 3), (1, 4, 2, 4),
(2, 3, 1, 3), (2, 3, 1, 4), (2, 3, 2, 3), (2, 3, 2, 4),
(2, 4, 1, 3), (2, 4, 1, 4), (2, 4, 2, 3), (2, 4, 2, 4)]
‘combinations(iterable, r)’ returns sub-sequences of length 'r'
from the elements of 'iterable'.
>>> list(itertools.combinations('123', 2))
[('1', '2'), ('1', '3'), ('2', '3')]
>>> list(itertools.combinations('123', 3))
[('1', '2', '3')]
>>> list(itertools.combinations('1234', 3))
[('1', '2', '3'), ('1', '2', '4'),
('1', '3', '4'), ('2', '3', '4')]
‘permutations(iter[, r])’ returns all the permutations of length
'r' of the iterable’s elements. If 'r' is not specified, it will
default to the number of elements produced by the iterable.
>>> list(itertools.permutations([1,2,3,4], 2))
[(1, 2), (1, 3), (1, 4),
(2, 1), (2, 3), (2, 4),
(3, 1), (3, 2), (3, 4),
(4, 1), (4, 2), (4, 3)]
‘itertools.chain(*iterables)’ is an existing function in *note
itertools: 81. that gained a new constructor in Python 2.6.
‘itertools.chain.from_iterable(iterable)’ takes a single iterable
that should return other iterables. ‘chain()’ will then return all
the elements of the first iterable, then all the elements of the
second, and so on.
>>> list(itertools.chain.from_iterable([[1,2,3], [4,5,6]]))
[1, 2, 3, 4, 5, 6]
(All contributed by Raymond Hettinger.)
* The *note logging: 87. module’s ‘FileHandler’ class and its
subclasses ‘WatchedFileHandler’, ‘RotatingFileHandler’, and
‘TimedRotatingFileHandler’ now have an optional 'delay' parameter
to their constructors. If 'delay' is true, opening of the log file
is deferred until the first ‘emit()’ call is made. (Contributed by
Vinay Sajip.)
‘TimedRotatingFileHandler’ also has a 'utc' constructor parameter.
If the argument is true, UTC time will be used in determining when
midnight occurs and in generating filenames; otherwise local time
will be used.
* Several new functions were added to the *note math: 8e. module:
* *note isinf(): 139d. and *note isnan(): 139e. determine
whether a given float is a (positive or negative) infinity or
a NaN (Not a Number), respectively.
* *note copysign(): 139f. copies the sign bit of an IEEE 754
number, returning the absolute value of 'x' combined with the
sign bit of 'y'. For example, ‘math.copysign(1, -0.0)’
returns -1.0. (Contributed by Christian Heimes.)
* *note factorial(): 947. computes the factorial of a number.
(Contributed by Raymond Hettinger; bpo-2138(12).)
* *note fsum(): 13a0. adds up the stream of numbers from an
iterable, and is careful to avoid loss of precision through
using partial sums. (Contributed by Jean Brouwers, Raymond
Hettinger, and Mark Dickinson; bpo-2819(13).)
* *note acosh(): 13a1, *note asinh(): 13a2. and *note atanh():
13a3. compute the inverse hyperbolic functions.
* *note log1p(): 13a4. returns the natural logarithm of '1+x'
(base 'e').
* ‘trunc()’ rounds a number toward zero, returning the closest
‘Integral’ that’s between the function’s argument and zero.
Added as part of the backport of *note PEP 3141's type
hierarchy for numbers: 12b0.
* The *note math: 8e. module has been improved to give more
consistent behaviour across platforms, especially with respect to
handling of floating-point exceptions and IEEE 754 special values.
Whenever possible, the module follows the recommendations of the
C99 standard about 754’s special values. For example, ‘sqrt(-1.)’
should now give a *note ValueError: 204. across almost all
platforms, while ‘sqrt(float('NaN'))’ should return a NaN on all
IEEE 754 platforms. Where Annex ‘F’ of the C99 standard recommends
signaling ‘divide-by-zero’ or ‘invalid’, Python will raise *note
ValueError: 204. Where Annex ‘F’ of the C99 standard recommends
signaling ‘overflow’, Python will raise *note OverflowError: 87f.
(See bpo-711019(14) and bpo-1640(15).)
(Contributed by Christian Heimes and Mark Dickinson.)
* *note mmap: 20d. objects now have a ‘rfind()’ method that searches
for a substring beginning at the end of the string and searching
backwards. The ‘find()’ method also gained an 'end' parameter
giving an index at which to stop searching. (Contributed by John
Lenton.)
* The *note operator: 9f. module gained a ‘methodcaller()’ function
that takes a name and an optional set of arguments, returning a
callable that will call the named function on any arguments passed
to it. For example:
>>> # Equivalent to lambda s: s.replace('old', 'new')
>>> replacer = operator.methodcaller('replace', 'old', 'new')
>>> replacer('old wine in old bottles')
'new wine in new bottles'
(Contributed by Georg Brandl, after a suggestion by Gregory
Petrosyan.)
The ‘attrgetter()’ function now accepts dotted names and performs
the corresponding attribute lookups:
>>> inst_name = operator.attrgetter(
... '__class__.__name__')
>>> inst_name('')
'str'
>>> inst_name(help)
'_Helper'
(Contributed by Georg Brandl, after a suggestion by Barry Warsaw.)
* The *note os: a1. module now wraps several new system calls.
‘fchmod(fd, mode)’ and ‘fchown(fd, uid, gid)’ change the mode and
ownership of an opened file, and ‘lchmod(path, mode)’ changes the
mode of a symlink. (Contributed by Georg Brandl and Christian
Heimes.)
‘chflags()’ and ‘lchflags()’ are wrappers for the corresponding
system calls (where they’re available), changing the flags set on a
file. Constants for the flag values are defined in the *note stat:
d1. module; some possible values include ‘UF_IMMUTABLE’ to signal
the file may not be changed and ‘UF_APPEND’ to indicate that data
can only be appended to the file. (Contributed by M. Levinson.)
‘os.closerange(low, high)’ efficiently closes all file descriptors
from 'low' to 'high', ignoring any errors and not including 'high'
itself. This function is now used by the *note subprocess: d6.
module to make starting processes faster. (Contributed by Georg
Brandl; bpo-1663329(16).)
* The ‘os.environ’ object’s ‘clear()’ method will now unset the
environment variables using *note os.unsetenv(): 91a. in addition
to clearing the object’s keys. (Contributed by Martin Horcicka;
bpo-1181(17).)
* The *note os.walk(): 4a5. function now has a ‘followlinks’
parameter. If set to True, it will follow symlinks pointing to
directories and visit the directory’s contents. For backward
compatibility, the parameter’s default value is false. Note that
the function can fall into an infinite recursion if there’s a
symlink that points to a parent directory. (bpo-1273829(18))
* In the *note os.path: a2. module, the ‘splitext()’ function has
been changed to not split on leading period characters. This
produces better results when operating on Unix’s dot-files. For
example, ‘os.path.splitext('.ipython')’ now returns ‘('.ipython',
'')’ instead of ‘('', '.ipython')’. (bpo-1115886(19))
A new function, ‘os.path.relpath(path, start='.')’, returns a
relative path from the ‘start’ path, if it’s supplied, or from the
current working directory to the destination ‘path’. (Contributed
by Richard Barran; bpo-1339796(20).)
On Windows, *note os.path.expandvars(): 13a5. will now expand
environment variables given in the form “%var%”, and “~user” will
be expanded into the user’s home directory path. (Contributed by
Josiah Carlson; bpo-957650(21).)
* The Python debugger provided by the *note pdb: a5. module gained a
new command: “run” restarts the Python program being debugged and
can optionally take new command-line arguments for the program.
(Contributed by Rocky Bernstein; bpo-1393667(22).)
* The *note pdb.post_mortem(): 13a6. function, used to begin
debugging a traceback, will now use the traceback returned by *note
sys.exc_info(): 686. if no traceback is supplied. (Contributed by
Facundo Batista; bpo-1106316(23).)
* The *note pickletools: a7. module now has an ‘optimize()’ function
that takes a string containing a pickle and removes some unused
opcodes, returning a shorter pickle that contains the same data
structure. (Contributed by Raymond Hettinger.)
* A ‘get_data()’ function was added to the *note pkgutil: a9. module
that returns the contents of resource files included with an
installed Python package. For example:
>>> import pkgutil
>>> print pkgutil.get_data('test', 'exception_hierarchy.txt')
BaseException
+-- SystemExit
+-- KeyboardInterrupt
+-- GeneratorExit
+-- Exception
+-- StopIteration
+-- StandardError
...
(Contributed by Paul Moore; bpo-2439(24).)
* The ‘pyexpat’ module’s ‘Parser’ objects now allow setting their
‘buffer_size’ attribute to change the size of the buffer used to
hold character data. (Contributed by Achim Gaedke; bpo-1137(25).)
* The ‘Queue’ module now provides queue variants that retrieve
entries in different orders. The ‘PriorityQueue’ class stores
queued items in a heap and retrieves them in priority order, and
‘LifoQueue’ retrieves the most recently added entries first,
meaning that it behaves like a stack. (Contributed by Raymond
Hettinger.)
* The *note random: b8. module’s ‘Random’ objects can now be pickled
on a 32-bit system and unpickled on a 64-bit system, and vice
versa. Unfortunately, this change also means that Python 2.6’s
‘Random’ objects can’t be unpickled correctly on earlier versions
of Python. (Contributed by Shawn Ligocki; bpo-1727780(26).)
The new ‘triangular(low, high, mode)’ function returns random
numbers following a triangular distribution. The returned values
are between 'low' and 'high', not including 'high' itself, and with
'mode' as the most frequently occurring value in the distribution.
(Contributed by Wladmir van der Laan and Raymond Hettinger;
bpo-1681432(27).)
* Long regular expression searches carried out by the *note re: b9.
module will check for signals being delivered, so time-consuming
searches can now be interrupted. (Contributed by Josh Hoyt and
Ralf Schmitt; bpo-846388(28).)
The regular expression module is implemented by compiling bytecodes
for a tiny regex-specific virtual machine. Untrusted code could
create malicious strings of bytecode directly and cause crashes, so
Python 2.6 includes a verifier for the regex bytecode.
(Contributed by Guido van Rossum from work for Google App Engine;
bpo-3487(29).)
* The *note rlcompleter: bd. module’s ‘Completer.complete()’ method
will now ignore exceptions triggered while evaluating a name.
(Fixed by Lorenz Quack; bpo-2250(30).)
* The *note sched: bf. module’s ‘scheduler’ instances now have a
read-only *note queue: b6. attribute that returns the contents of
the scheduler’s queue, represented as a list of named tuples with
the fields ‘(time, priority, action, argument)’. (Contributed by
Raymond Hettinger; bpo-1861(31).)
* The *note select: c1. module now has wrapper functions for the
Linux ‘epoll()’ and BSD ‘kqueue()’ system calls. ‘modify()’ method
was added to the existing ‘poll’ objects; ‘pollobj.modify(fd,
eventmask)’ takes a file descriptor or file object and an event
mask, modifying the recorded event mask for that file.
(Contributed by Christian Heimes; bpo-1657(32).)
* The *note shutil.copytree(): a28. function now has an optional
'ignore' argument that takes a callable object. This callable will
receive each directory path and a list of the directory’s contents,
and returns a list of names that will be ignored, not copied.
The *note shutil: c5. module also provides an ‘ignore_patterns()’
function for use with this new parameter. ‘ignore_patterns()’
takes an arbitrary number of glob-style patterns and returns a
callable that will ignore any files and directories that match any
of these patterns. The following example copies a directory tree,
but skips both ‘.svn’ directories and Emacs backup files, which
have names ending with ‘~’:
shutil.copytree('Doc/library', '/tmp/library',
ignore=shutil.ignore_patterns('*~', '.svn'))
(Contributed by Tarek Ziadé; bpo-2663(33).)
* Integrating signal handling with GUI handling event loops like
those used by Tkinter or GTk+ has long been a problem; most
software ends up polling, waking up every fraction of a second to
check if any GUI events have occurred. The *note signal: c6.
module can now make this more efficient. Calling
‘signal.set_wakeup_fd(fd)’ sets a file descriptor to be used; when
a signal is received, a byte is written to that file descriptor.
There’s also a C-level function, *note PySignal_SetWakeupFd():
13a7, for setting the descriptor.
Event loops will use this by opening a pipe to create two
descriptors, one for reading and one for writing. The writable
descriptor will be passed to ‘set_wakeup_fd()’, and the readable
descriptor will be added to the list of descriptors monitored by
the event loop via ‘select()’ or ‘poll()’. On receiving a signal,
a byte will be written and the main event loop will be woken up,
avoiding the need to poll.
(Contributed by Adam Olsen; bpo-1583(34).)
The ‘siginterrupt()’ function is now available from Python code,
and allows changing whether signals can interrupt system calls or
not. (Contributed by Ralf Schmitt.)
The ‘setitimer()’ and ‘getitimer()’ functions have also been added
(where they’re available). ‘setitimer()’ allows setting interval
timers that will cause a signal to be delivered to the process
after a specified time, measured in wall-clock time, consumed
process time, or combined process+system time. (Contributed by
Guilherme Polo; bpo-2240(35).)
* The *note smtplib: ca. module now supports SMTP over SSL thanks to
the addition of the ‘SMTP_SSL’ class. This class supports an
interface identical to the existing ‘SMTP’ class. (Contributed by
Monty Taylor.) Both class constructors also have an optional
‘timeout’ parameter that specifies a timeout for the initial
connection attempt, measured in seconds. (Contributed by Facundo
Batista.)
An implementation of the LMTP protocol ( RFC 2033(36)) was also
added to the module. LMTP is used in place of SMTP when
transferring e-mail between agents that don’t manage a mail queue.
(LMTP implemented by Leif Hedstrom; bpo-957003(37).)
‘SMTP.starttls()’ now complies with RFC 3207(38) and forgets any
knowledge obtained from the server not obtained from the TLS
negotiation itself. (Patch contributed by Bill Fenner;
bpo-829951(39).)
* The *note socket: cc. module now supports TIPC
(‘https://tipc.sourceforge.net/’), a high-performance non-IP-based
protocol designed for use in clustered environments. TIPC
addresses are 4- or 5-tuples. (Contributed by Alberto Bertogli;
bpo-1646(40).)
A new function, ‘create_connection()’, takes an address and
connects to it using an optional timeout value, returning the
connected socket object. This function also looks up the address’s
type and connects to it using IPv4 or IPv6 as appropriate.
Changing your code to use ‘create_connection()’ instead of
‘socket(socket.AF_INET, ...)’ may be all that’s required to make
your code work with IPv6.
* The base classes in the *note SocketServer: cd. module now support
calling a *note handle_timeout(): 1314. method after a span of
inactivity specified by the server’s *note timeout: 1313.
attribute. (Contributed by Michael Pomraning.) The *note
serve_forever(): 1133. method now takes an optional poll interval
measured in seconds, controlling how often the server will check
for a shutdown request. (Contributed by Pedro Werneck and Jeffrey
Yasskin; bpo-742598(41), bpo-1193577(42).)
* The *note sqlite3: cf. module, maintained by Gerhard Häring, has
been updated from version 2.3.2 in Python 2.5 to version 2.4.1.
* The *note struct: d5. module now supports the C99 _Bool type, using
the format character ‘'?'’. (Contributed by David Remahl.)
* The *note Popen: 199. objects provided by the *note subprocess: d6.
module now have *note terminate(): 13a8, *note kill(): 13a9, and
*note send_signal(): 13aa. methods. On Windows, ‘send_signal()’
only supports the *note SIGTERM: 13ab. signal, and all these
methods are aliases for the Win32 API function
‘TerminateProcess()’. (Contributed by Christian Heimes.)
* A new variable in the *note sys: d9. module, ‘float_info’, is an
object containing information derived from the ‘float.h’ file about
the platform’s floating-point support. Attributes of this object
include ‘mant_dig’ (number of digits in the mantissa), ‘epsilon’
(smallest difference between 1.0 and the next largest value
representable), and several others. (Contributed by Christian
Heimes; bpo-1534(43).)
Another new variable, ‘dont_write_bytecode’, controls whether
Python writes any ‘.pyc’ or ‘.pyo’ files on importing a module. If
this variable is true, the compiled files are not written. The
variable is initially set on start-up by supplying the *note -B:
1399. switch to the Python interpreter, or by setting the *note
PYTHONDONTWRITEBYTECODE: 139a. environment variable before running
the interpreter. Python code can subsequently change the value of
this variable to control whether bytecode files are written or not.
(Contributed by Neal Norwitz and Georg Brandl.)
Information about the command-line arguments supplied to the Python
interpreter is available by reading attributes of a named tuple
available as ‘sys.flags’. For example, the ‘verbose’ attribute is
true if Python was executed in verbose mode, ‘debug’ is true in
debugging mode, etc. These attributes are all read-only.
(Contributed by Christian Heimes.)
A new function, ‘getsizeof()’, takes a Python object and returns
the amount of memory used by the object, measured in bytes.
Built-in objects return correct results; third-party extensions may
not, but can define a ‘__sizeof__()’ method to return the object’s
size. (Contributed by Robert Schuppenies; bpo-2898(44).)
It’s now possible to determine the current profiler and tracer
functions by calling *note sys.getprofile(): 13ac. and *note
sys.gettrace(): 13ad. (Contributed by Georg Brandl; bpo-1648(45).)
* The *note tarfile: de. module now supports POSIX.1-2001 (pax)
tarfiles in addition to the POSIX.1-1988 (ustar) and GNU tar
formats that were already supported. The default format is GNU
tar; specify the ‘format’ parameter to open a file using a
different format:
tar = tarfile.open("output.tar", "w",
format=tarfile.PAX_FORMAT)
The new ‘encoding’ and ‘errors’ parameters specify an encoding and
an error handling scheme for character conversions. ‘'strict'’,
‘'ignore'’, and ‘'replace'’ are the three standard ways Python can
handle errors,; ‘'utf-8'’ is a special value that replaces bad
characters with their UTF-8 representation. (Character conversions
occur because the PAX format supports Unicode filenames, defaulting
to UTF-8 encoding.)
The ‘TarFile.add()’ method now accepts an ‘exclude’ argument that’s
a function that can be used to exclude certain filenames from an
archive. The function must take a filename and return true if the
file should be excluded or false if it should be archived. The
function is applied to both the name initially passed to ‘add()’
and to the names of files in recursively added directories.
(All changes contributed by Lars Gustäbel).
* An optional ‘timeout’ parameter was added to the ‘telnetlib.Telnet’
class constructor, specifying a timeout measured in seconds.
(Added by Facundo Batista.)
* The *note tempfile.NamedTemporaryFile: 4c2. class usually deletes
the temporary file it created when the file is closed. This
behaviour can now be changed by passing ‘delete=False’ to the
constructor. (Contributed by Damien Miller; bpo-1537850(46).)
A new class, ‘SpooledTemporaryFile’, behaves like a temporary file
but stores its data in memory until a maximum size is exceeded. On
reaching that limit, the contents will be written to an on-disk
temporary file. (Contributed by Dustin J. Mitchell.)
The ‘NamedTemporaryFile’ and ‘SpooledTemporaryFile’ classes both
work as context managers, so you can write ‘with
tempfile.NamedTemporaryFile() as tmp: ...’. (Contributed by
Alexander Belopolsky; bpo-2021(47).)
* The *note test.test_support: e4. module gained a number of context
managers useful for writing tests. *note EnvironmentVarGuard():
13ae. is a context manager that temporarily changes environment
variables and automatically restores them to their old values.
Another context manager, ‘TransientResource’, can surround calls to
resources that may or may not be available; it will catch and
ignore a specified list of exceptions. For example, a network test
may ignore certain failures when connecting to an external web
site:
with test_support.TransientResource(IOError,
errno=errno.ETIMEDOUT):
f = urllib.urlopen('https://sf.net')
...
Finally, ‘check_warnings()’ resets the ‘warning’ module’s warning
filters and returns an object that will record all warning messages
triggered (bpo-3781(48)):
with test_support.check_warnings() as wrec:
warnings.simplefilter("always")
# ... code that triggers a warning ...
assert str(wrec.message) == "function is outdated"
assert len(wrec.warnings) == 1, "Multiple warnings raised"
(Contributed by Brett Cannon.)
* The *note textwrap: ec. module can now preserve existing whitespace
at the beginnings and ends of the newly created lines by specifying
‘drop_whitespace=False’ as an argument:
>>> S = """This sentence has a bunch of
... extra whitespace."""
>>> print textwrap.fill(S, width=15)
This sentence
has a bunch
of extra
whitespace.
>>> print textwrap.fill(S, drop_whitespace=False, width=15)
This sentence
has a bunch
of extra
whitespace.
>>>
(Contributed by Dwayne Bailey; bpo-1581073(49).)
* The *note threading: ed. module API is being changed to use
properties such as ‘daemon’ instead of ‘setDaemon()’ and
‘isDaemon()’ methods, and some methods have been renamed to use
underscores instead of camel-case; for example, the ‘activeCount()’
method is renamed to ‘active_count()’. Both the 2.6 and 3.0
versions of the module support the same properties and renamed
methods, but don’t remove the old methods. No date has been set
for the deprecation of the old APIs in Python 3.x; the old APIs
won’t be removed in any 2.x version. (Carried out by several
people, most notably Benjamin Peterson.)
The *note threading: ed. module’s ‘Thread’ objects gained an
‘ident’ property that returns the thread’s identifier, a nonzero
integer. (Contributed by Gregory P. Smith; bpo-2871(50).)
* The *note timeit: ef. module now accepts callables as well as
strings for the statement being timed and for the setup code. Two
convenience functions were added for creating ‘Timer’ instances:
‘repeat(stmt, setup, time, repeat, number)’ and ‘timeit(stmt,
setup, time, number)’ create an instance and call the corresponding
method. (Contributed by Erik Demaine; bpo-1533909(51).)
* The ‘Tkinter’ module now accepts lists and tuples for options,
separating the elements by spaces before passing the resulting
value to Tcl/Tk. (Contributed by Guilherme Polo; bpo-2906(52).)
* The *note turtle: 101. module for turtle graphics was greatly
enhanced by Gregor Lingl. New features in the module include:
* Better animation of turtle movement and rotation.
* Control over turtle movement using the new ‘delay()’,
‘tracer()’, and ‘speed()’ methods.
* The ability to set new shapes for the turtle, and to define a
new coordinate system.
* Turtles now have an ‘undo()’ method that can roll back
actions.
* Simple support for reacting to input events such as mouse and
keyboard activity, making it possible to write simple games.
* A ‘turtle.cfg’ file can be used to customize the starting
appearance of the turtle’s screen.
* The module’s docstrings can be replaced by new docstrings that
have been translated into another language.
(bpo-1513695(53))
* An optional ‘timeout’ parameter was added to the *note
urllib.urlopen: 295. function and the ‘urllib.ftpwrapper’ class
constructor, as well as the *note urllib2.urlopen: 295. function.
The parameter specifies a timeout measured in seconds. For
example:
>>> u = urllib2.urlopen("http://slow.example.com",
timeout=3)
Traceback (most recent call last):
...
urllib2.URLError:
>>>
(Added by Facundo Batista.)
* The Unicode database provided by the *note unicodedata: 105. module
has been updated to version 5.1.0. (Updated by Martin von Löwis;
bpo-3811(54).)
* The *note warnings: 112. module’s ‘formatwarning()’ and
‘showwarning()’ gained an optional 'line' argument that can be used
to supply the line of source code. (Added as part of
bpo-1631171(55), which re-implemented part of the *note warnings:
112. module in C code.)
A new function, ‘catch_warnings()’, is a context manager intended
for testing purposes that lets you temporarily modify the warning
filters and then restore their original values (bpo-3781(56)).
* The XML-RPC *note SimpleXMLRPCServer: 12f. and *note
DocXMLRPCServer: 12f. classes can now be prevented from immediately
opening and binding to their socket by passing ‘False’ as the
'bind_and_activate' constructor parameter. This can be used to
modify the instance’s ‘allow_reuse_address’ attribute before
calling the ‘server_bind()’ and ‘server_activate()’ methods to open
the socket and begin listening for connections. (Contributed by
Peter Parente; bpo-1599845(57).)
‘SimpleXMLRPCServer’ also has a ‘_send_traceback_header’ attribute;
if true, the exception and formatted traceback are returned as HTTP
headers “X-Exception” and “X-Traceback”. This feature is for
debugging purposes only and should not be used on production
servers because the tracebacks might reveal passwords or other
sensitive information. (Contributed by Alan McIntyre as part of
his project for Google’s Summer of Code 2007.)
* The *note xmlrpclib: 12e. module no longer automatically converts
*note datetime.date: 1cd. and *note datetime.time: 1ce. to the
*note xmlrpclib.DateTime: 13af. type; the conversion semantics were
not necessarily correct for all applications. Code using
‘xmlrpclib’ should convert ‘date’ and *note time: 1ce. instances.
(bpo-1330538(58)) The code can also handle dates before 1900
(contributed by Ralf Schmitt; bpo-2014(59)) and 64-bit integers
represented by using ‘’ in XML-RPC responses (contributed by
Riku Lindblad; bpo-2985(60)).
* The *note zipfile: 131. module’s ‘ZipFile’ class now has
‘extract()’ and ‘extractall()’ methods that will unpack a single
file or all the files in the archive to the current directory, or
to a specified directory:
z = zipfile.ZipFile('python-251.zip')
# Unpack a single file, writing it relative
# to the /tmp directory.
z.extract('Python/sysmodule.c', '/tmp')
# Unpack all the files in the archive.
z.extractall()
(Contributed by Alan McIntyre; bpo-467924(61).)
The *note open(): 517, ‘read()’ and ‘extract()’ methods can now
take either a filename or a ‘ZipInfo’ object. This is useful when
an archive accidentally contains a duplicated filename.
(Contributed by Graham Horler; bpo-1775025(62).)
Finally, *note zipfile: 131. now supports using Unicode filenames
for archived files. (Contributed by Alexey Borzenkov;
bpo-1734346(63).)
* Menu:
* The ast module::
* The future_builtins module::
* The json module; JavaScript Object Notation: The json module JavaScript Object Notation.
* The plistlib module; A Property-List Parser: The plistlib module A Property-List Parser.
* ctypes Enhancements::
* Improved SSL Support::
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1736190
(2) https://www.jcea.es/programacion/pybsddb.htm
(3) https://bugs.python.org/issue?@action=redirect&bpo=1817
(4) https://bugs.python.org/issue?@action=redirect&bpo=600362
(5) https://bugs.python.org/issue?@action=redirect&bpo=1381
(6) https://bugs.python.org/issue?@action=redirect&bpo=1638033
(7) https://bugs.python.org/issue?@action=redirect&bpo=1158
(8) https://speleotrove.com/decimal/decarith.html
(9) https://bugs.python.org/issue?@action=redirect&bpo=1221598
(10) https://bugs.python.org/issue?@action=redirect&bpo=1739906
(11) https://bugs.python.org/issue?@action=redirect&bpo=1001604
(12) https://bugs.python.org/issue?@action=redirect&bpo=2138
(13) https://bugs.python.org/issue?@action=redirect&bpo=2819
(14) https://bugs.python.org/issue?@action=redirect&bpo=711019
(15) https://bugs.python.org/issue?@action=redirect&bpo=1640
(16) https://bugs.python.org/issue?@action=redirect&bpo=1663329
(17) https://bugs.python.org/issue?@action=redirect&bpo=1181
(18) https://bugs.python.org/issue?@action=redirect&bpo=1273829
(19) https://bugs.python.org/issue?@action=redirect&bpo=1115886
(20) https://bugs.python.org/issue?@action=redirect&bpo=1339796
(21) https://bugs.python.org/issue?@action=redirect&bpo=957650
(22) https://bugs.python.org/issue?@action=redirect&bpo=1393667
(23) https://bugs.python.org/issue?@action=redirect&bpo=1106316
(24) https://bugs.python.org/issue?@action=redirect&bpo=2439
(25) https://bugs.python.org/issue?@action=redirect&bpo=1137
(26) https://bugs.python.org/issue?@action=redirect&bpo=1727780
(27) https://bugs.python.org/issue?@action=redirect&bpo=1681432
(28) https://bugs.python.org/issue?@action=redirect&bpo=846388
(29) https://bugs.python.org/issue?@action=redirect&bpo=3487
(30) https://bugs.python.org/issue?@action=redirect&bpo=2250
(31) https://bugs.python.org/issue?@action=redirect&bpo=1861
(32) https://bugs.python.org/issue?@action=redirect&bpo=1657
(33) https://bugs.python.org/issue?@action=redirect&bpo=2663
(34) https://bugs.python.org/issue?@action=redirect&bpo=1583
(35) https://bugs.python.org/issue?@action=redirect&bpo=2240
(36) https://datatracker.ietf.org/doc/html/rfc2033.html
(37) https://bugs.python.org/issue?@action=redirect&bpo=957003
(38) https://datatracker.ietf.org/doc/html/rfc3207.html
(39) https://bugs.python.org/issue?@action=redirect&bpo=829951
(40) https://bugs.python.org/issue?@action=redirect&bpo=1646
(41) https://bugs.python.org/issue?@action=redirect&bpo=742598
(42) https://bugs.python.org/issue?@action=redirect&bpo=1193577
(43) https://bugs.python.org/issue?@action=redirect&bpo=1534
(44) https://bugs.python.org/issue?@action=redirect&bpo=2898
(45) https://bugs.python.org/issue?@action=redirect&bpo=1648
(46) https://bugs.python.org/issue?@action=redirect&bpo=1537850
(47) https://bugs.python.org/issue?@action=redirect&bpo=2021
(48) https://bugs.python.org/issue?@action=redirect&bpo=3781
(49) https://bugs.python.org/issue?@action=redirect&bpo=1581073
(50) https://bugs.python.org/issue?@action=redirect&bpo=2871
(51) https://bugs.python.org/issue?@action=redirect&bpo=1533909
(52) https://bugs.python.org/issue?@action=redirect&bpo=2906
(53) https://bugs.python.org/issue?@action=redirect&bpo=1513695
(54) https://bugs.python.org/issue?@action=redirect&bpo=3811
(55) https://bugs.python.org/issue?@action=redirect&bpo=1631171
(56) https://bugs.python.org/issue?@action=redirect&bpo=3781
(57) https://bugs.python.org/issue?@action=redirect&bpo=1599845
(58) https://bugs.python.org/issue?@action=redirect&bpo=1330538
(59) https://bugs.python.org/issue?@action=redirect&bpo=2014
(60) https://bugs.python.org/issue?@action=redirect&bpo=2985
(61) https://bugs.python.org/issue?@action=redirect&bpo=467924
(62) https://bugs.python.org/issue?@action=redirect&bpo=1775025
(63) https://bugs.python.org/issue?@action=redirect&bpo=1734346
File: python.info, Node: The ast module, Next: The future_builtins module, Up: New and Improved Modules<2>
1.16.18.1 The ‘ast’ module
..........................
The *note ast: 8. module provides an Abstract Syntax Tree representation
of Python code, and Armin Ronacher contributed a set of helper functions
that perform a variety of common tasks. These will be useful for HTML
templating packages, code analyzers, and similar tools that process
Python code.
The ‘parse()’ function takes an expression and returns an AST. The
‘dump()’ function outputs a representation of a tree, suitable for
debugging:
import ast
t = ast.parse("""
d = {}
for i in 'abcdefghijklm':
d[i + i] = ord(i) - ord('a') + 1
print d
""")
print ast.dump(t)
This outputs a deeply nested tree:
Module(body=[
Assign(targets=[
Name(id='d', ctx=Store())
], value=Dict(keys=[], values=[]))
For(target=Name(id='i', ctx=Store()),
iter=Str(s='abcdefghijklm'), body=[
Assign(targets=[
Subscript(value=
Name(id='d', ctx=Load()),
slice=
Index(value=
BinOp(left=Name(id='i', ctx=Load()), op=Add(),
right=Name(id='i', ctx=Load()))), ctx=Store())
], value=
BinOp(left=
BinOp(left=
Call(func=
Name(id='ord', ctx=Load()), args=[
Name(id='i', ctx=Load())
], keywords=[], starargs=None, kwargs=None),
op=Sub(), right=Call(func=
Name(id='ord', ctx=Load()), args=[
Str(s='a')
], keywords=[], starargs=None, kwargs=None)),
op=Add(), right=Num(n=1)))
], orelse=[])
Print(dest=None, values=[
Name(id='d', ctx=Load())
], nl=True)
])
The ‘literal_eval()’ method takes a string or an AST representing a
literal expression, parses and evaluates it, and returns the resulting
value. A literal expression is a Python expression containing only
strings, numbers, dictionaries, etc. but no statements or function
calls. If you need to evaluate an expression but cannot accept the
security risk of using an *note eval(): 180. call, ‘literal_eval()’ will
handle it safely:
>>> literal = '("a", "b", {2:4, 3:8, 1:2})'
>>> print ast.literal_eval(literal)
('a', 'b', {1: 2, 2: 4, 3: 8})
>>> print ast.literal_eval('"a" + "b"')
Traceback (most recent call last):
...
ValueError: malformed string
The module also includes ‘NodeVisitor’ and ‘NodeTransformer’ classes for
traversing and modifying an AST, and functions for common
transformations such as changing line numbers.
File: python.info, Node: The future_builtins module, Next: The json module JavaScript Object Notation, Prev: The ast module, Up: New and Improved Modules<2>
1.16.18.2 The ‘future_builtins’ module
......................................
Python 3.0 makes many changes to the repertoire of built-in functions,
and most of the changes can’t be introduced in the Python 2.x series
because they would break compatibility. The ‘future_builtins’ module
provides versions of these built-in functions that can be imported when
writing 3.0-compatible code.
The functions in this module currently include:
* ‘ascii(obj)’: equivalent to *note repr(): 7f9. In Python 3.0,
*note repr(): 7f9. will return a Unicode string, while *note
ascii(): 13b2. will return a pure ASCII bytestring.
* ‘filter(predicate, iterable)’, ‘map(func, iterable1, ...)’: the 3.0
versions return iterators, unlike the 2.x builtins which return
lists.
* ‘hex(value)’, ‘oct(value)’: instead of calling the ‘__hex__()’ or
‘__oct__()’ methods, these versions will call the ‘__index__()’
method and convert the result to hexadecimal or octal. *note
oct(): 12c1. will use the new ‘0o’ notation for its result.
File: python.info, Node: The json module JavaScript Object Notation, Next: The plistlib module A Property-List Parser, Prev: The future_builtins module, Up: New and Improved Modules<2>
1.16.18.3 The ‘json’ module: JavaScript Object Notation
.......................................................
The new *note json: 82. module supports the encoding and decoding of
Python types in JSON (Javascript Object Notation). JSON is a
lightweight interchange format often used in web applications. For more
information about JSON, see ‘http://www.json.org’.
*note json: 82. comes with support for decoding and encoding most
built-in Python types. The following example encodes and decodes a
dictionary:
>>> import json
>>> data = {"spam": "foo", "parrot": 42}
>>> in_json = json.dumps(data) # Encode the data
>>> in_json
'{"parrot": 42, "spam": "foo"}'
>>> json.loads(in_json) # Decode into a Python object
{"spam": "foo", "parrot": 42}
It’s also possible to write your own decoders and encoders to support
more types. Pretty-printing of the JSON strings is also supported.
*note json: 82. (originally called simplejson) was written by Bob
Ippolito.
File: python.info, Node: The plistlib module A Property-List Parser, Next: ctypes Enhancements, Prev: The json module JavaScript Object Notation, Up: New and Improved Modules<2>
1.16.18.4 The ‘plistlib’ module: A Property-List Parser
.......................................................
The ‘.plist’ format is commonly used on Mac OS X to store basic data
types (numbers, strings, lists, and dictionaries) by serializing them
into an XML-based format. It resembles the XML-RPC serialization of
data types.
Despite being primarily used on Mac OS X, the format has nothing
Mac-specific about it and the Python implementation works on any
platform that Python supports, so the *note plistlib: ab. module has
been promoted to the standard library.
Using the module is simple:
import sys
import plistlib
import datetime
# Create data structure
data_struct = dict(lastAccessed=datetime.datetime.now(),
version=1,
categories=('Personal','Shared','Private'))
# Create string containing XML.
plist_str = plistlib.writePlistToString(data_struct)
new_struct = plistlib.readPlistFromString(plist_str)
print data_struct
print new_struct
# Write data structure to a file and read it back.
plistlib.writePlist(data_struct, '/tmp/customizations.plist')
new_struct = plistlib.readPlist('/tmp/customizations.plist')
# read/writePlist accepts file-like objects as well as paths.
plistlib.writePlist(data_struct, sys.stdout)
File: python.info, Node: ctypes Enhancements, Next: Improved SSL Support, Prev: The plistlib module A Property-List Parser, Up: New and Improved Modules<2>
1.16.18.5 ctypes Enhancements
.............................
Thomas Heller continued to maintain and enhance the *note ctypes: 2a.
module.
*note ctypes: 2a. now supports a ‘c_bool’ datatype that represents the
C99 ‘bool’ type. (Contributed by David Remahl; bpo-1649190(1).)
The *note ctypes: 2a. string, buffer and array types have improved
support for extended slicing syntax, where various combinations of
‘(start, stop, step)’ are supplied. (Implemented by Thomas Wouters.)
All *note ctypes: 2a. data types now support ‘from_buffer()’ and
‘from_buffer_copy()’ methods that create a ctypes instance based on a
provided buffer object. ‘from_buffer_copy()’ copies the contents of the
object, while ‘from_buffer()’ will share the same memory area.
A new calling convention tells *note ctypes: 2a. to clear the ‘errno’ or
Win32 LastError variables at the outset of each wrapped call.
(Implemented by Thomas Heller; bpo-1798(2).)
You can now retrieve the Unix ‘errno’ variable after a function call.
When creating a wrapped function, you can supply ‘use_errno=True’ as a
keyword parameter to the ‘DLL()’ function and then call the module-level
methods ‘set_errno()’ and ‘get_errno()’ to set and retrieve the error
value.
The Win32 LastError variable is similarly supported by the ‘DLL()’,
‘OleDLL()’, and ‘WinDLL()’ functions. You supply ‘use_last_error=True’
as a keyword parameter and then call the module-level methods
‘set_last_error()’ and ‘get_last_error()’.
The ‘byref()’ function, used to retrieve a pointer to a ctypes instance,
now has an optional 'offset' parameter that is a byte count that will be
added to the returned pointer.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1649190
(2) https://bugs.python.org/issue?@action=redirect&bpo=1798
File: python.info, Node: Improved SSL Support, Prev: ctypes Enhancements, Up: New and Improved Modules<2>
1.16.18.6 Improved SSL Support
..............................
Bill Janssen made extensive improvements to Python 2.6’s support for the
Secure Sockets Layer by adding a new module, *note ssl: d0, that’s built
atop the OpenSSL(1) library. This new module provides more control over
the protocol negotiated, the X.509 certificates used, and has better
support for writing SSL servers (as opposed to clients) in Python. The
existing SSL support in the *note socket: cc. module hasn’t been removed
and continues to work, though it will be removed in Python 3.0.
To use the new module, you must first create a TCP connection in the
usual way and then pass it to the ‘ssl.wrap_socket()’ function. It’s
possible to specify whether a certificate is required, and to obtain
certificate info by calling the ‘getpeercert()’ method.
See also
........
The documentation for the *note ssl: d0. module.
---------- Footnotes ----------
(1) https://www.openssl.org/
File: python.info, Node: Deprecations and Removals, Next: Build and C API Changes<9>, Prev: New and Improved Modules<2>, Up: What’s New in Python 2 6
1.16.19 Deprecations and Removals
---------------------------------
* String exceptions have been removed. Attempting to use them raises
a *note TypeError: 534.
* Changes to the *note Exception: 9d9. interface as dictated by PEP
352(1) continue to be made. For 2.6, the ‘message’ attribute is
being deprecated in favor of the *note args: 569. attribute.
* (3.0-warning mode) Python 3.0 will feature a reorganized standard
library that will drop many outdated modules and rename others.
Python 2.6 running in 3.0-warning mode will warn about these
modules when they are imported.
The list of deprecated modules is: ‘audiodev’, ‘bgenlocations’,
‘buildtools’, ‘bundlebuilder’, ‘Canvas’, ‘compiler’, ‘dircache’,
‘dl’, ‘fpformat’, ‘gensuitemodule’, ‘ihooks’, ‘imageop’, ‘imgfile’,
‘linuxaudiodev’, ‘mhlib’, ‘mimetools’, ‘multifile’, ‘new’, ‘pure’,
‘statvfs’, ‘sunaudiodev’, ‘test.testall’, and ‘toaiff’.
* The ‘gopherlib’ module has been removed.
* The ‘MimeWriter’ module and ‘mimify’ module have been deprecated;
use the *note email: 3b. package instead.
* The ‘md5’ module has been deprecated; use the *note hashlib: 68.
module instead.
* The ‘posixfile’ module has been deprecated; *note fcntl.lockf():
13b8. provides better locking.
* The ‘popen2’ module has been deprecated; use the *note subprocess:
d6. module.
* The ‘rgbimg’ module has been removed.
* The ‘sets’ module has been deprecated; it’s better to use the
built-in *note set: 5d5. and *note frozenset: 5d6. types.
* The ‘sha’ module has been deprecated; use the *note hashlib: 68.
module instead.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0352/
File: python.info, Node: Build and C API Changes<9>, Next: Porting to Python 2 6, Prev: Deprecations and Removals, Up: What’s New in Python 2 6
1.16.20 Build and C API Changes
-------------------------------
Changes to Python’s build process and to the C API include:
* Python now must be compiled with C89 compilers (after 19 years!).
This means that the Python source tree has dropped its own
implementations of ‘memmove()’ and ‘strerror()’, which are in the
C89 standard library.
* Python 2.6 can be built with Microsoft Visual Studio 2008 (version
9.0), and this is the new default compiler. See the ‘PCbuild’
directory for the build files. (Implemented by Christian Heimes.)
* On Mac OS X, Python 2.6 can be compiled as a 4-way universal build.
The ‘configure’ script can take a
‘--with-universal-archs=[32-bit|64-bit|all]’ switch, controlling
whether the binaries are built for 32-bit architectures (x86,
PowerPC), 64-bit (x86-64 and PPC-64), or both. (Contributed by
Ronald Oussoren.)
* A new function added in Python 2.6.6, ‘PySys_SetArgvEx()’, sets the
value of ‘sys.argv’ and can optionally update ‘sys.path’ to include
the directory containing the script named by ‘sys.argv[0]’
depending on the value of an 'updatepath' parameter.
This function was added to close a security hole for applications
that embed Python. The old function, ‘PySys_SetArgv()’, would
always update ‘sys.path’, and sometimes it would add the current
directory. This meant that, if you ran an application embedding
Python in a directory controlled by someone else, attackers could
put a Trojan-horse module in the directory (say, a file named
‘os.py’) that your application would then import and run.
If you maintain a C/C++ application that embeds Python, check
whether you’re calling ‘PySys_SetArgv()’ and carefully consider
whether the application should be using ‘PySys_SetArgvEx()’ with
'updatepath' set to false. Note that using this function will
break compatibility with Python versions 2.6.5 and earlier; if you
have to continue working with earlier versions, you can leave the
call to ‘PySys_SetArgv()’ alone and call
‘PyRun_SimpleString("sys.path.pop(0)\n")’ afterwards to discard the
first ‘sys.path’ component.
Security issue reported as CVE 2008-5983(1); discussed in
gh-50003(2), and fixed by Antoine Pitrou.
* The BerkeleyDB module now has a C API object, available as
‘bsddb.db.api’. This object can be used by other C extensions that
wish to use the ‘bsddb’ module for their own purposes.
(Contributed by Duncan Grisby.)
* The new buffer interface, previously described in *note the PEP
3118 section: 1385, adds *note PyObject_GetBuffer(): 395. and *note
PyBuffer_Release(): 396, as well as a few other functions.
* Python’s use of the C stdio library is now thread-safe, or at least
as thread-safe as the underlying library is. A long-standing
potential bug occurred if one thread closed a file object while
another thread was reading from or writing to the object. In 2.6
file objects have a reference count, manipulated by the
‘PyFile_IncUseCount()’ and ‘PyFile_DecUseCount()’ functions. File
objects can’t be closed unless the reference count is zero.
‘PyFile_IncUseCount()’ should be called while the GIL is still
held, before carrying out an I/O operation using the ‘FILE *’
pointer, and ‘PyFile_DecUseCount()’ should be called immediately
after the GIL is re-acquired. (Contributed by Antoine Pitrou and
Gregory P. Smith.)
* Importing modules simultaneously in two different threads no longer
deadlocks; it will now raise an *note ImportError: 415. A new API
function, *note PyImport_ImportModuleNoBlock(): 3b9, will look for
a module in ‘sys.modules’ first, then try to import it after
acquiring an import lock. If the import lock is held by another
thread, an *note ImportError: 415. is raised. (Contributed by
Christian Heimes.)
* Several functions return information about the platform’s
floating-point support. *note PyFloat_GetMax(): 13ba. returns the
maximum representable floating-point value, and *note
PyFloat_GetMin(): 13bb. returns the minimum positive value. *note
PyFloat_GetInfo(): 13bc. returns an object containing more
information from the ‘float.h’ file, such as ‘"mant_dig"’ (number
of digits in the mantissa), ‘"epsilon"’ (smallest difference
between 1.0 and the next largest value representable), and several
others. (Contributed by Christian Heimes; bpo-1534(3).)
* C functions and methods that use *note PyComplex_AsCComplex():
13bd. will now accept arguments that have a ‘__complex__()’ method.
In particular, the functions in the *note cmath: 18. module will
now accept objects with this method. This is a backport of a
Python 3.0 change. (Contributed by Mark Dickinson;
bpo-1675423(4).)
* Python’s C API now includes two functions for case-insensitive
string comparisons, ‘PyOS_stricmp(char*, char*)’ and
‘PyOS_strnicmp(char*, char*, Py_ssize_t)’. (Contributed by
Christian Heimes; bpo-1635(5).)
* Many C extensions define their own little macro for adding integers
and strings to the module’s dictionary in the ‘init*’ function.
Python 2.6 finally defines standard macros for adding values to a
module, *note PyModule_AddStringMacro: 13be. and *note
PyModule_AddIntMacro(): 13bf. (Contributed by Christian Heimes.)
* Some macros were renamed in both 3.0 and 2.6 to make it clearer
that they are macros, not functions. ‘Py_Size()’ became *note
Py_SIZE(): 77d, ‘Py_Type()’ became *note Py_TYPE(): 77b, and
‘Py_Refcnt()’ became *note Py_REFCNT(): 8a8. The mixed-case macros
are still available in Python 2.6 for backward compatibility.
(bpo-1629(6))
* Distutils now places C extensions it builds in a different
directory when running on a debug version of Python. (Contributed
by Collin Winter; bpo-1530959(7).)
* Several basic data types, such as integers and strings, maintain
internal free lists of objects that can be re-used. The data
structures for these free lists now follow a naming convention: the
variable is always named ‘free_list’, the counter is always named
‘numfree’, and a macro ‘Py_MAXFREELIST’ is always
defined.
* A new Makefile target, “make patchcheck”, prepares the Python
source tree for making a patch: it fixes trailing whitespace in all
modified ‘.py’ files, checks whether the documentation has been
changed, and reports whether the ‘Misc/ACKS’ and ‘Misc/NEWS’ files
have been updated. (Contributed by Brett Cannon.)
Another new target, “make profile-opt”, compiles a Python binary
using GCC’s profile-guided optimization. It compiles Python with
profiling enabled, runs the test suite to obtain a set of profiling
results, and then compiles using these results for optimization.
(Contributed by Gregory P. Smith.)
* Menu:
* Port-Specific Changes; Windows: Port-Specific Changes Windows<2>.
* Port-Specific Changes; Mac OS X: Port-Specific Changes Mac OS X<2>.
* Port-Specific Changes; IRIX: Port-Specific Changes IRIX.
---------- Footnotes ----------
(1) https://www.cve.org/CVERecord?id=CVE-2008-5983
(2) https://github.com/python/cpython/issues/50003
(3) https://bugs.python.org/issue?@action=redirect&bpo=1534
(4) https://bugs.python.org/issue?@action=redirect&bpo=1675423
(5) https://bugs.python.org/issue?@action=redirect&bpo=1635
(6) https://bugs.python.org/issue?@action=redirect&bpo=1629
(7) https://bugs.python.org/issue?@action=redirect&bpo=1530959
File: python.info, Node: Port-Specific Changes Windows<2>, Next: Port-Specific Changes Mac OS X<2>, Up: Build and C API Changes<9>
1.16.20.1 Port-Specific Changes: Windows
........................................
* The support for Windows 95, 98, ME and NT4 has been dropped.
Python 2.6 requires at least Windows 2000 SP4.
* The new default compiler on Windows is Visual Studio 2008 (version
9.0). The build directories for Visual Studio 2003 (version 7.1)
and 2005 (version 8.0) were moved into the PC/ directory. The new
‘PCbuild’ directory supports cross compilation for X64, debug
builds and Profile Guided Optimization (PGO). PGO builds are
roughly 10% faster than normal builds. (Contributed by Christian
Heimes with help from Amaury Forgeot d’Arc and Martin von Löwis.)
* The *note msvcrt: 93. module now supports both the normal and wide
char variants of the console I/O API. The *note getwch(): 13c1.
function reads a keypress and returns a Unicode value, as does the
*note getwche(): 13c2. function. The *note putwch(): 13c3.
function takes a Unicode character and writes it to the console.
(Contributed by Christian Heimes.)
* *note os.path.expandvars(): 13a5. will now expand environment
variables in the form “%var%”, and “~user” will be expanded into
the user’s home directory path. (Contributed by Josiah Carlson;
bpo-957650(1).)
* The *note socket: cc. module’s socket objects now have an *note
ioctl(): cd3. method that provides a limited interface to the
‘WSAIoctl()’ system interface.
* The *note _winreg: 116. module now has a function, *note
ExpandEnvironmentStrings(): 13c4, that expands environment variable
references such as ‘%NAME%’ in an input string. The handle objects
provided by this module now support the context protocol, so they
can be used in *note with: 5ce. statements. (Contributed by
Christian Heimes.)
*note _winreg: 116. also has better support for x64 systems,
exposing the *note DisableReflectionKey(): 1350, *note
EnableReflectionKey(): 1351, and *note QueryReflectionKey(): 1352.
functions, which enable and disable registry reflection for 32-bit
processes running on 64-bit systems. (bpo-1753245(2))
* The ‘msilib’ module’s ‘Record’ object gained ‘GetInteger()’ and
‘GetString()’ methods that return field values as an integer or a
string. (Contributed by Floris Bruynooghe; bpo-2125(3).)
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=957650
(2) https://bugs.python.org/issue?@action=redirect&bpo=1753245
(3) https://bugs.python.org/issue?@action=redirect&bpo=2125
File: python.info, Node: Port-Specific Changes Mac OS X<2>, Next: Port-Specific Changes IRIX, Prev: Port-Specific Changes Windows<2>, Up: Build and C API Changes<9>
1.16.20.2 Port-Specific Changes: Mac OS X
.........................................
* When compiling a framework build of Python, you can now specify the
framework name to be used by providing the ‘--with-framework-name=’
option to the ‘configure’ script.
* The ‘macfs’ module has been removed. This in turn required the
‘macostools.touched()’ function to be removed because it depended
on the ‘macfs’ module. (bpo-1490190(1))
* Many other Mac OS modules have been deprecated and will be removed
in Python 3.0: ‘_builtinSuites’, ‘aepack’, ‘aetools’, ‘aetypes’,
‘applesingle’, ‘appletrawmain’, ‘appletrunner’, ‘argvemulator’,
‘Audio_mac’, ‘autoGIL’, ‘Carbon’, ‘cfmfile’, ‘CodeWarrior’,
‘ColorPicker’, ‘EasyDialogs’, ‘Explorer’, ‘Finder’, ‘FrameWork’,
‘findertools’, ‘ic’, ‘icglue’, ‘icopen’, ‘macerrors’, ‘MacOS’,
‘macfs’, ‘macostools’, ‘macresource’, ‘MiniAEFrame’, ‘Nav’,
‘Netscape’, ‘OSATerminology’, ‘pimp’, ‘PixMapWrapper’, ‘StdSuites’,
‘SystemEvents’, ‘Terminal’, and ‘terminalcommand’.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1490190
File: python.info, Node: Port-Specific Changes IRIX, Prev: Port-Specific Changes Mac OS X<2>, Up: Build and C API Changes<9>
1.16.20.3 Port-Specific Changes: IRIX
.....................................
A number of old IRIX-specific modules were deprecated and will be
removed in Python 3.0: ‘al’ and ‘AL’, ‘cd’, ‘cddb’, ‘cdplayer’, ‘CL’ and
‘cl’, ‘DEVICE’, ‘ERRNO’, ‘FILE’, ‘FL’ and ‘fl’, ‘flp’, ‘fm’, ‘GET’,
‘GLWS’, ‘GL’ and ‘gl’, ‘IN’, ‘IOCTL’, ‘jpeg’, ‘panelparser’, ‘readcd’,
‘SV’ and ‘sv’, ‘torgb’, ‘videoreader’, and ‘WAIT’.
File: python.info, Node: Porting to Python 2 6, Next: Acknowledgements<2>, Prev: Build and C API Changes<9>, Up: What’s New in Python 2 6
1.16.21 Porting to Python 2.6
-----------------------------
This section lists previously described changes and other bugfixes that
may require changes to your code:
* Classes that aren’t supposed to be hashable should set ‘__hash__ =
None’ in their definitions to indicate the fact.
* String exceptions have been removed. Attempting to use them raises
a *note TypeError: 534.
* The ‘__init__()’ method of *note collections.deque: 5d8. now clears
any existing contents of the deque before adding elements from the
iterable. This change makes the behavior match ‘list.__init__()’.
* *note object.__init__(): 6ac. previously accepted arbitrary
arguments and keyword arguments, ignoring them. In Python 2.6,
this is no longer allowed and will result in a *note TypeError:
534. This will affect ‘__init__()’ methods that end up calling the
corresponding method on *note object: a8c. (perhaps through using
*note super(): 4d7.). See bpo-1683368(1) for discussion.
* The ‘Decimal’ constructor now accepts leading and trailing
whitespace when passed a string. Previously it would raise an
‘InvalidOperation’ exception. On the other hand, the
‘create_decimal()’ method of ‘Context’ objects now explicitly
disallows extra whitespace, raising a ‘ConversionSyntax’ exception.
* Due to an implementation accident, if you passed a file path to the
built-in *note __import__(): 8d3. function, it would actually
import the specified file. This was never intended to work,
however, and the implementation now explicitly checks for this case
and raises an *note ImportError: 415.
* C API: the *note PyImport_Import(): 13c8. and *note
PyImport_ImportModule(): 3ba. functions now default to absolute
imports, not relative imports. This will affect C extensions that
import other modules.
* C API: extension data types that shouldn’t be hashable should
define their ‘tp_hash’ slot to *note PyObject_HashNotImplemented():
1392.
* The *note socket: cc. module exception *note socket.error: 104e.
now inherits from *note IOError: 104b. Previously it wasn’t a
subclass of ‘StandardError’ but now it is, through *note IOError:
104b. (Implemented by Gregory P. Smith; bpo-1706815(2).)
* The *note xmlrpclib: 12e. module no longer automatically converts
*note datetime.date: 1cd. and *note datetime.time: 1ce. to the
*note xmlrpclib.DateTime: 13af. type; the conversion semantics were
not necessarily correct for all applications. Code using
‘xmlrpclib’ should convert ‘date’ and *note time: 1ce. instances.
(bpo-1330538(3))
* (3.0-warning mode) The *note Exception: 9d9. class now warns when
accessed using slicing or index access; having *note Exception:
9d9. behave like a tuple is being phased out.
* (3.0-warning mode) inequality comparisons between two dictionaries
or two objects that don’t implement comparison methods are reported
as warnings. ‘dict1 == dict2’ still works, but ‘dict1 < dict2’ is
being phased out.
Comparisons between cells, which are an implementation detail of
Python’s scoping rules, also cause warnings because such
comparisons are forbidden entirely in 3.0.
For applications that embed Python:
* The ‘PySys_SetArgvEx()’ function was added in Python 2.6.6, letting
applications close a security hole when the existing
‘PySys_SetArgv()’ function was used. Check whether you’re calling
‘PySys_SetArgv()’ and carefully consider whether the application
should be using ‘PySys_SetArgvEx()’ with 'updatepath' set to false.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=1683368
(2) https://bugs.python.org/issue?@action=redirect&bpo=1706815
(3) https://bugs.python.org/issue?@action=redirect&bpo=1330538
File: python.info, Node: Acknowledgements<2>, Prev: Porting to Python 2 6, Up: What’s New in Python 2 6
1.16.22 Acknowledgements
------------------------
The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Georg Brandl, Steve Brown, Nick Coghlan, Ralph Corderoy, Jim
Jewett, Kent Johnson, Chris Lambacher, Martin Michlmayr, Antoine Pitrou,
Brian Warner.
File: python.info, Node: What’s New in Python 2 5, Next: What’s New in Python 2 4, Prev: What’s New in Python 2 6, Up: What’s New in Python
1.17 What’s New in Python 2.5
=============================
Author: A.M. Kuchling
This article explains the new features in Python 2.5. The final release
of Python 2.5 is scheduled for August 2006; PEP 356(1) describes the
planned release schedule. Python 2.5 was released on September 19,
2006.
The changes in Python 2.5 are an interesting mix of language and library
improvements. The library enhancements will be more important to
Python’s user community, I think, because several widely useful packages
were added. New modules include ElementTree for XML processing
(‘xml.etree’), the SQLite database module (‘sqlite’), and the *note
ctypes: 2a. module for calling C functions.
The language changes are of middling significance. Some pleasant new
features were added, but most of them aren’t features that you’ll use
every day. Conditional expressions were finally added to the language
using a novel syntax; see section *note PEP 308; Conditional
Expressions: 13cd. The new ‘*note with: 5ce.’ statement will make
writing cleanup code easier (section *note PEP 343; The ‘with’
statement: 13ce.). Values can now be passed into generators (section
*note PEP 342; New Generator Features: 13cf.). Imports are now visible
as either absolute or relative (section *note PEP 328; Absolute and
Relative Imports: 13d0.). Some corner cases of exception handling are
handled better (section *note PEP 341; Unified try/except/finally:
13d1.). All these improvements are worthwhile, but they’re improvements
to one specific language feature or another; none of them are broad
modifications to Python’s semantics.
As well as the language and library additions, other improvements and
bugfixes were made throughout the source tree. A search through the SVN
change logs finds there were 353 patches applied and 458 bugs fixed
between Python 2.4 and 2.5. (Both figures are likely to be
underestimates.)
This article doesn’t try to be a complete specification of the new
features; instead changes are briefly introduced using helpful examples.
For full details, you should always refer to the documentation for
Python 2.5 at ‘https://docs.python.org’. If you want to understand the
complete implementation and design rationale, refer to the PEP for a
particular new feature.
Comments, suggestions, and error reports for this document are welcome;
please e-mail them to the author or open a bug in the Python bug
tracker.
* Menu:
* PEP 308; Conditional Expressions: PEP 308 Conditional Expressions.
* PEP 309; Partial Function Application: PEP 309 Partial Function Application.
* PEP 314; Metadata for Python Software Packages v1.1: PEP 314 Metadata for Python Software Packages v1 1.
* PEP 328; Absolute and Relative Imports: PEP 328 Absolute and Relative Imports.
* PEP 338; Executing Modules as Scripts: PEP 338 Executing Modules as Scripts.
* PEP 341; Unified try/except/finally: PEP 341 Unified try/except/finally.
* PEP 342; New Generator Features: PEP 342 New Generator Features.
* PEP 343; The ‘with’ statement: PEP 343 The ‘with’ statement<2>.
* PEP 352; Exceptions as New-Style Classes: PEP 352 Exceptions as New-Style Classes.
* PEP 353; Using ssize_t as the index type: PEP 353 Using ssize_t as the index type.
* PEP 357; The ‘__index__’ method: PEP 357 The ‘__index__’ method.
* Other Language Changes: Other Language Changes<16>.
* New, Improved, and Removed Modules: New Improved and Removed Modules.
* Build and C API Changes: Build and C API Changes<10>.
* Porting to Python 2.5: Porting to Python 2 5.
* Acknowledgements: Acknowledgements<3>.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0356/
File: python.info, Node: PEP 308 Conditional Expressions, Next: PEP 309 Partial Function Application, Up: What’s New in Python 2 5
1.17.1 PEP 308: Conditional Expressions
---------------------------------------
For a long time, people have been requesting a way to write conditional
expressions, which are expressions that return value A or value B
depending on whether a Boolean value is true or false. A conditional
expression lets you write a single assignment statement that has the
same effect as the following:
if condition:
x = true_value
else:
x = false_value
There have been endless tedious discussions of syntax on both python-dev
and comp.lang.python. A vote was even held that found the majority of
voters wanted conditional expressions in some form, but there was no
syntax that was preferred by a clear majority. Candidates included C’s
‘cond ? true_v : false_v’, ‘if cond then true_v else false_v’, and 16
other variations.
Guido van Rossum eventually chose a surprising syntax:
x = true_value if condition else false_value
Evaluation is still lazy as in existing Boolean expressions, so the
order of evaluation jumps around a bit. The 'condition' expression in
the middle is evaluated first, and the 'true_value' expression is
evaluated only if the condition was true. Similarly, the 'false_value'
expression is only evaluated when the condition is false.
This syntax may seem strange and backwards; why does the condition go in
the 'middle' of the expression, and not in the front as in C’s ‘c ? x :
y’? The decision was checked by applying the new syntax to the modules
in the standard library and seeing how the resulting code read. In many
cases where a conditional expression is used, one value seems to be the
‘common case’ and one value is an ‘exceptional case’, used only on rarer
occasions when the condition isn’t met. The conditional syntax makes
this pattern a bit more obvious:
contents = ((doc + '\n') if doc else '')
I read the above statement as meaning “here 'contents' is usually
assigned a value of ‘doc+'\n'’; sometimes 'doc' is empty, in which
special case an empty string is returned.” I doubt I will use
conditional expressions very often where there isn’t a clear common and
uncommon case.
There was some discussion of whether the language should require
surrounding conditional expressions with parentheses. The decision was
made to 'not' require parentheses in the Python language’s grammar, but
as a matter of style I think you should always use them. Consider these
two statements:
# First version -- no parens
level = 1 if logging else 0
# Second version -- with parens
level = (1 if logging else 0)
In the first version, I think a reader’s eye might group the statement
into ‘level = 1’, ‘if logging’, ‘else 0’, and think that the condition
decides whether the assignment to 'level' is performed. The second
version reads better, in my opinion, because it makes it clear that the
assignment is always performed and the choice is being made between two
values.
Another reason for including the brackets: a few odd combinations of
list comprehensions and lambdas could look like incorrect conditional
expressions. See PEP 308(1) for some examples. If you put parentheses
around your conditional expressions, you won’t run into this case.
See also
........
PEP 308(2) - Conditional Expressions
PEP written by Guido van Rossum and Raymond D. Hettinger;
implemented by Thomas Wouters.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0308/
(2) https://peps.python.org/pep-0308/
File: python.info, Node: PEP 309 Partial Function Application, Next: PEP 314 Metadata for Python Software Packages v1 1, Prev: PEP 308 Conditional Expressions, Up: What’s New in Python 2 5
1.17.2 PEP 309: Partial Function Application
--------------------------------------------
The *note functools: 5f. module is intended to contain tools for
functional-style programming.
One useful tool in this module is the ‘partial()’ function. For
programs written in a functional style, you’ll sometimes want to
construct variants of existing functions that have some of the
parameters filled in. Consider a Python function ‘f(a, b, c)’; you
could create a new function ‘g(b, c)’ that was equivalent to ‘f(1, b,
c)’. This is called “partial function application”.
‘partial()’ takes the arguments ‘(function, arg1, arg2, ...
kwarg1=value1, kwarg2=value2)’. The resulting object is callable, so
you can just call it to invoke 'function' with the filled-in arguments.
Here’s a small but realistic example:
import functools
def log (message, subsystem):
"Write the contents of 'message' to the specified subsystem."
print '%s: %s' % (subsystem, message)
...
server_log = functools.partial(log, subsystem='server')
server_log('Unable to open socket')
Here’s another example, from a program that uses PyGTK. Here a
context-sensitive pop-up menu is being constructed dynamically. The
callback provided for the menu option is a partially applied version of
the ‘open_item()’ method, where the first argument has been provided.
...
class Application:
def open_item(self, path):
...
def init (self):
open_func = functools.partial(self.open_item, item_path)
popup_menu.append( ("Open", open_func, 1) )
Another function in the *note functools: 5f. module is the
‘update_wrapper(wrapper, wrapped)’ function that helps you write
well-behaved decorators. ‘update_wrapper()’ copies the name, module,
and docstring attribute to a wrapper function so that tracebacks inside
the wrapped function are easier to understand. For example, you might
write:
def my_decorator(f):
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
functools.update_wrapper(wrapper, f)
return wrapper
‘wraps()’ is a decorator that can be used inside your own decorators to
copy the wrapped function’s information. An alternate version of the
previous example would be:
def my_decorator(f):
@functools.wraps(f)
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
return wrapper
See also
........
PEP 309(1) - Partial Function Application
PEP proposed and written by Peter Harris; implemented by Hye-Shik
Chang and Nick Coghlan, with adaptations by Raymond Hettinger.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0309/
File: python.info, Node: PEP 314 Metadata for Python Software Packages v1 1, Next: PEP 328 Absolute and Relative Imports, Prev: PEP 309 Partial Function Application, Up: What’s New in Python 2 5
1.17.3 PEP 314: Metadata for Python Software Packages v1.1
----------------------------------------------------------
Some simple dependency support was added to Distutils. The ‘setup()’
function now has ‘requires’, ‘provides’, and ‘obsoletes’ keyword
parameters. When you build a source distribution using the ‘sdist’
command, the dependency information will be recorded in the ‘PKG-INFO’
file.
Another new keyword parameter is ‘download_url’, which should be set to
a URL for the package’s source code. This means it’s now possible to
look up an entry in the package index, determine the dependencies for a
package, and download the required packages.
VERSION = '1.0'
setup(name='PyPackage',
version=VERSION,
requires=['numarray', 'zlib (>=1.1.4)'],
obsoletes=['OldPackage']
download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz'
% VERSION),
)
Another new enhancement to the Python package index at
‘https://pypi.org’ is storing source and binary archives for a package.
The new ‘upload’ Distutils command will upload a package to the
repository.
Before a package can be uploaded, you must be able to build a
distribution using the ‘sdist’ Distutils command. Once that works, you
can run ‘python setup.py upload’ to add your package to the PyPI
archive. Optionally you can GPG-sign the package by supplying the
‘--sign’ and ‘--identity’ options.
Package uploading was implemented by Martin von Löwis and Richard Jones.
See also
........
PEP 314(1) - Metadata for Python Software Packages v1.1
PEP proposed and written by A.M. Kuchling, Richard Jones, and Fred
Drake; implemented by Richard Jones and Fred Drake.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0314/
File: python.info, Node: PEP 328 Absolute and Relative Imports, Next: PEP 338 Executing Modules as Scripts, Prev: PEP 314 Metadata for Python Software Packages v1 1, Up: What’s New in Python 2 5
1.17.4 PEP 328: Absolute and Relative Imports
---------------------------------------------
The simpler part of PEP 328(1) was implemented in Python 2.4:
parentheses could now be used to enclose the names imported from a
module using the ‘from ... import ...’ statement, making it easier to
import many different names.
The more complicated part has been implemented in Python 2.5: importing
a module can be specified to use absolute or package-relative imports.
The plan is to move toward making absolute imports the default in future
versions of Python.
Let’s say you have a package directory like this:
pkg/
pkg/__init__.py
pkg/main.py
pkg/string.py
This defines a package named ‘pkg’ containing the ‘pkg.main’ and
‘pkg.string’ submodules.
Consider the code in the ‘main.py’ module. What happens if it executes
the statement ‘import string’? In Python 2.4 and earlier, it will first
look in the package’s directory to perform a relative import, finds
‘pkg/string.py’, imports the contents of that file as the ‘pkg.string’
module, and that module is bound to the name ‘string’ in the ‘pkg.main’
module’s namespace.
That’s fine if ‘pkg.string’ was what you wanted. But what if you wanted
Python’s standard *note string: d3. module? There’s no clean way to
ignore ‘pkg.string’ and look for the standard module; generally you had
to look at the contents of ‘sys.modules’, which is slightly unclean.
Holger Krekel’s ‘py.std’ package provides a tidier way to perform
imports from the standard library, ‘import py; py.std.string.join()’,
but that package isn’t available on all Python installations.
Reading code which relies on relative imports is also less clear,
because a reader may be confused about which module, *note string: d3.
or ‘pkg.string’, is intended to be used. Python users soon learned not
to duplicate the names of standard library modules in the names of their
packages’ submodules, but you can’t protect against having your
submodule’s name being used for a new module added in a future version
of Python.
In Python 2.5, you can switch *note import: 5de.’s behaviour to absolute
imports using a ‘from __future__ import absolute_import’ directive.
This absolute-import behaviour will become the default in a future
version (probably Python 2.7). Once absolute imports are the default,
‘import string’ will always find the standard library’s version. It’s
suggested that users should begin using absolute imports as much as
possible, so it’s preferable to begin writing ‘from pkg import string’
in your code.
Relative imports are still possible by adding a leading period to the
module name when using the ‘from ... import’ form:
# Import names from pkg.string
from .string import name1, name2
# Import pkg.string
from . import string
This imports the *note string: d3. module relative to the current
package, so in ‘pkg.main’ this will import 'name1' and 'name2' from
‘pkg.string’. Additional leading periods perform the relative import
starting from the parent of the current package. For example, code in
the ‘A.B.C’ module can do:
from . import D # Imports A.B.D
from .. import E # Imports A.E
from ..F import G # Imports A.F.G
Leading periods cannot be used with the ‘import modname’ form of the
import statement, only the ‘from ... import’ form.
See also
........
PEP 328(2) - Imports: Multi-Line and Absolute/Relative
PEP written by Aahz; implemented by Thomas Wouters.
‘https://pylib.readthedocs.io/’
The py library by Holger Krekel, which contains the ‘py.std’
package.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0328/
(2) https://peps.python.org/pep-0328/
File: python.info, Node: PEP 338 Executing Modules as Scripts, Next: PEP 341 Unified try/except/finally, Prev: PEP 328 Absolute and Relative Imports, Up: What’s New in Python 2 5
1.17.5 PEP 338: Executing Modules as Scripts
--------------------------------------------
The *note -m: 5dd. switch added in Python 2.4 to execute a module as a
script gained a few more abilities. Instead of being implemented in C
code inside the Python interpreter, the switch now uses an
implementation in a new module, *note runpy: be.
The *note runpy: be. module implements a more sophisticated import
mechanism so that it’s now possible to run modules in a package such as
‘pychecker.checker’. The module also supports alternative import
mechanisms such as the *note zipimport: 132. module. This means you can
add a .zip archive’s path to ‘sys.path’ and then use the *note -m: 5dd.
switch to execute code from the archive.
See also
........
PEP 338(1) - Executing modules as scripts
PEP written and implemented by Nick Coghlan.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0338/
File: python.info, Node: PEP 341 Unified try/except/finally, Next: PEP 342 New Generator Features, Prev: PEP 338 Executing Modules as Scripts, Up: What’s New in Python 2 5
1.17.6 PEP 341: Unified try/except/finally
------------------------------------------
Until Python 2.5, the *note try: 6e4. statement came in two flavours.
You could use a *note finally: 9ca. block to ensure that code is always
executed, or one or more *note except: 18b. blocks to catch specific
exceptions. You couldn’t combine both ‘except’ blocks and a ‘finally’
block, because generating the right bytecode for the combined version
was complicated and it wasn’t clear what the semantics of the combined
statement should be.
Guido van Rossum spent some time working with Java, which does support
the equivalent of combining *note except: 18b. blocks and a *note
finally: 9ca. block, and this clarified what the statement should mean.
In Python 2.5, you can now write:
try:
block-1 ...
except Exception1:
handler-1 ...
except Exception2:
handler-2 ...
else:
else-block
finally:
final-block
The code in 'block-1' is executed. If the code raises an exception, the
various *note except: 18b. blocks are tested: if the exception is of
class ‘Exception1’, 'handler-1' is executed; otherwise if it’s of class
‘Exception2’, 'handler-2' is executed, and so forth. If no exception is
raised, the 'else-block' is executed.
No matter what happened previously, the 'final-block' is executed once
the code block is complete and any raised exceptions handled. Even if
there’s an error in an exception handler or the 'else-block' and a new
exception is raised, the code in the 'final-block' is still run.
See also
........
PEP 341(1) - Unifying try-except and try-finally
PEP written by Georg Brandl; implementation by Thomas Lee.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0341/
File: python.info, Node: PEP 342 New Generator Features, Next: PEP 343 The ‘with’ statement<2>, Prev: PEP 341 Unified try/except/finally, Up: What’s New in Python 2 5
1.17.7 PEP 342: New Generator Features
--------------------------------------
Python 2.5 adds a simple way to pass values 'into' a generator. As
introduced in Python 2.3, generators only produce output; once a
generator’s code was invoked to create an iterator, there was no way to
pass any new information into the function when its execution is
resumed. Sometimes the ability to pass in some information would be
useful. Hackish solutions to this include making the generator’s code
look at a global variable and then changing the global variable’s value,
or passing in some mutable object that callers then modify.
To refresh your memory of basic generators, here’s a simple example:
def counter (maximum):
i = 0
while i < maximum:
yield i
i += 1
When you call ‘counter(10)’, the result is an iterator that returns the
values from 0 up to 9. On encountering the *note yield: 9cd. statement,
the iterator returns the provided value and suspends the function’s
execution, preserving the local variables. Execution resumes on the
following call to the iterator’s *note next(): 7d3. method, picking up
after the ‘yield’ statement.
In Python 2.3, *note yield: 9cd. was a statement; it didn’t return any
value. In 2.5, ‘yield’ is now an expression, returning a value that can
be assigned to a variable or otherwise operated on:
val = (yield i)
I recommend that you always put parentheses around a *note yield: 9cd.
expression when you’re doing something with the returned value, as in
the above example. The parentheses aren’t always necessary, but it’s
easier to always add them instead of having to remember when they’re
needed.
( PEP 342(1) explains the exact rules, which are that a *note yield:
9cd.-expression must always be parenthesized except when it occurs at
the top-level expression on the right-hand side of an assignment. This
means you can write ‘val = yield i’ but have to use parentheses when
there’s an operation, as in ‘val = (yield i) + 12’.)
Values are sent into a generator by calling its ‘send(value)’ method.
The generator’s code is then resumed and the *note yield: 9cd.
expression returns the specified 'value'. If the regular *note next():
7d3. method is called, the ‘yield’ returns *note None: 671.
Here’s the previous example, modified to allow changing the value of the
internal counter.
def counter (maximum):
i = 0
while i < maximum:
val = (yield i)
# If value provided, change counter
if val is not None:
i = val
else:
i += 1
And here’s an example of changing the counter:
>>> it = counter(10)
>>> print it.next()
0
>>> print it.next()
1
>>> print it.send(8)
8
>>> print it.next()
9
>>> print it.next()
Traceback (most recent call last):
File "t.py", line 15, in ?
print it.next()
StopIteration
*note yield: 9cd. will usually return *note None: 671, so you should
always check for this case. Don’t just use its value in expressions
unless you’re sure that the ‘send()’ method will be the only method used
to resume your generator function.
In addition to ‘send()’, there are two other new methods on generators:
* ‘throw(type, value=None, traceback=None)’ is used to raise an
exception inside the generator; the exception is raised by the
*note yield: 9cd. expression where the generator’s execution is
paused.
* ‘close()’ raises a new *note GeneratorExit: 1393. exception inside
the generator to terminate the iteration. On receiving this
exception, the generator’s code must either raise *note
GeneratorExit: 1393. or *note StopIteration: bfa. Catching the
*note GeneratorExit: 1393. exception and returning a value is
illegal and will trigger a *note RuntimeError: 195.; if the
function raises some other exception, that exception is propagated
to the caller. ‘close()’ will also be called by Python’s garbage
collector when the generator is garbage-collected.
If you need to run cleanup code when a *note GeneratorExit: 1393.
occurs, I suggest using a ‘try: ... finally:’ suite instead of
catching *note GeneratorExit: 1393.
The cumulative effect of these changes is to turn generators from
one-way producers of information into both producers and consumers.
Generators also become 'coroutines', a more generalized form of
subroutines. Subroutines are entered at one point and exited at another
point (the top of the function, and a *note return: 9ce. statement), but
coroutines can be entered, exited, and resumed at many different points
(the *note yield: 9cd. statements). We’ll have to figure out patterns
for using coroutines effectively in Python.
The addition of the ‘close()’ method has one side effect that isn’t
obvious. ‘close()’ is called when a generator is garbage-collected, so
this means the generator’s code gets one last chance to run before the
generator is destroyed. This last chance means that ‘try...finally’
statements in generators can now be guaranteed to work; the *note
finally: 9ca. clause will now always get a chance to run. The syntactic
restriction that you couldn’t mix *note yield: 9cd. statements with a
‘try...finally’ suite has therefore been removed. This seems like a
minor bit of language trivia, but using generators and ‘try...finally’
is actually necessary in order to implement the *note with: 5ce.
statement described by PEP 343(2). I’ll look at this new statement in
the following section.
Another even more esoteric effect of this change: previously, the
‘gi_frame’ attribute of a generator was always a frame object. It’s now
possible for ‘gi_frame’ to be ‘None’ once the generator has been
exhausted.
See also
........
PEP 342(3) - Coroutines via Enhanced Generators
PEP written by Guido van Rossum and Phillip J. Eby; implemented by
Phillip J. Eby. Includes examples of some fancier uses of
generators as coroutines.
Earlier versions of these features were proposed in PEP 288(4) by
Raymond Hettinger and PEP 325(5) by Samuele Pedroni.
‘https://en.wikipedia.org/wiki/Coroutine’
The Wikipedia entry for coroutines.
‘https://web.archive.org/web/20160321211320/http://www.sidhe.org/~dan/blog/archives/000178.html’
An explanation of coroutines from a Perl point of view, written by
Dan Sugalski.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0342/
(2) https://peps.python.org/pep-0343/
(3) https://peps.python.org/pep-0342/
(4) https://peps.python.org/pep-0288/
(5) https://peps.python.org/pep-0325/
File: python.info, Node: PEP 343 The ‘with’ statement<2>, Next: PEP 352 Exceptions as New-Style Classes, Prev: PEP 342 New Generator Features, Up: What’s New in Python 2 5
1.17.8 PEP 343: The ‘with’ statement
------------------------------------
The ‘*note with: 5ce.’ statement clarifies code that previously would
use ‘try...finally’ blocks to ensure that clean-up code is executed. In
this section, I’ll discuss the statement as it will commonly be used.
In the next section, I’ll examine the implementation details and show
how to write objects for use with this statement.
The ‘*note with: 5ce.’ statement is a new control-flow structure whose
basic structure is:
with expression [as variable]:
with-block
The expression is evaluated, and it should result in an object that
supports the context management protocol (that is, has *note
__enter__(): 5c4. and *note __exit__(): 12f3. methods.
The object’s *note __enter__(): 5c4. is called before 'with-block' is
executed and therefore can run set-up code. It also may return a value
that is bound to the name 'variable', if given. (Note carefully that
'variable' is 'not' assigned the result of 'expression'.)
After execution of the 'with-block' is finished, the object’s *note
__exit__(): 12f3. method is called, even if the block raised an
exception, and can therefore run clean-up code.
To enable the statement in Python 2.5, you need to add the following
directive to your module:
from __future__ import with_statement
The statement will always be enabled in Python 2.6.
Some standard Python objects now support the context management protocol
and can be used with the ‘*note with: 5ce.’ statement. File objects are
one example:
with open('/etc/passwd', 'r') as f:
for line in f:
print line
... more processing code ...
After this statement has executed, the file object in 'f' will have been
automatically closed, even if the *note for: 2ec. loop raised an
exception part-way through the block.
Note: In this case, 'f' is the same object created by *note open():
517, because *note __enter__(): 5c4. returns 'self'.
The *note threading: ed. module’s locks and condition variables also
support the ‘*note with: 5ce.’ statement:
lock = threading.Lock()
with lock:
# Critical section of code
...
The lock is acquired before the block is executed and always released
once the block is complete.
The new ‘localcontext()’ function in the *note decimal: 36. module makes
it easy to save and restore the current decimal context, which
encapsulates the desired precision and rounding characteristics for
computations:
from decimal import Decimal, Context, localcontext
# Displays with default precision of 28 digits
v = Decimal('578')
print v.sqrt()
with localcontext(Context(prec=16)):
# All code in this block uses a precision of 16 digits.
# The original context is restored on exiting the block.
print v.sqrt()
* Menu:
* Writing Context Managers: Writing Context Managers<2>.
* The contextlib module: The contextlib module<2>.
File: python.info, Node: Writing Context Managers<2>, Next: The contextlib module<2>, Up: PEP 343 The ‘with’ statement<2>
1.17.8.1 Writing Context Managers
.................................
Under the hood, the ‘*note with: 5ce.’ statement is fairly complicated.
Most people will only use ‘‘with’’ in company with existing objects and
don’t need to know these details, so you can skip the rest of this
section if you like. Authors of new objects will need to understand the
details of the underlying implementation and should keep reading.
A high-level explanation of the context management protocol is:
* The expression is evaluated and should result in an object called a
“context manager”. The context manager must have *note
__enter__(): 5c4. and *note __exit__(): 12f3. methods.
* The context manager’s *note __enter__(): 5c4. method is called.
The value returned is assigned to 'VAR'. If no ‘'as VAR'’ clause is
present, the value is simply discarded.
* The code in 'BLOCK' is executed.
* If 'BLOCK' raises an exception, the ‘__exit__(type, value,
traceback)’ is called with the exception details, the same values
returned by *note sys.exc_info(): 686. The method’s return value
controls whether the exception is re-raised: any false value
re-raises the exception, and ‘True’ will result in suppressing it.
You’ll only rarely want to suppress the exception, because if you
do the author of the code containing the ‘*note with: 5ce.’
statement will never realize anything went wrong.
* If 'BLOCK' didn’t raise an exception, the *note __exit__(): 12f3.
method is still called, but 'type', 'value', and 'traceback' are
all ‘None’.
Let’s think through an example. I won’t present detailed code but will
only sketch the methods necessary for a database that supports
transactions.
(For people unfamiliar with database terminology: a set of changes to
the database are grouped into a transaction. Transactions can be either
committed, meaning that all the changes are written into the database,
or rolled back, meaning that the changes are all discarded and the
database is unchanged. See any database textbook for more information.)
Let’s assume there’s an object representing a database connection. Our
goal will be to let the user write code like this:
db_connection = DatabaseConnection()
with db_connection as cursor:
cursor.execute('insert into ...')
cursor.execute('delete from ...')
# ... more operations ...
The transaction should be committed if the code in the block runs
flawlessly or rolled back if there’s an exception. Here’s the basic
interface for ‘DatabaseConnection’ that I’ll assume:
class DatabaseConnection:
# Database interface
def cursor (self):
"Returns a cursor object and starts a new transaction"
def commit (self):
"Commits current transaction"
def rollback (self):
"Rolls back current transaction"
The *note __enter__(): 5c4. method is pretty easy, having only to start
a new transaction. For this application the resulting cursor object
would be a useful result, so the method will return it. The user can
then add ‘as cursor’ to their ‘*note with: 5ce.’ statement to bind the
cursor to a variable name.
class DatabaseConnection:
...
def __enter__ (self):
# Code to start a new transaction
cursor = self.cursor()
return cursor
The *note __exit__(): 12f3. method is the most complicated because it’s
where most of the work has to be done. The method has to check if an
exception occurred. If there was no exception, the transaction is
committed. The transaction is rolled back if there was an exception.
In the code below, execution will just fall off the end of the function,
returning the default value of ‘None’. ‘None’ is false, so the
exception will be re-raised automatically. If you wished, you could be
more explicit and add a *note return: 9ce. statement at the marked
location.
class DatabaseConnection:
...
def __exit__ (self, type, value, tb):
if tb is None:
# No exception, so commit
self.commit()
else:
# Exception occurred, so rollback.
self.rollback()
# return False
File: python.info, Node: The contextlib module<2>, Prev: Writing Context Managers<2>, Up: PEP 343 The ‘with’ statement<2>
1.17.8.2 The contextlib module
..............................
The new *note contextlib: 23. module provides some functions and a
decorator that are useful for writing objects for use with the ‘*note
with: 5ce.’ statement.
The decorator is called ‘contextmanager()’, and lets you write a single
generator function instead of defining a new class. The generator
should yield exactly one value. The code up to the *note yield: 9cd.
will be executed as the *note __enter__(): 5c4. method, and the value
yielded will be the method’s return value that will get bound to the
variable in the ‘*note with: 5ce.’ statement’s ‘as’ clause, if any. The
code after the *note yield: 9cd. will be executed in the *note
__exit__(): 12f3. method. Any exception raised in the block will be
raised by the ‘yield’ statement.
Our database example from the previous section could be written using
this decorator as:
from contextlib import contextmanager
@contextmanager
def db_transaction (connection):
cursor = connection.cursor()
try:
yield cursor
except:
connection.rollback()
raise
else:
connection.commit()
db = DatabaseConnection()
with db_transaction(db) as cursor:
...
The *note contextlib: 23. module also has a ‘nested(mgr1, mgr2, ...)’
function that combines a number of context managers so you don’t need to
write nested ‘*note with: 5ce.’ statements. In this example, the single
‘‘with’’ statement both starts a database transaction and acquires a
thread lock:
lock = threading.Lock()
with nested (db_transaction(db), lock) as (cursor, locked):
...
Finally, the ‘closing(object)’ function returns 'object' so that it can
be bound to a variable, and calls ‘object.close’ at the end of the
block.
import urllib, sys
from contextlib import closing
with closing(urllib.urlopen('http://www.yahoo.com')) as f:
for line in f:
sys.stdout.write(line)
See also
........
PEP 343(1) - The “with” statement
PEP written by Guido van Rossum and Nick Coghlan; implemented by
Mike Bland, Guido van Rossum, and Neal Norwitz. The PEP shows the
code generated for a ‘*note with: 5ce.’ statement, which can be
helpful in learning how the statement works.
The documentation for the *note contextlib: 23. module.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0343/
File: python.info, Node: PEP 352 Exceptions as New-Style Classes, Next: PEP 353 Using ssize_t as the index type, Prev: PEP 343 The ‘with’ statement<2>, Up: What’s New in Python 2 5
1.17.9 PEP 352: Exceptions as New-Style Classes
-----------------------------------------------
Exception classes can now be new-style classes, not just classic
classes, and the built-in *note Exception: 9d9. class and all the
standard built-in exceptions (*note NameError: 43a, *note ValueError:
204, etc.) are now new-style classes.
The inheritance hierarchy for exceptions has been rearranged a bit. In
2.5, the inheritance relationships are:
BaseException # New in Python 2.5
|- KeyboardInterrupt
|- SystemExit
|- Exception
|- (all other current built-in exceptions)
This rearrangement was done because people often want to catch all
exceptions that indicate program errors. *note KeyboardInterrupt: 9d1.
and *note SystemExit: d40. aren’t errors, though, and usually represent
an explicit action such as the user hitting ‘Control’-‘C’ or code
calling *note sys.exit(): 133c. A bare ‘except:’ will catch all
exceptions, so you commonly need to list *note KeyboardInterrupt: 9d1.
and *note SystemExit: d40. in order to re-raise them. The usual pattern
is:
try:
...
except (KeyboardInterrupt, SystemExit):
raise
except:
# Log error...
# Continue running program...
In Python 2.5, you can now write ‘except Exception’ to achieve the same
result, catching all the exceptions that usually indicate errors but
leaving *note KeyboardInterrupt: 9d1. and *note SystemExit: d40. alone.
As in previous versions, a bare ‘except:’ still catches all exceptions.
The goal for Python 3.0 is to require any class raised as an exception
to derive from *note BaseException: 5b7. or some descendant of *note
BaseException: 5b7, and future releases in the Python 2.x series may
begin to enforce this constraint. Therefore, I suggest you begin making
all your exception classes derive from *note Exception: 9d9. now. It’s
been suggested that the bare ‘except:’ form should be removed in Python
3.0, but Guido van Rossum hasn’t decided whether to do this or not.
Raising of strings as exceptions, as in the statement ‘raise "Error
occurred"’, is deprecated in Python 2.5 and will trigger a warning. The
aim is to be able to remove the string-exception feature in a few
releases.
See also
........
PEP 352(1) - Required Superclass for Exceptions
PEP written by Brett Cannon and Guido van Rossum; implemented by
Brett Cannon.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0352/
File: python.info, Node: PEP 353 Using ssize_t as the index type, Next: PEP 357 The ‘__index__’ method, Prev: PEP 352 Exceptions as New-Style Classes, Up: What’s New in Python 2 5
1.17.10 PEP 353: Using ssize_t as the index type
------------------------------------------------
A wide-ranging change to Python’s C API, using a new *note Py_ssize_t:
a5f. type definition instead of int, will permit the interpreter to
handle more data on 64-bit platforms. This change doesn’t affect
Python’s capacity on 32-bit platforms.
Various pieces of the Python interpreter used C’s int type to store
sizes or counts; for example, the number of items in a list or tuple
were stored in an int. The C compilers for most 64-bit platforms still
define int as a 32-bit type, so that meant that lists could only hold up
to ‘2**31 - 1’ = 2147483647 items. (There are actually a few different
programming models that 64-bit C compilers can use – see
‘https://unix.org/version2/whatsnew/lp64_wp.html’ for a discussion – but
the most commonly available model leaves int as 32 bits.)
A limit of 2147483647 items doesn’t really matter on a 32-bit platform
because you’ll run out of memory before hitting the length limit. Each
list item requires space for a pointer, which is 4 bytes, plus space for
a *note PyObject: 334. representing the item. 2147483647*4 is already
more bytes than a 32-bit address space can contain.
It’s possible to address that much memory on a 64-bit platform, however.
The pointers for a list that size would only require 16 GiB of space, so
it’s not unreasonable that Python programmers might construct lists that
large. Therefore, the Python interpreter had to be changed to use some
type other than int, and this will be a 64-bit type on 64-bit platforms.
The change will cause incompatibilities on 64-bit machines, so it was
deemed worth making the transition now, while the number of 64-bit users
is still relatively small. (In 5 or 10 years, we may 'all' be on 64-bit
machines, and the transition would be more painful then.)
This change most strongly affects authors of C extension modules.
Python strings and container types such as lists and tuples now use
*note Py_ssize_t: a5f. to store their size. Functions such as *note
PyList_Size(): 13e5. now return *note Py_ssize_t: a5f. Code in
extension modules may therefore need to have some variables changed to
*note Py_ssize_t: a5f.
The *note PyArg_ParseTuple(): 56e. and *note Py_BuildValue(): 8a6.
functions have a new conversion code, ‘n’, for *note Py_ssize_t: a5f.
*note PyArg_ParseTuple(): 56e.’s ‘s#’ and ‘t#’ still output int by
default, but you can define the macro ‘PY_SSIZE_T_CLEAN’ before
including ‘Python.h’ to make them return *note Py_ssize_t: a5f.
PEP 353(1) has a section on conversion guidelines that extension authors
should read to learn about supporting 64-bit platforms.
See also
........
PEP 353(2) - Using ssize_t as the index type
PEP written and implemented by Martin von Löwis.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0353/
(2) https://peps.python.org/pep-0353/
File: python.info, Node: PEP 357 The ‘__index__’ method, Next: Other Language Changes<16>, Prev: PEP 353 Using ssize_t as the index type, Up: What’s New in Python 2 5
1.17.11 PEP 357: The ‘__index__’ method
---------------------------------------
The NumPy developers had a problem that could only be solved by adding a
new special method, ‘__index__()’. When using slice notation, as in
‘[start:stop:step]’, the values of the 'start', 'stop', and 'step'
indexes must all be either integers or long integers. NumPy defines a
variety of specialized integer types corresponding to unsigned and
signed integers of 8, 16, 32, and 64 bits, but there was no way to
signal that these types could be used as slice indexes.
Slicing can’t just use the existing ‘__int__()’ method because that
method is also used to implement coercion to integers. If slicing used
‘__int__()’, floating-point numbers would also become legal slice
indexes and that’s clearly an undesirable behaviour.
Instead, a new special method called ‘__index__()’ was added. It takes
no arguments and returns an integer giving the slice index to use. For
example:
class C:
def __index__ (self):
return self.value
The return value must be either a Python integer or long integer. The
interpreter will check that the type returned is correct, and raises a
*note TypeError: 534. if this requirement isn’t met.
A corresponding *note nb_index: 13e8. slot was added to the C-level
*note PyNumberMethods: 13e9. structure to let C extensions implement
this protocol. ‘PyNumber_Index(obj)’ can be used in extension code to
call the ‘__index__()’ function and retrieve its result.
See also
........
PEP 357(1) - Allowing Any Object to be Used for Slicing
PEP written and implemented by Travis Oliphant.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0357/
File: python.info, Node: Other Language Changes<16>, Next: New Improved and Removed Modules, Prev: PEP 357 The ‘__index__’ method, Up: What’s New in Python 2 5
1.17.12 Other Language Changes
------------------------------
Here are all of the changes that Python 2.5 makes to the core Python
language.
* The *note dict: 258. type has a new hook for letting subclasses
provide a default value when a key isn’t contained in the
dictionary. When a key isn’t found, the dictionary’s
‘__missing__(key)’ method will be called. This hook is used to
implement the new ‘defaultdict’ class in the *note collections: 1d.
module. The following example defines a dictionary that returns
zero for any missing key:
class zerodict (dict):
def __missing__ (self, key):
return 0
d = zerodict({1:1, 2:2})
print d[1], d[2] # Prints 1, 2
print d[3], d[4] # Prints 0, 0
* Both 8-bit and Unicode strings have new ‘partition(sep)’ and
‘rpartition(sep)’ methods that simplify a common use case.
The ‘find(S)’ method is often used to get an index which is then
used to slice the string and obtain the pieces that are before and
after the separator. ‘partition(sep)’ condenses this pattern into
a single method call that returns a 3-tuple containing the
substring before the separator, the separator itself, and the
substring after the separator. If the separator isn’t found, the
first element of the tuple is the entire string and the other two
elements are empty. ‘rpartition(sep)’ also returns a 3-tuple but
starts searching from the end of the string; the ‘r’ stands for
‘reverse’.
Some examples:
>>> ('http://www.python.org').partition('://')
('http', '://', 'www.python.org')
>>> ('file:/usr/share/doc/index.html').partition('://')
('file:/usr/share/doc/index.html', '', '')
>>> (u'Subject: a quick question').partition(':')
(u'Subject', u':', u' a quick question')
>>> 'www.python.org'.rpartition('.')
('www.python', '.', 'org')
>>> 'www.python.org'.rpartition(':')
('', '', 'www.python.org')
(Implemented by Fredrik Lundh following a suggestion by Raymond
Hettinger.)
* The ‘startswith()’ and ‘endswith()’ methods of string types now
accept tuples of strings to check for.
def is_image_file (filename):
return filename.endswith(('.gif', '.jpg', '.tiff'))
(Implemented by Georg Brandl following a suggestion by Tom Lynn.)
* The *note min(): f03. and *note max(): f04. built-in functions
gained a ‘key’ keyword parameter analogous to the ‘key’ argument
for ‘sort()’. This parameter supplies a function that takes a
single argument and is called for every value in the list; *note
min(): f03./*note max(): f04. will return the element with the
smallest/largest return value from this function. For example, to
find the longest string in a list, you can do:
L = ['medium', 'longest', 'short']
# Prints 'longest'
print max(L, key=len)
# Prints 'short', because lexicographically 'short' has the largest value
print max(L)
(Contributed by Steven Bethard and Raymond Hettinger.)
* Two new built-in functions, *note any(): 13ec. and *note all():
13ed, evaluate whether an iterator contains any true or false
values. *note any(): 13ec. returns *note True: c0d. if any value
returned by the iterator is true; otherwise it will return *note
False: b37. *note all(): 13ed. returns *note True: c0d. only if
all of the values returned by the iterator evaluate as true.
(Suggested by Guido van Rossum, and implemented by Raymond
Hettinger.)
* The result of a class’s ‘__hash__()’ method can now be either a
long integer or a regular integer. If a long integer is returned,
the hash of that value is taken. In earlier versions the hash
value was required to be a regular integer, but in 2.5 the *note
id(): 13ee. built-in was changed to always return non-negative
numbers, and users often seem to use ‘id(self)’ in ‘__hash__()’
methods (though this is discouraged).
* ASCII is now the default encoding for modules. It’s now a syntax
error if a module contains string literals with 8-bit characters
but doesn’t have an encoding declaration. In Python 2.4 this
triggered a warning, not a syntax error. See PEP 263(1) for how to
declare a module’s encoding; for example, you might add a line like
this near the top of the source file:
# -*- coding: latin1 -*-
* A new warning, *note UnicodeWarning: 13ef, is triggered when you
attempt to compare a Unicode string and an 8-bit string that can’t
be converted to Unicode using the default ASCII encoding. The
result of the comparison is false:
>>> chr(128) == unichr(128) # Can't convert chr(128) to Unicode
__main__:1: UnicodeWarning: Unicode equal comparison failed
to convert both arguments to Unicode - interpreting them
as being unequal
False
>>> chr(127) == unichr(127) # chr(127) can be converted
True
Previously this would raise a *note UnicodeDecodeError: a12.
exception, but in 2.5 this could result in puzzling problems when
accessing a dictionary. If you looked up ‘unichr(128)’ and
‘chr(128)’ was being used as a key, you’d get a *note
UnicodeDecodeError: a12. exception. Other changes in 2.5 resulted
in this exception being raised instead of suppressed by the code in
‘dictobject.c’ that implements dictionaries.
Raising an exception for such a comparison is strictly correct, but
the change might have broken code, so instead *note UnicodeWarning:
13ef. was introduced.
(Implemented by Marc-André Lemburg.)
* One error that Python programmers sometimes make is forgetting to
include an ‘__init__.py’ module in a package directory. Debugging
this mistake can be confusing, and usually requires running Python
with the *note -v: 13f0. switch to log all the paths searched. In
Python 2.5, a new *note ImportWarning: 4f6. warning is triggered
when an import would have picked up a directory as a package but no
‘__init__.py’ was found. This warning is silently ignored by
default; provide the *note -Wd: 8c6. option when running the Python
executable to display the warning message. (Implemented by Thomas
Wouters.)
* The list of base classes in a class definition can now be empty.
As an example, this is now legal:
class C():
pass
(Implemented by Brett Cannon.)
* Menu:
* Interactive Interpreter Changes::
* Optimizations: Optimizations<15>.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0263/
File: python.info, Node: Interactive Interpreter Changes, Next: Optimizations<15>, Up: Other Language Changes<16>
1.17.12.1 Interactive Interpreter Changes
.........................................
In the interactive interpreter, ‘quit’ and ‘exit’ have long been strings
so that new users get a somewhat helpful message when they try to quit:
>>> quit
'Use Ctrl-D (i.e. EOF) to exit.'
In Python 2.5, ‘quit’ and ‘exit’ are now objects that still produce
string representations of themselves, but are also callable. Newbies
who try ‘quit()’ or ‘exit()’ will now exit the interpreter as they
expect. (Implemented by Georg Brandl.)
The Python executable now accepts the standard long options *note -help:
5ec. and *note -version: d1b.; on Windows, it also accepts the *note /?:
13f3. option for displaying a help message. (Implemented by Georg
Brandl.)
File: python.info, Node: Optimizations<15>, Prev: Interactive Interpreter Changes, Up: Other Language Changes<16>
1.17.12.2 Optimizations
.......................
Several of the optimizations were developed at the NeedForSpeed sprint,
an event held in Reykjavik, Iceland, from May 21–28 2006. The sprint
focused on speed enhancements to the CPython implementation and was
funded by EWT LLC with local support from CCP Games. Those
optimizations added at this sprint are specially marked in the following
list.
* When they were introduced in Python 2.4, the built-in *note set:
5d5. and *note frozenset: 5d6. types were built on top of Python’s
dictionary type. In 2.5 the internal data structure has been
customized for implementing sets, and as a result sets will use a
third less memory and are somewhat faster. (Implemented by Raymond
Hettinger.)
* The speed of some Unicode operations, such as finding substrings,
string splitting, and character map encoding and decoding, has been
improved. (Substring search and splitting improvements were added
by Fredrik Lundh and Andrew Dalke at the NeedForSpeed sprint.
Character maps were improved by Walter Dörwald and Martin von
Löwis.)
* The ‘long(str, base)’ function is now faster on long digit strings
because fewer intermediate results are calculated. The peak is for
strings of around 800–1000 digits where the function is 6 times
faster. (Contributed by Alan McIntyre and committed at the
NeedForSpeed sprint.)
* It’s now illegal to mix iterating over a file with ‘for line in
file’ and calling the file object’s ‘read()’/*note readline():
ba./‘readlines()’ methods. Iteration uses an internal buffer and
the ‘read*()’ methods don’t use that buffer. Instead they would
return the data following the buffer, causing the data to appear
out of order. Mixing iteration and these methods will now trigger
a *note ValueError: 204. from the ‘read*()’ method. (Implemented
by Thomas Wouters.)
* The *note struct: d5. module now compiles structure format strings
into an internal representation and caches this representation,
yielding a 20% speedup. (Contributed by Bob Ippolito at the
NeedForSpeed sprint.)
* The *note re: b9. module got a 1 or 2% speedup by switching to
Python’s allocator functions instead of the system’s ‘malloc()’ and
‘free()’. (Contributed by Jack Diederich at the NeedForSpeed
sprint.)
* The code generator’s peephole optimizer now performs simple
constant folding in expressions. If you write something like ‘a =
2+3’, the code generator will do the arithmetic and produce code
corresponding to ‘a = 5’. (Proposed and implemented by Raymond
Hettinger.)
* Function calls are now faster because code objects now keep the
most recently finished frame (a “zombie frame”) in an internal
field of the code object, reusing it the next time the code object
is invoked. (Original patch by Michael Hudson, modified by Armin
Rigo and Richard Jones; committed at the NeedForSpeed sprint.)
Frame objects are also slightly smaller, which may improve cache
locality and reduce memory usage a bit. (Contributed by Neal
Norwitz.)
* Python’s built-in exceptions are now new-style classes, a change
that speeds up instantiation considerably. Exception handling in
Python 2.5 is therefore about 30% faster than in 2.4. (Contributed
by Richard Jones, Georg Brandl and Sean Reifschneider at the
NeedForSpeed sprint.)
* Importing now caches the paths tried, recording whether they exist
or not so that the interpreter makes fewer ‘open()’ and ‘stat()’
calls on startup. (Contributed by Martin von Löwis and Georg
Brandl.)
File: python.info, Node: New Improved and Removed Modules, Next: Build and C API Changes<10>, Prev: Other Language Changes<16>, Up: What’s New in Python 2 5
1.17.13 New, Improved, and Removed Modules
------------------------------------------
The standard library received many enhancements and bug fixes in Python
2.5. Here’s a partial list of the most notable changes, sorted
alphabetically by module name. Consult the ‘Misc/NEWS’ file in the
source tree for a more complete list of changes, or look through the SVN
logs for all the details.
* The ‘audioop’ module now supports the a-LAW encoding, and the code
for u-LAW encoding has been improved. (Contributed by Lars
Immisch.)
* The *note codecs: 1b. module gained support for incremental codecs.
The ‘codec.lookup()’ function now returns a ‘CodecInfo’ instance
instead of a tuple. ‘CodecInfo’ instances behave like a 4-tuple to
preserve backward compatibility but also have the attributes
‘encode’, ‘decode’, ‘incrementalencoder’, ‘incrementaldecoder’,
‘streamwriter’, and ‘streamreader’. Incremental codecs can receive
input and produce output in multiple chunks; the output is the same
as if the entire input was fed to the non-incremental codec. See
the *note codecs: 1b. module documentation for details. (Designed
and implemented by Walter Dörwald.)
* The *note collections: 1d. module gained a new type, ‘defaultdict’,
that subclasses the standard *note dict: 258. type. The new type
mostly behaves like a dictionary but constructs a default value
when a key isn’t present, automatically adding it to the dictionary
for the requested key value.
The first argument to ‘defaultdict’’s constructor is a factory
function that gets called whenever a key is requested but not
found. This factory function receives no arguments, so you can use
built-in type constructors such as *note list(): 60d. or *note
int(): 259. For example, you can make an index of words based on
their initial letter like this:
words = """Nel mezzo del cammin di nostra vita
mi ritrovai per una selva oscura
che la diritta via era smarrita""".lower().split()
index = defaultdict(list)
for w in words:
init_letter = w[0]
index[init_letter].append(w)
Printing ‘index’ results in the following output:
defaultdict(, {'c': ['cammin', 'che'], 'e': ['era'],
'd': ['del', 'di', 'diritta'], 'm': ['mezzo', 'mi'],
'l': ['la'], 'o': ['oscura'], 'n': ['nel', 'nostra'],
'p': ['per'], 's': ['selva', 'smarrita'],
'r': ['ritrovai'], 'u': ['una'], 'v': ['vita', 'via']}
(Contributed by Guido van Rossum.)
* The ‘deque’ double-ended queue type supplied by the *note
collections: 1d. module now has a ‘remove(value)’ method that
removes the first occurrence of 'value' in the queue, raising *note
ValueError: 204. if the value isn’t found. (Contributed by Raymond
Hettinger.)
* New module: The *note contextlib: 23. module contains helper
functions for use with the new ‘*note with: 5ce.’ statement. See
section *note The contextlib module: 13df. for more about this
module.
* New module: The *note cProfile: 27. module is a C implementation of
the existing *note profile: af. module that has much lower
overhead. The module’s interface is the same as *note profile:
af.: you run ‘cProfile.run('main()')’ to profile a function, can
save profile data to a file, etc. It’s not yet known if the
Hotshot profiler, which is also written in C but doesn’t match the
*note profile: af. module’s interface, will continue to be
maintained in future versions of Python. (Contributed by Armin
Rigo.)
Also, the *note pstats: b0. module for analyzing the data measured
by the profiler now supports directing the output to any file
object by supplying a 'stream' argument to the ‘Stats’ constructor.
(Contributed by Skip Montanaro.)
* The *note csv: 29. module, which parses files in comma-separated
value format, received several enhancements and a number of
bugfixes. You can now set the maximum size in bytes of a field by
calling the ‘csv.field_size_limit(new_limit)’ function; omitting
the 'new_limit' argument will return the currently set limit. The
‘reader’ class now has a ‘line_num’ attribute that counts the
number of physical lines read from the source; records can span
multiple physical lines, so ‘line_num’ is not the same as the
number of records read.
The CSV parser is now stricter about multi-line quoted fields.
Previously, if a line ended within a quoted field without a
terminating newline character, a newline would be inserted into the
returned field. This behavior caused problems when reading files
that contained carriage return characters within fields, so the
code was changed to return the field without inserting newlines.
As a consequence, if newlines embedded within fields are important,
the input should be split into lines in a manner that preserves the
newline characters.
(Contributed by Skip Montanaro and Andrew McNamara.)
* The *note datetime: 1cc. class in the *note datetime: 30. module
now has a ‘strptime(string, format)’ method for parsing date
strings, contributed by Josh Spoerri. It uses the same format
characters as *note time.strptime(): 13f8. and *note
time.strftime(): 11db.:
from datetime import datetime
ts = datetime.strptime('10:13:15 2006-03-07',
'%H:%M:%S %Y-%m-%d')
* The ‘SequenceMatcher.get_matching_blocks()’ method in the *note
difflib: 37. module now guarantees to return a minimal list of
blocks describing matching subsequences. Previously, the algorithm
would occasionally break a block of matching elements into two list
entries. (Enhancement by Tim Peters.)
* The *note doctest: 3a. module gained a ‘SKIP’ option that keeps an
example from being executed at all. This is intended for code
snippets that are usage examples intended for the reader and aren’t
actually test cases.
An 'encoding' parameter was added to the ‘testfile()’ function and
the ‘DocFileSuite’ class to specify the file’s encoding. This
makes it easier to use non-ASCII characters in tests contained
within a docstring. (Contributed by Bjorn Tillenius.)
* The *note email: 3b. package has been updated to version 4.0.
(Contributed by Barry Warsaw.)
* The *note fileinput: 5b. module was made more flexible. Unicode
filenames are now supported, and a 'mode' parameter that defaults
to ‘"r"’ was added to the *note input(): 12cb. function to allow
opening files in binary or *note universal newlines: d3a. mode.
Another new parameter, 'openhook', lets you use a function other
than *note open(): 517. to open the input files. Once you’re
iterating over the set of files, the ‘FileInput’ object’s new *note
fileno(): 13f9. returns the file descriptor for the currently
opened file. (Contributed by Georg Brandl.)
* In the *note gc: 60. module, the new ‘get_count()’ function returns
a 3-tuple containing the current collection counts for the three GC
generations. This is accounting information for the garbage
collector; when these counts reach a specified threshold, a garbage
collection sweep will be made. The existing *note gc.collect():
a39. function now takes an optional 'generation' argument of 0, 1,
or 2 to specify which generation to collect. (Contributed by Barry
Warsaw.)
* The ‘nsmallest()’ and ‘nlargest()’ functions in the *note heapq:
69. module now support a ‘key’ keyword parameter similar to the one
provided by the *note min(): f03./*note max(): f04. functions and
the ‘sort()’ methods. For example:
>>> import heapq
>>> L = ["short", 'medium', 'longest', 'longer still']
>>> heapq.nsmallest(2, L) # Return two lowest elements, lexicographically
['longer still', 'longest']
>>> heapq.nsmallest(2, L, key=len) # Return two shortest elements
['short', 'medium']
(Contributed by Raymond Hettinger.)
* The *note itertools.islice(): b4c. function now accepts ‘None’ for
the start and step arguments. This makes it more compatible with
the attributes of slice objects, so that you can now write the
following:
s = slice(5) # Create slice object
itertools.islice(iterable, s.start, s.stop, s.step)
(Contributed by Raymond Hettinger.)
* The *note format(): 61b. function in the *note locale: 86. module
has been modified and two new functions were added,
‘format_string()’ and ‘currency()’.
The *note format(): 61b. function’s 'val' parameter could
previously be a string as long as no more than one %char specifier
appeared; now the parameter must be exactly one %char specifier
with no surrounding text. An optional 'monetary' parameter was
also added which, if ‘True’, will use the locale’s rules for
formatting currency in placing a separator between groups of three
digits.
To format strings with multiple %char specifiers, use the new
‘format_string()’ function that works like *note format(): 61b. but
also supports mixing %char specifiers with arbitrary text.
A new ‘currency()’ function was also added that formats a number
according to the current locale’s settings.
(Contributed by Georg Brandl.)
* The *note mailbox: 8b. module underwent a massive rewrite to add
the capability to modify mailboxes in addition to reading them. A
new set of classes that include ‘mbox’, ‘MH’, and ‘Maildir’ are
used to read mailboxes, and have an ‘add(message)’ method to add
messages, ‘remove(key)’ to remove messages, and ‘lock()’/‘unlock()’
to lock/unlock the mailbox. The following example converts a
maildir-format mailbox into an mbox-format one:
import mailbox
# 'factory=None' uses email.Message.Message as the class representing
# individual messages.
src = mailbox.Maildir('maildir', factory=None)
dest = mailbox.mbox('/tmp/mbox')
for msg in src:
dest.add(msg)
(Contributed by Gregory K. Johnson. Funding was provided by
Google’s 2005 Summer of Code.)
* New module: the ‘msilib’ module allows creating Microsoft Installer
‘.msi’ files and CAB files. Some support for reading the ‘.msi’
database is also included. (Contributed by Martin von Löwis.)
* The ‘nis’ module now supports accessing domains other than the
system default domain by supplying a 'domain' argument to the
‘nis.match()’ and ‘nis.maps()’ functions. (Contributed by Ben
Bell.)
* The *note operator: 9f. module’s ‘itemgetter()’ and ‘attrgetter()’
functions now support multiple fields. A call such as
‘operator.attrgetter('a', 'b')’ will return a function that
retrieves the ‘a’ and ‘b’ attributes. Combining this new feature
with the ‘sort()’ method’s ‘key’ parameter lets you easily sort
lists using multiple fields. (Contributed by Raymond Hettinger.)
* The *note optparse: a0. module was updated to version 1.5.1 of the
Optik library. The ‘OptionParser’ class gained an ‘epilog’
attribute, a string that will be printed after the help message,
and a ‘destroy()’ method to break reference cycles created by the
object. (Contributed by Greg Ward.)
* The *note os: a1. module underwent several changes. The
‘stat_float_times’ variable now defaults to true, meaning that
*note os.stat(): 49c. will now return time values as floats. (This
doesn’t necessarily mean that *note os.stat(): 49c. will return
times that are precise to fractions of a second; not all systems
support such precision.)
Constants named *note os.SEEK_SET: 1280, *note os.SEEK_CUR: 1281,
and *note os.SEEK_END: 1282. have been added; these are the
parameters to the *note os.lseek(): 110b. function. Two new
constants for locking are *note os.O_SHLOCK: 13fa. and *note
os.O_EXLOCK: 13fb.
Two new functions, ‘wait3()’ and ‘wait4()’, were added. They’re
similar the ‘waitpid()’ function which waits for a child process to
exit and returns a tuple of the process ID and its exit status, but
‘wait3()’ and ‘wait4()’ return additional information. ‘wait3()’
doesn’t take a process ID as input, so it waits for any child
process to exit and returns a 3-tuple of 'process-id',
'exit-status', 'resource-usage' as returned from the *note
resource.getrusage(): 13fc. function. ‘wait4(pid)’ does take a
process ID. (Contributed by Chad J. Schroeder.)
On FreeBSD, the *note os.stat(): 49c. function now returns times
with nanosecond resolution, and the returned object now has
‘st_gen’ and ‘st_birthtime’. The ‘st_flags’ attribute is also
available, if the platform supports it. (Contributed by Antti
Louko and Diego Pettenò.)
* The Python debugger provided by the *note pdb: a5. module can now
store lists of commands to execute when a breakpoint is reached and
execution stops. Once breakpoint #1 has been created, enter
‘commands 1’ and enter a series of commands to be executed,
finishing the list with ‘end’. The command list can include
commands that resume execution, such as ‘continue’ or ‘next’.
(Contributed by Grégoire Dooms.)
* The *note pickle: a6. and ‘cPickle’ modules no longer accept a
return value of ‘None’ from the *note __reduce__(): 9d3. method;
the method must return a tuple of arguments instead. The ability
to return ‘None’ was deprecated in Python 2.4, so this completes
the removal of the feature.
* The *note pkgutil: a9. module, containing various utility functions
for finding packages, was enhanced to support PEP 302(1)’s import
hooks and now also works for packages stored in ZIP-format
archives. (Contributed by Phillip J. Eby.)
* The pybench benchmark suite by Marc-André Lemburg is now included
in the ‘Tools/pybench’ directory. The pybench suite is an
improvement on the commonly used ‘pystone.py’ program because
pybench provides a more detailed measurement of the interpreter’s
speed. It times particular operations such as function calls,
tuple slicing, method lookups, and numeric operations, instead of
performing many different operations and reducing the result to a
single number as ‘pystone.py’ does.
* The ‘pyexpat’ module now uses version 2.0 of the Expat parser.
(Contributed by Trent Mick.)
* The *note Queue: 137a. class provided by the ‘Queue’ module gained
two new methods. ‘join()’ blocks until all items in the queue have
been retrieved and all processing work on the items have been
completed. Worker threads call the other new method,
‘task_done()’, to signal that processing for an item has been
completed. (Contributed by Raymond Hettinger.)
* The old ‘regex’ and ‘regsub’ modules, which have been deprecated
ever since Python 2.0, have finally been deleted. Other deleted
modules: ‘statcache’, ‘tzparse’, ‘whrandom’.
* Also deleted: the ‘lib-old’ directory, which includes ancient
modules such as ‘dircmp’ and ‘ni’, was removed. ‘lib-old’ wasn’t
on the default ‘sys.path’, so unless your programs explicitly added
the directory to ‘sys.path’, this removal shouldn’t affect your
code.
* The *note rlcompleter: bd. module is no longer dependent on
importing the *note readline: ba. module and therefore now works on
non-Unix platforms. (Patch from Robert Kiendl.)
* The *note SimpleXMLRPCServer: 12f. and *note DocXMLRPCServer: 12f.
classes now have a ‘rpc_paths’ attribute that constrains XML-RPC
operations to a limited set of URL paths; the default is to allow
only ‘'/'’ and ‘'/RPC2'’. Setting ‘rpc_paths’ to ‘None’ or an
empty tuple disables this path checking.
* The *note socket: cc. module now supports ‘AF_NETLINK’ sockets on
Linux, thanks to a patch from Philippe Biondi. Netlink sockets are
a Linux-specific mechanism for communications between a user-space
process and kernel code; an introductory article about them is at
‘https://www.linuxjournal.com/article/7356’. In Python code,
netlink addresses are represented as a tuple of 2 integers, ‘(pid,
group_mask)’.
Two new methods on socket objects, ‘recv_into(buffer)’ and
‘recvfrom_into(buffer)’, store the received data in an object that
supports the buffer protocol instead of returning the data as a
string. This means you can put the data directly into an array or
a memory-mapped file.
Socket objects also gained ‘getfamily()’, ‘gettype()’, and
‘getproto()’ accessor methods to retrieve the family, type, and
protocol values for the socket.
* New module: the ‘spwd’ module provides functions for accessing the
shadow password database on systems that support shadow passwords.
* The *note struct: d5. is now faster because it compiles format
strings into ‘Struct’ objects with ‘pack()’ and ‘unpack()’ methods.
This is similar to how the *note re: b9. module lets you create
compiled regular expression objects. You can still use the
module-level ‘pack()’ and ‘unpack()’ functions; they’ll create
‘Struct’ objects and cache them. Or you can use ‘Struct’ instances
directly:
s = struct.Struct('ih3s')
data = s.pack(1972, 187, 'abc')
year, number, name = s.unpack(data)
You can also pack and unpack data to and from buffer objects
directly using the ‘pack_into(buffer, offset, v1, v2, ...)’ and
‘unpack_from(buffer, offset)’ methods. This lets you store data
directly into an array or a memory-mapped file.
(‘Struct’ objects were implemented by Bob Ippolito at the
NeedForSpeed sprint. Support for buffer objects was added by
Martin Blais, also at the NeedForSpeed sprint.)
* The Python developers switched from CVS to Subversion during the
2.5 development process. Information about the exact build version
is available as the ‘sys.subversion’ variable, a 3-tuple of
‘(interpreter-name, branch-name, revision-range)’. For example, at
the time of writing my copy of 2.5 was reporting ‘('CPython',
'trunk', '45313:45315')’.
This information is also available to C extensions via the *note
Py_GetBuildInfo(): 13fd. function that returns a string of build
information like this: ‘"trunk:45355:45356M, Apr 13 2006,
07:42:19"’. (Contributed by Barry Warsaw.)
* Another new function, *note sys._current_frames(): 13fe, returns
the current stack frames for all running threads as a dictionary
mapping thread identifiers to the topmost stack frame currently
active in that thread at the time the function is called.
(Contributed by Tim Peters.)
* The ‘TarFile’ class in the *note tarfile: de. module now has an
‘extractall()’ method that extracts all members from the archive
into the current working directory. It’s also possible to set a
different directory as the extraction target, and to unpack only a
subset of the archive’s members.
The compression used for a tarfile opened in stream mode can now be
autodetected using the mode ‘'r|*'’. (Contributed by Lars
Gustäbel.)
* The *note threading: ed. module now lets you set the stack size
used when new threads are created. The ‘stack_size([*size*])’
function returns the currently configured stack size, and supplying
the optional 'size' parameter sets a new value. Not all platforms
support changing the stack size, but Windows, POSIX threading, and
OS/2 all do. (Contributed by Andrew MacIntyre.)
* The *note unicodedata: 105. module has been updated to use version
4.1.0 of the Unicode character database. Version 3.2.0 is required
by some specifications, so it’s still available as *note
unicodedata.ucd_3_2_0: 13ff.
* New module: the *note uuid: 110. module generates universally
unique identifiers (UUIDs) according to RFC 4122(2). The RFC
defines several different UUID versions that are generated from a
starting string, from system properties, or purely randomly. This
module contains a ‘UUID’ class and functions named ‘uuid1()’,
‘uuid3()’, ‘uuid4()’, and ‘uuid5()’ to generate different versions
of UUID. (Version 2 UUIDs are not specified in RFC 4122(3) and are
not supported by this module.)
>>> import uuid
>>> # make a UUID based on the host ID and current time
>>> uuid.uuid1()
UUID('a8098c1a-f86e-11da-bd1a-00112444be1e')
>>> # make a UUID using an MD5 hash of a namespace UUID and a name
>>> uuid.uuid3(uuid.NAMESPACE_DNS, 'python.org')
UUID('6fa459ea-ee8a-3ca4-894e-db77e160355e')
>>> # make a random UUID
>>> uuid.uuid4()
UUID('16fd2706-8baf-433b-82eb-8c7fada847da')
>>> # make a UUID using a SHA-1 hash of a namespace UUID and a name
>>> uuid.uuid5(uuid.NAMESPACE_DNS, 'python.org')
UUID('886313e1-3b8a-5372-9b90-0c9aee199e5d')
(Contributed by Ka-Ping Yee.)
* The *note weakref: 114. module’s ‘WeakKeyDictionary’ and
‘WeakValueDictionary’ types gained new methods for iterating over
the weak references contained in the dictionary. ‘iterkeyrefs()’
and ‘keyrefs()’ methods were added to ‘WeakKeyDictionary’, and
‘itervaluerefs()’ and ‘valuerefs()’ were added to
‘WeakValueDictionary’. (Contributed by Fred L. Drake, Jr.)
* The *note webbrowser: 115. module received a number of
enhancements. It’s now usable as a script with ‘python -m
webbrowser’, taking a URL as the argument; there are a number of
switches to control the behaviour (‘-n’ for a new browser window,
‘-t’ for a new tab). New module-level functions, ‘open_new()’ and
‘open_new_tab()’, were added to support this. The module’s *note
open(): 517. function supports an additional feature, an
'autoraise' parameter that signals whether to raise the open window
when possible. A number of additional browsers were added to the
supported list such as Firefox, Opera, Konqueror, and elinks.
(Contributed by Oleg Broytmann and Georg Brandl.)
* The *note xmlrpclib: 12e. module now supports returning *note
datetime: 1cc. objects for the XML-RPC date type. Supply
‘use_datetime=True’ to the *note loads(): 1400. function or the
‘Unmarshaller’ class to enable this feature. (Contributed by Skip
Montanaro.)
* The *note zipfile: 131. module now supports the ZIP64 version of
the format, meaning that a .zip archive can now be larger than 4
GiB and can contain individual files larger than 4 GiB.
(Contributed by Ronald Oussoren.)
* The *note zlib: 133. module’s ‘Compress’ and ‘Decompress’ objects
now support a *note copy(): 25. method that makes a copy of the
object’s internal state and returns a new ‘Compress’ or
‘Decompress’ object. (Contributed by Chris AtLee.)
* Menu:
* The ctypes package::
* The ElementTree package::
* The hashlib package::
* The sqlite3 package::
* The wsgiref package::
---------- Footnotes ----------
(1) https://peps.python.org/pep-0302/
(2) https://datatracker.ietf.org/doc/html/rfc4122.html
(3) https://datatracker.ietf.org/doc/html/rfc4122.html
File: python.info, Node: The ctypes package, Next: The ElementTree package, Up: New Improved and Removed Modules
1.17.13.1 The ctypes package
............................
The *note ctypes: 2a. package, written by Thomas Heller, has been added
to the standard library. *note ctypes: 2a. lets you call arbitrary
functions in shared libraries or DLLs. Long-time users may remember the
‘dl’ module, which provides functions for loading shared libraries and
calling functions in them. The *note ctypes: 2a. package is much
fancier.
To load a shared library or DLL, you must create an instance of the
‘CDLL’ class and provide the name or path of the shared library or DLL.
Once that’s done, you can call arbitrary functions by accessing them as
attributes of the ‘CDLL’ object.
import ctypes
libc = ctypes.CDLL('libc.so.6')
result = libc.printf("Line of output\n")
Type constructors for the various C types are provided: ‘c_int()’,
‘c_float()’, ‘c_double()’, ‘c_char_p()’ (equivalent to char*), and so
forth. Unlike Python’s types, the C versions are all mutable; you can
assign to their ‘value’ attribute to change the wrapped value. Python
integers and strings will be automatically converted to the
corresponding C types, but for other types you must call the correct
type constructor. (And I mean 'must'; getting it wrong will often
result in the interpreter crashing with a segmentation fault.)
You shouldn’t use ‘c_char_p()’ with a Python string when the C function
will be modifying the memory area, because Python strings are supposed
to be immutable; breaking this rule will cause puzzling bugs. When you
need a modifiable memory area, use ‘create_string_buffer()’:
s = "this is a string"
buf = ctypes.create_string_buffer(s)
libc.strfry(buf)
C functions are assumed to return integers, but you can set the
‘restype’ attribute of the function object to change this:
>>> libc.atof('2.71828')
-1783957616
>>> libc.atof.restype = ctypes.c_double
>>> libc.atof('2.71828')
2.71828
*note ctypes: 2a. also provides a wrapper for Python’s C API as the
‘ctypes.pythonapi’ object. This object does 'not' release the global
interpreter lock before calling a function, because the lock must be
held when calling into the interpreter’s code. There’s a *note
py_object: 1403. type constructor that will create a *note PyObject:
334.* pointer. A simple usage:
import ctypes
d = {}
ctypes.pythonapi.PyObject_SetItem(ctypes.py_object(d),
ctypes.py_object("abc"), ctypes.py_object(1))
# d is now {'abc', 1}.
Don’t forget to use *note py_object(): 1403.; if it’s omitted you end up
with a segmentation fault.
*note ctypes: 2a. has been around for a while, but people still write
and distribution hand-coded extension modules because you can’t rely on
*note ctypes: 2a. being present. Perhaps developers will begin to write
Python wrappers atop a library accessed through *note ctypes: 2a.
instead of extension modules, now that *note ctypes: 2a. is included
with core Python.
See also
........
‘https://web.archive.org/web/20180410025338/http://starship.python.net/crew/theller/ctypes/’
The pre-stdlib ctypes web page, with a tutorial, reference, and
FAQ.
The documentation for the *note ctypes: 2a. module.
File: python.info, Node: The ElementTree package, Next: The hashlib package, Prev: The ctypes package, Up: New Improved and Removed Modules
1.17.13.2 The ElementTree package
.................................
A subset of Fredrik Lundh’s ElementTree library for processing XML has
been added to the standard library as ‘xml.etree’. The available
modules are ‘ElementTree’, ‘ElementPath’, and ‘ElementInclude’ from
ElementTree 1.2.6. The ‘cElementTree’ accelerator module is also
included.
The rest of this section will provide a brief overview of using
ElementTree. Full documentation for ElementTree is available at
‘https://web.archive.org/web/20201124024954/http://effbot.org/zone/element-index.htm’.
ElementTree represents an XML document as a tree of element nodes. The
text content of the document is stored as the ‘text’ and ‘tail’
attributes of (This is one of the major differences between ElementTree
and the Document Object Model; in the DOM there are many different types
of node, including ‘TextNode’.)
The most commonly used parsing function is ‘parse()’, that takes either
a string (assumed to contain a filename) or a file-like object and
returns an ‘ElementTree’ instance:
from xml.etree import ElementTree as ET
tree = ET.parse('ex-1.xml')
feed = urllib.urlopen(
'http://planet.python.org/rss10.xml')
tree = ET.parse(feed)
Once you have an ‘ElementTree’ instance, you can call its ‘getroot()’
method to get the root ‘Element’ node.
There’s also an ‘XML()’ function that takes a string literal and returns
an ‘Element’ node (not an ‘ElementTree’). This function provides a tidy
way to incorporate XML fragments, approaching the convenience of an XML
literal:
svg = ET.XML("""
""")
svg.set('height', '320px')
svg.append(elem1)
Each XML element supports some dictionary-like and some list-like access
methods. Dictionary-like operations are used to access attribute
values, and list-like operations are used to access child nodes.
Operation Result
-------------------------------------------------------------------------------------
‘elem[n]’ Returns n’th child element.
‘elem[m:n]’ Returns list of m’th through n’th child
elements.
‘len(elem)’ Returns number of child elements.
‘list(elem)’ Returns list of child elements.
‘elem.append(elem2)’ Adds 'elem2' as a child.
‘elem.insert(index, elem2)’ Inserts 'elem2' at the specified location.
‘del elem[n]’ Deletes n’th child element.
‘elem.keys()’ Returns list of attribute names.
‘elem.get(name)’ Returns value of attribute 'name'.
‘elem.set(name, value)’ Sets new value for attribute 'name'.
‘elem.attrib’ Retrieves the dictionary containing
attributes.
‘del elem.attrib[name]’ Deletes attribute 'name'.
Comments and processing instructions are also represented as ‘Element’
nodes. To check if a node is a comment or processing instructions:
if elem.tag is ET.Comment:
...
elif elem.tag is ET.ProcessingInstruction:
...
To generate XML output, you should call the ‘ElementTree.write()’
method. Like ‘parse()’, it can take either a string or a file-like
object:
# Encoding is US-ASCII
tree.write('output.xml')
# Encoding is UTF-8
f = open('output.xml', 'w')
tree.write(f, encoding='utf-8')
(Caution: the default encoding used for output is ASCII. For general XML
work, where an element’s name may contain arbitrary Unicode characters,
ASCII isn’t a very useful encoding because it will raise an exception if
an element’s name contains any characters with values greater than 127.
Therefore, it’s best to specify a different encoding such as UTF-8 that
can handle any Unicode character.)
This section is only a partial description of the ElementTree
interfaces. Please read the package’s official documentation for more
details.
See also
........
‘https://web.archive.org/web/20201124024954/http://effbot.org/zone/element-index.htm’
Official documentation for ElementTree.
File: python.info, Node: The hashlib package, Next: The sqlite3 package, Prev: The ElementTree package, Up: New Improved and Removed Modules
1.17.13.3 The hashlib package
.............................
A new *note hashlib: 68. module, written by Gregory P. Smith, has been
added to replace the ‘md5’ and ‘sha’ modules. *note hashlib: 68. adds
support for additional secure hashes (SHA-224, SHA-256, SHA-384, and
SHA-512). When available, the module uses OpenSSL for fast platform
optimized implementations of algorithms.
The old ‘md5’ and ‘sha’ modules still exist as wrappers around hashlib
to preserve backwards compatibility. The new module’s interface is very
close to that of the old modules, but not identical. The most
significant difference is that the constructor functions for creating
new hashing objects are named differently.
# Old versions
h = md5.md5()
h = md5.new()
# New version
h = hashlib.md5()
# Old versions
h = sha.sha()
h = sha.new()
# New version
h = hashlib.sha1()
# Hash that weren't previously available
h = hashlib.sha224()
h = hashlib.sha256()
h = hashlib.sha384()
h = hashlib.sha512()
# Alternative form
h = hashlib.new('md5') # Provide algorithm as a string
Once a hash object has been created, its methods are the same as before:
‘update(string)’ hashes the specified string into the current digest
state, ‘digest()’ and ‘hexdigest()’ return the digest value as a binary
string or a string of hex digits, and *note copy(): 25. returns a new
hashing object with the same digest state.
See also
........
The documentation for the *note hashlib: 68. module.
File: python.info, Node: The sqlite3 package, Next: The wsgiref package, Prev: The hashlib package, Up: New Improved and Removed Modules
1.17.13.4 The sqlite3 package
.............................
The pysqlite module (‘https://www.pysqlite.org’), a wrapper for the
SQLite embedded database, has been added to the standard library under
the package name *note sqlite3: cf.
SQLite is a C library that provides a lightweight disk-based database
that doesn’t require a separate server process and allows accessing the
database using a nonstandard variant of the SQL query language. Some
applications can use SQLite for internal data storage. It’s also
possible to prototype an application using SQLite and then port the code
to a larger database such as PostgreSQL or Oracle.
pysqlite was written by Gerhard Häring and provides a SQL interface
compliant with the DB-API 2.0 specification described by PEP 249(1).
If you’re compiling the Python source yourself, note that the source
tree doesn’t include the SQLite code, only the wrapper module. You’ll
need to have the SQLite libraries and headers installed before compiling
Python, and the build process will compile the module when the necessary
headers are available.
To use the module, you must first create a ‘Connection’ object that
represents the database. Here the data will be stored in the
‘/tmp/example’ file:
conn = sqlite3.connect('/tmp/example')
You can also supply the special name ‘:memory:’ to create a database in
RAM.
Once you have a ‘Connection’, you can create a ‘Cursor’ object and call
its ‘execute()’ method to perform SQL commands:
c = conn.cursor()
# Create table
c.execute('''create table stocks
(date text, trans text, symbol text,
qty real, price real)''')
# Insert a row of data
c.execute("""insert into stocks
values ('2006-01-05','BUY','RHAT',100,35.14)""")
Usually your SQL operations will need to use values from Python
variables. You shouldn’t assemble your query using Python’s string
operations because doing so is insecure; it makes your program
vulnerable to an SQL injection attack.
Instead, use the DB-API’s parameter substitution. Put ‘?’ as a
placeholder wherever you want to use a value, and then provide a tuple
of values as the second argument to the cursor’s ‘execute()’ method.
(Other database modules may use a different placeholder, such as ‘%s’ or
‘:1’.) For example:
# Never do this -- insecure!
symbol = 'IBM'
c.execute("... where symbol = '%s'" % symbol)
# Do this instead
t = (symbol,)
c.execute('select * from stocks where symbol=?', t)
# Larger example
for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
):
c.execute('insert into stocks values (?,?,?,?,?)', t)
To retrieve data after executing a SELECT statement, you can either
treat the cursor as an iterator, call the cursor’s ‘fetchone()’ method
to retrieve a single matching row, or call ‘fetchall()’ to get a list of
the matching rows.
This example uses the iterator form:
>>> c = conn.cursor()
>>> c.execute('select * from stocks order by price')
>>> for row in c:
... print row
...
(u'2006-01-05', u'BUY', u'RHAT', 100, 35.140000000000001)
(u'2006-03-28', u'BUY', u'IBM', 1000, 45.0)
(u'2006-04-06', u'SELL', u'IBM', 500, 53.0)
(u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0)
>>>
For more information about the SQL dialect supported by SQLite, see
‘https://www.sqlite.org’.
See also
........
‘https://www.pysqlite.org’
The pysqlite web page.
‘https://www.sqlite.org’
The SQLite web page; the documentation describes the syntax and the
available data types for the supported SQL dialect.
The documentation for the *note sqlite3: cf. module.
PEP 249(2) - Database API Specification 2.0
PEP written by Marc-André Lemburg.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0249/
(2) https://peps.python.org/pep-0249/
File: python.info, Node: The wsgiref package, Prev: The sqlite3 package, Up: New Improved and Removed Modules
1.17.13.5 The wsgiref package
.............................
The Web Server Gateway Interface (WSGI) v1.0 defines a standard
interface between web servers and Python web applications and is
described in PEP 333(1). The *note wsgiref: 118. package is a reference
implementation of the WSGI specification.
The package includes a basic HTTP server that will run a WSGI
application; this server is useful for debugging but isn’t intended for
production use. Setting up a server takes only a few lines of code:
from wsgiref import simple_server
wsgi_app = ...
host = ''
port = 8000
httpd = simple_server.make_server(host, port, wsgi_app)
httpd.serve_forever()
See also
........
‘https://web.archive.org/web/20160331090247/http://wsgi.readthedocs.org/en/latest/’
A central web site for WSGI-related resources.
PEP 333(2) - Python Web Server Gateway Interface v1.0
PEP written by Phillip J. Eby.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0333/
(2) https://peps.python.org/pep-0333/
File: python.info, Node: Build and C API Changes<10>, Next: Porting to Python 2 5, Prev: New Improved and Removed Modules, Up: What’s New in Python 2 5
1.17.14 Build and C API Changes
-------------------------------
Changes to Python’s build process and to the C API include:
* The Python source tree was converted from CVS to Subversion, in a
complex migration procedure that was supervised and flawlessly
carried out by Martin von Löwis. The procedure was developed as
PEP 347(1).
* Coverity, a company that markets a source code analysis tool called
Prevent, provided the results of their examination of the Python
source code. The analysis found about 60 bugs that were quickly
fixed. Many of the bugs were refcounting problems, often occurring
in error-handling code. See ‘https://scan.coverity.com’ for the
statistics.
* The largest change to the C API came from PEP 353(2), which
modifies the interpreter to use a *note Py_ssize_t: a5f. type
definition instead of int. See the earlier section *note PEP 353;
Using ssize_t as the index type: 13e3. for a discussion of this
change.
* The design of the bytecode compiler has changed a great deal, no
longer generating bytecode by traversing the parse tree. Instead
the parse tree is converted to an abstract syntax tree (or AST),
and it is the abstract syntax tree that’s traversed to produce the
bytecode.
It’s possible for Python code to obtain AST objects by using the
*note compile(): 192. built-in and specifying ‘_ast.PyCF_ONLY_AST’
as the value of the 'flags' parameter:
from _ast import PyCF_ONLY_AST
ast = compile("""a=0
for i in range(10):
a += i
""", "", 'exec', PyCF_ONLY_AST)
assignment = ast.body[0]
for_loop = ast.body[1]
No official documentation has been written for the AST code yet,
but PEP 339(3) discusses the design. To start learning about the
code, read the definition of the various AST nodes in
‘Parser/Python.asdl’. A Python script reads this file and
generates a set of C structure definitions in
‘Include/Python-ast.h’. The ‘PyParser_ASTFromString()’ and
‘PyParser_ASTFromFile()’, defined in ‘Include/pythonrun.h’, take
Python source as input and return the root of an AST representing
the contents. This AST can then be turned into a code object by
‘PyAST_Compile()’. For more information, read the source code, and
then ask questions on python-dev.
The AST code was developed under Jeremy Hylton’s management, and
implemented by (in alphabetical order) Brett Cannon, Nick Coghlan,
Grant Edwards, John Ehresman, Kurt Kaiser, Neal Norwitz, Tim
Peters, Armin Rigo, and Neil Schemenauer, plus the participants in
a number of AST sprints at conferences such as PyCon.
* Evan Jones’s patch to obmalloc, first described in a talk at PyCon
DC 2005, was applied. Python 2.4 allocated small objects in
256K-sized arenas, but never freed arenas. With this patch, Python
will free arenas when they’re empty. The net effect is that on
some platforms, when you allocate many objects, Python’s memory
usage may actually drop when you delete them and the memory may be
returned to the operating system. (Implemented by Evan Jones, and
reworked by Tim Peters.)
Note that this change means extension modules must be more careful
when allocating memory. Python’s API has many different functions
for allocating memory that are grouped into families. For example,
*note PyMem_Malloc(): c66, *note PyMem_Realloc(): 102b, and *note
PyMem_Free(): 140e. are one family that allocates raw memory, while
*note PyObject_Malloc(): c68, *note PyObject_Realloc(): 140f, and
*note PyObject_Free(): c65. are another family that’s supposed to
be used for creating Python objects.
Previously these different families all reduced to the platform’s
‘malloc()’ and ‘free()’ functions. This meant it didn’t matter if
you got things wrong and allocated memory with the ‘PyMem’ function
but freed it with the ‘PyObject’ function. With 2.5’s changes to
obmalloc, these families now do different things and mismatches
will probably result in a segfault. You should carefully test your
C extension modules with Python 2.5.
* The built-in set types now have an official C API. Call *note
PySet_New(): 1410. and *note PyFrozenSet_New(): 1411. to create a
new set, *note PySet_Add(): 1412. and *note PySet_Discard(): 1413.
to add and remove elements, and *note PySet_Contains(): 1414. and
*note PySet_Size(): 1415. to examine the set’s state. (Contributed
by Raymond Hettinger.)
* C code can now obtain information about the exact revision of the
Python interpreter by calling the *note Py_GetBuildInfo(): 13fd.
function that returns a string of build information like this:
‘"trunk:45355:45356M, Apr 13 2006, 07:42:19"’. (Contributed by
Barry Warsaw.)
* Two new macros can be used to indicate C functions that are local
to the current file so that a faster calling convention can be
used. ‘Py_LOCAL(type)’ declares the function as returning a value
of the specified 'type' and uses a fast-calling qualifier.
‘Py_LOCAL_INLINE(type)’ does the same thing and also requests the
function be inlined. If macro ‘PY_LOCAL_AGGRESSIVE’ is defined
before ‘python.h’ is included, a set of more aggressive
optimizations are enabled for the module; you should benchmark the
results to find out if these optimizations actually make the code
faster. (Contributed by Fredrik Lundh at the NeedForSpeed sprint.)
* ‘PyErr_NewException(name, base, dict)’ can now accept a tuple of
base classes as its 'base' argument. (Contributed by Georg
Brandl.)
* The ‘PyErr_Warn()’ function for issuing warnings is now deprecated
in favour of ‘PyErr_WarnEx(category, message, stacklevel)’ which
lets you specify the number of stack frames separating this
function and the caller. A 'stacklevel' of 1 is the function
calling *note PyErr_WarnEx(): 1416, 2 is the function above that,
and so forth. (Added by Neal Norwitz.)
* The CPython interpreter is still written in C, but the code can now
be compiled with a C++ compiler without errors. (Implemented by
Anthony Baxter, Martin von Löwis, Skip Montanaro.)
* The ‘PyRange_New()’ function was removed. It was never documented,
never used in the core code, and had dangerously lax error
checking. In the unlikely case that your extensions were using it,
you can replace it by something like the following:
range = PyObject_CallFunction((PyObject*) &PyRange_Type, "lll",
start, stop, step);
* Menu:
* Port-Specific Changes::
---------- Footnotes ----------
(1) https://peps.python.org/pep-0347/
(2) https://peps.python.org/pep-0353/
(3) https://peps.python.org/pep-0339/
File: python.info, Node: Port-Specific Changes, Up: Build and C API Changes<10>
1.17.14.1 Port-Specific Changes
...............................
* MacOS X (10.3 and higher): dynamic loading of modules now uses the
‘dlopen()’ function instead of MacOS-specific functions.
* MacOS X: an ‘--enable-universalsdk’ switch was added to the
‘configure’ script that compiles the interpreter as a universal
binary able to run on both PowerPC and Intel processors.
(Contributed by Ronald Oussoren; bpo-2573(1).)
* Windows: ‘.dll’ is no longer supported as a filename extension for
extension modules. ‘.pyd’ is now the only filename extension that
will be searched for.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=2573
File: python.info, Node: Porting to Python 2 5, Next: Acknowledgements<3>, Prev: Build and C API Changes<10>, Up: What’s New in Python 2 5
1.17.15 Porting to Python 2.5
-----------------------------
This section lists previously described changes that may require changes
to your code:
* ASCII is now the default encoding for modules. It’s now a syntax
error if a module contains string literals with 8-bit characters
but doesn’t have an encoding declaration. In Python 2.4 this
triggered a warning, not a syntax error.
* Previously, the ‘gi_frame’ attribute of a generator was always a
frame object. Because of the PEP 342(1) changes described in
section *note PEP 342; New Generator Features: 13cf, it’s now
possible for ‘gi_frame’ to be ‘None’.
* A new warning, *note UnicodeWarning: 13ef, is triggered when you
attempt to compare a Unicode string and an 8-bit string that can’t
be converted to Unicode using the default ASCII encoding.
Previously such comparisons would raise a *note UnicodeDecodeError:
a12. exception.
* Library: the *note csv: 29. module is now stricter about multi-line
quoted fields. If your files contain newlines embedded within
fields, the input should be split into lines in a manner which
preserves the newline characters.
* Library: the *note locale: 86. module’s *note format(): 61b.
function’s would previously accept any string as long as no more
than one %char specifier appeared. In Python 2.5, the argument
must be exactly one %char specifier with no surrounding text.
* Library: The *note pickle: a6. and ‘cPickle’ modules no longer
accept a return value of ‘None’ from the *note __reduce__(): 9d3.
method; the method must return a tuple of arguments instead. The
modules also no longer accept the deprecated 'bin' keyword
parameter.
* Library: The *note SimpleXMLRPCServer: 12f. and *note
DocXMLRPCServer: 12f. classes now have a ‘rpc_paths’ attribute that
constrains XML-RPC operations to a limited set of URL paths; the
default is to allow only ‘'/'’ and ‘'/RPC2'’. Setting ‘rpc_paths’
to ‘None’ or an empty tuple disables this path checking.
* C API: Many functions now use *note Py_ssize_t: a5f. instead of int
to allow processing more data on 64-bit machines. Extension code
may need to make the same change to avoid warnings and to support
64-bit machines. See the earlier section *note PEP 353; Using
ssize_t as the index type: 13e3. for a discussion of this change.
* C API: The obmalloc changes mean that you must be careful to not
mix usage of the ‘PyMem_*’ and ‘PyObject_*’ families of functions.
Memory allocated with one family’s ‘*_Malloc’ must be freed with
the corresponding family’s ‘*_Free’ function.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0342/
File: python.info, Node: Acknowledgements<3>, Prev: Porting to Python 2 5, Up: What’s New in Python 2 5
1.17.16 Acknowledgements
------------------------
The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Georg Brandl, Nick Coghlan, Phillip J. Eby, Lars Gustäbel,
Raymond Hettinger, Ralf W. Grosse-Kunstleve, Kent Johnson, Iain Lowe,
Martin von Löwis, Fredrik Lundh, Andrew McNamara, Skip Montanaro,
Gustavo Niemeyer, Paul Prescod, James Pryor, Mike Rovner, Scott Weikart,
Barry Warsaw, Thomas Wouters.
File: python.info, Node: What’s New in Python 2 4, Next: What’s New in Python 2 3, Prev: What’s New in Python 2 5, Up: What’s New in Python
1.18 What’s New in Python 2.4
=============================
Author: A.M. Kuchling
This article explains the new features in Python 2.4.1, released on
March 30, 2005.
Python 2.4 is a medium-sized release. It doesn’t introduce as many
changes as the radical Python 2.2, but introduces more features than the
conservative 2.3 release. The most significant new language features
are function decorators and generator expressions; most other changes
are to the standard library.
According to the CVS change logs, there were 481 patches applied and 502
bugs fixed between Python 2.3 and 2.4. Both figures are likely to be
underestimates.
This article doesn’t attempt to provide a complete specification of
every single new feature, but instead provides a brief introduction to
each feature. For full details, you should refer to the documentation
for Python 2.4, such as the Python Library Reference and the Python
Reference Manual. Often you will be referred to the PEP for a
particular new feature for explanations of the implementation and design
rationale.
* Menu:
* PEP 218; Built-In Set Objects: PEP 218 Built-In Set Objects.
* PEP 237; Unifying Long Integers and Integers: PEP 237 Unifying Long Integers and Integers.
* PEP 289; Generator Expressions: PEP 289 Generator Expressions.
* PEP 292; Simpler String Substitutions: PEP 292 Simpler String Substitutions.
* PEP 318; Decorators for Functions and Methods: PEP 318 Decorators for Functions and Methods.
* PEP 322; Reverse Iteration: PEP 322 Reverse Iteration.
* PEP 324; New subprocess Module: PEP 324 New subprocess Module.
* PEP 327; Decimal Data Type: PEP 327 Decimal Data Type.
* PEP 328; Multi-line Imports: PEP 328 Multi-line Imports.
* PEP 331; Locale-Independent Float/String Conversions: PEP 331 Locale-Independent Float/String Conversions.
* Other Language Changes: Other Language Changes<17>.
* New, Improved, and Deprecated Modules: New Improved and Deprecated Modules<3>.
* Build and C API Changes: Build and C API Changes<11>.
* Porting to Python 2.4: Porting to Python 2 4.
* Acknowledgements: Acknowledgements<4>.
File: python.info, Node: PEP 218 Built-In Set Objects, Next: PEP 237 Unifying Long Integers and Integers, Up: What’s New in Python 2 4
1.18.1 PEP 218: Built-In Set Objects
------------------------------------
Python 2.3 introduced the ‘sets’ module. C implementations of set data
types have now been added to the Python core as two new built-in types,
‘set(iterable)’ and ‘frozenset(iterable)’. They provide high speed
operations for membership testing, for eliminating duplicates from
sequences, and for mathematical operations like unions, intersections,
differences, and symmetric differences.
>>> a = set('abracadabra') # form a set from a string
>>> 'z' in a # fast membership testing
False
>>> a # unique letters in a
set(['a', 'r', 'b', 'c', 'd'])
>>> ''.join(a) # convert back into a string
'arbcd'
>>> b = set('alacazam') # form a second set
>>> a - b # letters in a but not in b
set(['r', 'd', 'b'])
>>> a | b # letters in either a or b
set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
>>> a & b # letters in both a and b
set(['a', 'c'])
>>> a ^ b # letters in a or b but not both
set(['r', 'd', 'b', 'm', 'z', 'l'])
>>> a.add('z') # add a new element
>>> a.update('wxy') # add multiple new elements
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])
>>> a.remove('x') # take one element out
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])
The *note frozenset(): 5d6. type is an immutable version of *note set():
5d5. Since it is immutable and hashable, it may be used as a dictionary
key or as a member of another set.
The ‘sets’ module remains in the standard library, and may be useful if
you wish to subclass the ‘Set’ or ‘ImmutableSet’ classes. There are
currently no plans to deprecate the module.
See also
........
PEP 218(1) - Adding a Built-In Set Object Type
Originally proposed by Greg Wilson and ultimately implemented by
Raymond Hettinger.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0218/
File: python.info, Node: PEP 237 Unifying Long Integers and Integers, Next: PEP 289 Generator Expressions, Prev: PEP 218 Built-In Set Objects, Up: What’s New in Python 2 4
1.18.2 PEP 237: Unifying Long Integers and Integers
---------------------------------------------------
The lengthy transition process for this PEP, begun in Python 2.2, takes
another step forward in Python 2.4. In 2.3, certain integer operations
that would behave differently after int/long unification triggered *note
FutureWarning: 411. warnings and returned values limited to 32 or 64
bits (depending on your platform). In 2.4, these expressions no longer
produce a warning and instead produce a different result that’s usually
a long integer.
The problematic expressions are primarily left shifts and lengthy
hexadecimal and octal constants. For example, ‘2 << 32’ results in a
warning in 2.3, evaluating to 0 on 32-bit platforms. In Python 2.4,
this expression now returns the correct answer, 8589934592.
See also
........
PEP 237(1) - Unifying Long Integers and Integers
Original PEP written by Moshe Zadka and GvR. The changes for 2.4
were implemented by Kalle Svensson.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0237/
File: python.info, Node: PEP 289 Generator Expressions, Next: PEP 292 Simpler String Substitutions, Prev: PEP 237 Unifying Long Integers and Integers, Up: What’s New in Python 2 4
1.18.3 PEP 289: Generator Expressions
-------------------------------------
The iterator feature introduced in Python 2.2 and the *note itertools:
81. module make it easier to write programs that loop through large data
sets without having the entire data set in memory at one time. List
comprehensions don’t fit into this picture very well because they
produce a Python list object containing all of the items. This
unavoidably pulls all of the objects into memory, which can be a problem
if your data set is very large. When trying to write a functionally
styled program, it would be natural to write something like:
links = [link for link in get_all_links() if not link.followed]
for link in links:
...
instead of
for link in get_all_links():
if link.followed:
continue
...
The first form is more concise and perhaps more readable, but if you’re
dealing with a large number of link objects you’d have to write the
second form to avoid having all link objects in memory at the same time.
Generator expressions work similarly to list comprehensions but don’t
materialize the entire list; instead they create a generator that will
return elements one by one. The above example could be written as:
links = (link for link in get_all_links() if not link.followed)
for link in links:
...
Generator expressions always have to be written inside parentheses, as
in the above example. The parentheses signalling a function call also
count, so if you want to create an iterator that will be immediately
passed to a function you could write:
print sum(obj.count for obj in list_all_objects())
Generator expressions differ from list comprehensions in various small
ways. Most notably, the loop variable ('obj' in the above example) is
not accessible outside of the generator expression. List comprehensions
leave the variable assigned to its last value; future versions of Python
will change this, making list comprehensions match generator expressions
in this respect.
See also
........
PEP 289(1) - Generator Expressions
Proposed by Raymond Hettinger and implemented by Jiwon Seo with
early efforts steered by Hye-Shik Chang.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0289/
File: python.info, Node: PEP 292 Simpler String Substitutions, Next: PEP 318 Decorators for Functions and Methods, Prev: PEP 289 Generator Expressions, Up: What’s New in Python 2 4
1.18.4 PEP 292: Simpler String Substitutions
--------------------------------------------
Some new classes in the standard library provide an alternative
mechanism for substituting variables into strings; this style of
substitution may be better for applications where untrained users need
to edit templates.
The usual way of substituting variables by name is the ‘%’ operator:
>>> '%(page)i: %(title)s' % {'page':2, 'title': 'The Best of Times'}
'2: The Best of Times'
When writing the template string, it can be easy to forget the ‘i’ or
‘s’ after the closing parenthesis. This isn’t a big problem if the
template is in a Python module, because you run the code, get an
“Unsupported format character” *note ValueError: 204, and fix the
problem. However, consider an application such as Mailman where
template strings or translations are being edited by users who aren’t
aware of the Python language. The format string’s syntax is complicated
to explain to such users, and if they make a mistake, it’s difficult to
provide helpful feedback to them.
PEP 292 adds a ‘Template’ class to the *note string: d3. module that
uses ‘$’ to indicate a substitution:
>>> import string
>>> t = string.Template('$page: $title')
>>> t.substitute({'page':2, 'title': 'The Best of Times'})
'2: The Best of Times'
If a key is missing from the dictionary, the ‘substitute()’ method will
raise a *note KeyError: 33f. There’s also a ‘safe_substitute()’ method
that ignores missing keys:
>>> t = string.Template('$page: $title')
>>> t.safe_substitute({'page':3})
'3: $title'
See also
........
PEP 292(1) - Simpler String Substitutions
Written and implemented by Barry Warsaw.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0292/
File: python.info, Node: PEP 318 Decorators for Functions and Methods, Next: PEP 322 Reverse Iteration, Prev: PEP 292 Simpler String Substitutions, Up: What’s New in Python 2 4
1.18.5 PEP 318: Decorators for Functions and Methods
----------------------------------------------------
Python 2.2 extended Python’s object model by adding static methods and
class methods, but it didn’t extend Python’s syntax to provide any new
way of defining static or class methods. Instead, you had to write a
*note def: 1423. statement in the usual way, and pass the resulting
method to a *note staticmethod(): 412. or *note classmethod(): 166.
function that would wrap up the function as a method of the new type.
Your code would look like this:
class C:
def meth (cls):
...
meth = classmethod(meth) # Rebind name to wrapped-up class method
If the method was very long, it would be easy to miss or forget the
*note classmethod(): 166. invocation after the function body.
The intention was always to add some syntax to make such definitions
more readable, but at the time of 2.2’s release a good syntax was not
obvious. Today a good syntax 'still' isn’t obvious but users are asking
for easier access to the feature; a new syntactic feature has been added
to meet this need.
The new feature is called “function decorators”. The name comes from
the idea that *note classmethod(): 166, *note staticmethod(): 412, and
friends are storing additional information on a function object; they’re
'decorating' functions with more details.
The notation borrows from Java and uses the ‘'@'’ character as an
indicator. Using the new syntax, the example above would be written:
class C:
@classmethod
def meth (cls):
...
The ‘@classmethod’ is shorthand for the ‘meth=classmethod(meth)’
assignment. More generally, if you have the following:
@A
@B
@C
def f ():
...
It’s equivalent to the following pre-decorator code:
def f(): ...
f = A(B(C(f)))
Decorators must come on the line before a function definition, one
decorator per line, and can’t be on the same line as the def statement,
meaning that ‘@A def f(): ...’ is illegal. You can only decorate
function definitions, either at the module level or inside a class; you
can’t decorate class definitions.
A decorator is just a function that takes the function to be decorated
as an argument and returns either the same function or some new object.
The return value of the decorator need not be callable (though it
typically is), unless further decorators will be applied to the result.
It’s easy to write your own decorators. The following simple example
just sets an attribute on the function object:
>>> def deco(func):
... func.attr = 'decorated'
... return func
...
>>> @deco
... def f(): pass
...
>>> f
>>> f.attr
'decorated'
>>>
As a slightly more realistic example, the following decorator checks
that the supplied argument is an integer:
def require_int (func):
def wrapper (arg):
assert isinstance(arg, int)
return func(arg)
return wrapper
@require_int
def p1 (arg):
print arg
@require_int
def p2(arg):
print arg*2
An example in PEP 318(1) contains a fancier version of this idea that
lets you both specify the required type and check the returned type.
Decorator functions can take arguments. If arguments are supplied, your
decorator function is called with only those arguments and must return a
new decorator function; this function must take a single function and
return a function, as previously described. In other words, ‘@A @B
@C(args)’ becomes:
def f(): ...
_deco = C(args)
f = A(B(_deco(f)))
Getting this right can be slightly brain-bending, but it’s not too
difficult.
A small related change makes the *note func_name: 12c7. attribute of
functions writable. This attribute is used to display function names in
tracebacks, so decorators should change the name of any new function
that’s constructed and returned.
See also
........
PEP 318(2) - Decorators for Functions, Methods and Classes
Written by Kevin D. Smith, Jim Jewett, and Skip Montanaro. Several
people wrote patches implementing function decorators, but the one
that was actually checked in was patch #979728, written by Mark
Russell.
‘https://wiki.python.org/moin/PythonDecoratorLibrary’
This Wiki page contains several examples of decorators.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0318/
(2) https://peps.python.org/pep-0318/
File: python.info, Node: PEP 322 Reverse Iteration, Next: PEP 324 New subprocess Module, Prev: PEP 318 Decorators for Functions and Methods, Up: What’s New in Python 2 4
1.18.6 PEP 322: Reverse Iteration
---------------------------------
A new built-in function, ‘reversed(seq)’, takes a sequence and returns
an iterator that loops over the elements of the sequence in reverse
order.
>>> for i in reversed(xrange(1,4)):
... print i
...
3
2
1
Compared to extended slicing, such as ‘range(1,4)[::-1]’, *note
reversed(): 862. is easier to read, runs faster, and uses substantially
less memory.
Note that *note reversed(): 862. only accepts sequences, not arbitrary
iterators. If you want to reverse an iterator, first convert it to a
list with *note list(): 60d.
>>> input = open('/etc/passwd', 'r')
>>> for line in reversed(list(input)):
... print line
...
root:*:0:0:System Administrator:/var/root:/bin/tcsh
...
See also
........
PEP 322(1) - Reverse Iteration
Written and implemented by Raymond Hettinger.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0322/
File: python.info, Node: PEP 324 New subprocess Module, Next: PEP 327 Decimal Data Type, Prev: PEP 322 Reverse Iteration, Up: What’s New in Python 2 4
1.18.7 PEP 324: New subprocess Module
-------------------------------------
The standard library provides a number of ways to execute a subprocess,
offering different features and different levels of complexity.
‘os.system(command)’ is easy to use, but slow (it runs a shell process
which executes the command) and dangerous (you have to be careful about
escaping the shell’s metacharacters). The ‘popen2’ module offers
classes that can capture standard output and standard error from the
subprocess, but the naming is confusing. The *note subprocess: d6.
module cleans this up, providing a unified interface that offers all the
features you might need.
Instead of ‘popen2’’s collection of classes, *note subprocess: d6.
contains a single class called *note subprocess.Popen: 199. whose
constructor supports a number of different keyword arguments.
class Popen(args, bufsize=0, executable=None,
stdin=None, stdout=None, stderr=None,
preexec_fn=None, close_fds=False, shell=False,
cwd=None, env=None, universal_newlines=False,
startupinfo=None, creationflags=0):
'args' is commonly a sequence of strings that will be the arguments to
the program executed as the subprocess. (If the 'shell' argument is
true, 'args' can be a string which will then be passed on to the shell
for interpretation, just as *note os.system(): 1426. does.)
'stdin', 'stdout', and 'stderr' specify what the subprocess’s input,
output, and error streams will be. You can provide a file object or a
file descriptor, or you can use the constant ‘subprocess.PIPE’ to create
a pipe between the subprocess and the parent.
The constructor has a number of handy options:
* 'close_fds' requests that all file descriptors be closed before
running the subprocess.
* 'cwd' specifies the working directory in which the subprocess will
be executed (defaulting to whatever the parent’s working directory
is).
* 'env' is a dictionary specifying environment variables.
* 'preexec_fn' is a function that gets called before the child is
started.
* 'universal_newlines' opens the child’s input and output using
Python’s *note universal newlines: d3a. feature.
Once you’ve created the ‘Popen’ instance, you can call its ‘wait()’
method to pause until the subprocess has exited, ‘poll()’ to check if
it’s exited without pausing, or ‘communicate(data)’ to send the string
'data' to the subprocess’s standard input. ‘communicate(data)’ then
reads any data that the subprocess has sent to its standard output or
standard error, returning a tuple ‘(stdout_data, stderr_data)’.
‘call()’ is a shortcut that passes its arguments along to the ‘Popen’
constructor, waits for the command to complete, and returns the status
code of the subprocess. It can serve as a safer analog to *note
os.system(): 1426.:
sts = subprocess.call(['dpkg', '-i', '/tmp/new-package.deb'])
if sts == 0:
# Success
...
else:
# dpkg returned an error
...
The command is invoked without use of the shell. If you really do want
to use the shell, you can add ‘shell=True’ as a keyword argument and
provide a string instead of a sequence:
sts = subprocess.call('dpkg -i /tmp/new-package.deb', shell=True)
The PEP takes various examples of shell and Python code and shows how
they’d be translated into Python code that uses *note subprocess: d6.
Reading this section of the PEP is highly recommended.
See also
........
PEP 324(1) - subprocess - New process module
Written and implemented by Peter Åstrand, with assistance from
Fredrik Lundh and others.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0324/
File: python.info, Node: PEP 327 Decimal Data Type, Next: PEP 328 Multi-line Imports, Prev: PEP 324 New subprocess Module, Up: What’s New in Python 2 4
1.18.8 PEP 327: Decimal Data Type
---------------------------------
Python has always supported floating-point (FP) numbers, based on the
underlying C double type, as a data type. However, while most
programming languages provide a floating-point type, many people (even
programmers) are unaware that floating-point numbers don’t represent
certain decimal fractions accurately. The new ‘Decimal’ type can
represent these fractions accurately, up to a user-specified precision
limit.
* Menu:
* Why is Decimal needed?::
* The Decimal type::
* The Context type::
File: python.info, Node: Why is Decimal needed?, Next: The Decimal type, Up: PEP 327 Decimal Data Type
1.18.8.1 Why is Decimal needed?
...............................
The limitations arise from the representation used for floating-point
numbers. FP numbers are made up of three components:
* The sign, which is positive or negative.
* The mantissa, which is a single-digit binary number followed by a
fractional part. For example, ‘1.01’ in base-2 notation is ‘1 +
0/2 + 1/4’, or 1.25 in decimal notation.
* The exponent, which tells where the decimal point is located in the
number represented.
For example, the number 1.25 has positive sign, a mantissa value of 1.01
(in binary), and an exponent of 0 (the decimal point doesn’t need to be
shifted). The number 5 has the same sign and mantissa, but the exponent
is 2 because the mantissa is multiplied by 4 (2 to the power of the
exponent 2); 1.25 * 4 equals 5.
Modern systems usually provide floating-point support that conforms to a
standard called IEEE 754. C’s double type is usually implemented as a
64-bit IEEE 754 number, which uses 52 bits of space for the mantissa.
This means that numbers can only be specified to 52 bits of precision.
If you’re trying to represent numbers whose expansion repeats endlessly,
the expansion is cut off after 52 bits. Unfortunately, most software
needs to produce output in base 10, and common fractions in base 10 are
often repeating decimals in binary. For example, 1.1 decimal is binary
‘1.0001100110011 ...’; .1 = 1/16 + 1/32 + 1/256 plus an infinite number
of additional terms. IEEE 754 has to chop off that infinitely repeated
decimal after 52 digits, so the representation is slightly inaccurate.
Sometimes you can see this inaccuracy when the number is printed:
>>> 1.1
1.1000000000000001
The inaccuracy isn’t always visible when you print the number because
the FP-to-decimal-string conversion is provided by the C library, and
most C libraries try to produce sensible output. Even if it’s not
displayed, however, the inaccuracy is still there and subsequent
operations can magnify the error.
For many applications this doesn’t matter. If I’m plotting points and
displaying them on my monitor, the difference between 1.1 and
1.1000000000000001 is too small to be visible. Reports often limit
output to a certain number of decimal places, and if you round the
number to two or three or even eight decimal places, the error is never
apparent. However, for applications where it does matter, it’s a lot of
work to implement your own custom arithmetic routines.
Hence, the ‘Decimal’ type was created.
File: python.info, Node: The Decimal type, Next: The Context type, Prev: Why is Decimal needed?, Up: PEP 327 Decimal Data Type
1.18.8.2 The ‘Decimal’ type
...........................
A new module, *note decimal: 36, was added to Python’s standard library.
It contains two classes, ‘Decimal’ and ‘Context’. ‘Decimal’ instances
represent numbers, and ‘Context’ instances are used to wrap up various
settings such as the precision and default rounding mode.
‘Decimal’ instances are immutable, like regular Python integers and FP
numbers; once it’s been created, you can’t change the value an instance
represents. ‘Decimal’ instances can be created from integers or
strings:
>>> import decimal
>>> decimal.Decimal(1972)
Decimal("1972")
>>> decimal.Decimal("1.1")
Decimal("1.1")
You can also provide tuples containing the sign, the mantissa
represented as a tuple of decimal digits, and the exponent:
>>> decimal.Decimal((1, (1, 4, 7, 5), -2))
Decimal("-14.75")
Cautionary note: the sign bit is a Boolean value, so 0 is positive and 1
is negative.
Converting from floating-point numbers poses a bit of a problem: should
the FP number representing 1.1 turn into the decimal number for exactly
1.1, or for 1.1 plus whatever inaccuracies are introduced? The decision
was to dodge the issue and leave such a conversion out of the API.
Instead, you should convert the floating-point number into a string
using the desired precision and pass the string to the ‘Decimal’
constructor:
>>> f = 1.1
>>> decimal.Decimal(str(f))
Decimal("1.1")
>>> decimal.Decimal('%.12f' % f)
Decimal("1.100000000000")
Once you have ‘Decimal’ instances, you can perform the usual
mathematical operations on them. One limitation: exponentiation
requires an integer exponent:
>>> a = decimal.Decimal('35.72')
>>> b = decimal.Decimal('1.73')
>>> a+b
Decimal("37.45")
>>> a-b
Decimal("33.99")
>>> a*b
Decimal("61.7956")
>>> a/b
Decimal("20.64739884393063583815028902")
>>> a ** 2
Decimal("1275.9184")
>>> a**b
Traceback (most recent call last):
...
decimal.InvalidOperation: x ** (non-integer)
You can combine ‘Decimal’ instances with integers, but not with
floating-point numbers:
>>> a + 4
Decimal("39.72")
>>> a + 4.5
Traceback (most recent call last):
...
TypeError: You can interact Decimal only with int, long or Decimal data types.
>>>
‘Decimal’ numbers can be used with the *note math: 8e. and *note cmath:
18. modules, but note that they’ll be immediately converted to
floating-point numbers before the operation is performed, resulting in a
possible loss of precision and accuracy. You’ll also get back a regular
floating-point number and not a ‘Decimal’.
>>> import math, cmath
>>> d = decimal.Decimal('123456789012.345')
>>> math.sqrt(d)
351364.18288201344
>>> cmath.sqrt(-d)
351364.18288201344j
‘Decimal’ instances have a ‘sqrt()’ method that returns a ‘Decimal’, but
if you need other things such as trigonometric functions you’ll have to
implement them.
>>> d.sqrt()
Decimal("351364.1828820134592177245001")
File: python.info, Node: The Context type, Prev: The Decimal type, Up: PEP 327 Decimal Data Type
1.18.8.3 The ‘Context’ type
...........................
Instances of the ‘Context’ class encapsulate several settings for
decimal operations:
* ‘prec’ is the precision, the number of decimal places.
* ‘rounding’ specifies the rounding mode. The *note decimal: 36.
module has constants for the various possibilities: ‘ROUND_DOWN’,
‘ROUND_CEILING’, ‘ROUND_HALF_EVEN’, and various others.
* ‘traps’ is a dictionary specifying what happens on encountering
certain error conditions: either an exception is raised or a value
is returned. Some examples of error conditions are division by
zero, loss of precision, and overflow.
There’s a thread-local default context available by calling
‘getcontext()’; you can change the properties of this context to alter
the default precision, rounding, or trap handling. The following
example shows the effect of changing the precision of the default
context:
>>> decimal.getcontext().prec
28
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal("0.1428571428571428571428571429")
>>> decimal.getcontext().prec = 9
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal("0.142857143")
The default action for error conditions is selectable; the module can
either return a special value such as infinity or not-a-number, or
exceptions can be raised:
>>> decimal.Decimal(1) / decimal.Decimal(0)
Traceback (most recent call last):
...
decimal.DivisionByZero: x / 0
>>> decimal.getcontext().traps[decimal.DivisionByZero] = False
>>> decimal.Decimal(1) / decimal.Decimal(0)
Decimal("Infinity")
>>>
The ‘Context’ instance also has various methods for formatting numbers
such as ‘to_eng_string()’ and ‘to_sci_string()’.
For more information, see the documentation for the *note decimal: 36.
module, which includes a quick-start tutorial and a reference.
See also
........
PEP 327(1) - Decimal Data Type
Written by Facundo Batista and implemented by Facundo Batista, Eric
Price, Raymond Hettinger, Aahz, and Tim Peters.
http://www.lahey.com/float.htm(2)
The article uses Fortran code to illustrate many of the problems
that floating-point inaccuracy can cause.
‘https://speleotrove.com/decimal/’
A description of a decimal-based representation. This
representation is being proposed as a standard, and underlies the
new Python decimal type. Much of this material was written by Mike
Cowlishaw, designer of the Rexx language.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0327/
(2)
https://web.archive.org/web/20230604072523/http://www.lahey.com/float.htm
File: python.info, Node: PEP 328 Multi-line Imports, Next: PEP 331 Locale-Independent Float/String Conversions, Prev: PEP 327 Decimal Data Type, Up: What’s New in Python 2 4
1.18.9 PEP 328: Multi-line Imports
----------------------------------
One language change is a small syntactic tweak aimed at making it easier
to import many names from a module. In a ‘from module import names’
statement, 'names' is a sequence of names separated by commas. If the
sequence is very long, you can either write multiple imports from the
same module, or you can use backslashes to escape the line endings like
this:
from SimpleXMLRPCServer import SimpleXMLRPCServer,\
SimpleXMLRPCRequestHandler,\
CGIXMLRPCRequestHandler,\
resolve_dotted_attribute
The syntactic change in Python 2.4 simply allows putting the names
within parentheses. Python ignores newlines within a parenthesized
expression, so the backslashes are no longer needed:
from SimpleXMLRPCServer import (SimpleXMLRPCServer,
SimpleXMLRPCRequestHandler,
CGIXMLRPCRequestHandler,
resolve_dotted_attribute)
The PEP also proposes that all *note import: 5de. statements be absolute
imports, with a leading ‘.’ character to indicate a relative import.
This part of the PEP was not implemented for Python 2.4, but was
completed for Python 2.5.
See also
........
PEP 328(1) - Imports: Multi-Line and Absolute/Relative
Written by Aahz. Multi-line imports were implemented by Dima
Dorfman.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0328/
File: python.info, Node: PEP 331 Locale-Independent Float/String Conversions, Next: Other Language Changes<17>, Prev: PEP 328 Multi-line Imports, Up: What’s New in Python 2 4
1.18.10 PEP 331: Locale-Independent Float/String Conversions
------------------------------------------------------------
The *note locale: 86. modules lets Python software select various
conversions and display conventions that are localized to a particular
country or language. However, the module was careful to not change the
numeric locale because various functions in Python’s implementation
required that the numeric locale remain set to the ‘'C'’ locale. Often
this was because the code was using the C library’s ‘atof()’ function.
Not setting the numeric locale caused trouble for extensions that used
third-party C libraries, however, because they wouldn’t have the correct
locale set. The motivating example was GTK+, whose user interface
widgets weren’t displaying numbers in the current locale.
The solution described in the PEP is to add three new functions to the
Python API that perform ASCII-only conversions, ignoring the locale
setting:
* ‘PyOS_ascii_strtod(str, ptr)’ and ‘PyOS_ascii_atof(str, ptr)’ both
convert a string to a C double.
* ‘PyOS_ascii_formatd(buffer, buf_len, format, d)’ converts a double
to an ASCII string.
The code for these functions came from the GLib library
(https://developer-old.gnome.org/glib/2.26/(1)), whose developers kindly
relicensed the relevant functions and donated them to the Python
Software Foundation. The *note locale: 86. module can now change the
numeric locale, letting extensions such as GTK+ produce the correct
results.
See also
........
PEP 331(2) - Locale-Independent Float/String Conversions
Written by Christian R. Reis, and implemented by Gustavo Carneiro.
---------- Footnotes ----------
(1)
http://web.archive.org/web/20210306104320/https://developer.gnome.org/glib/2.26/
(2) https://peps.python.org/pep-0331/
File: python.info, Node: Other Language Changes<17>, Next: New Improved and Deprecated Modules<3>, Prev: PEP 331 Locale-Independent Float/String Conversions, Up: What’s New in Python 2 4
1.18.11 Other Language Changes
------------------------------
Here are all of the changes that Python 2.4 makes to the core Python
language.
* Decorators for functions and methods were added ( PEP 318(1)).
* Built-in *note set(): 5d5. and *note frozenset(): 5d6. types were
added ( PEP 218(2)). Other new built-ins include the
‘reversed(seq)’ function ( PEP 322(3)).
* Generator expressions were added ( PEP 289(4)).
* Certain numeric expressions no longer return values restricted to
32 or 64 bits ( PEP 237(5)).
* You can now put parentheses around the list of names in a ‘from
module import names’ statement ( PEP 328(6)).
* The *note dict.update(): 142e. method now accepts the same argument
forms as the *note dict: 258. constructor. This includes any
mapping, any iterable of key/value pairs, and keyword arguments.
(Contributed by Raymond Hettinger.)
* The string methods ‘ljust()’, ‘rjust()’, and ‘center()’ now take an
optional argument for specifying a fill character other than a
space. (Contributed by Raymond Hettinger.)
* Strings also gained an ‘rsplit()’ method that works like the
‘split()’ method but splits from the end of the string.
(Contributed by Sean Reifschneider.)
>>> 'www.python.org'.split('.', 1)
['www', 'python.org']
'www.python.org'.rsplit('.', 1)
['www.python', 'org']
* Three keyword parameters, 'cmp', 'key', and 'reverse', were added
to the ‘sort()’ method of lists. These parameters make some common
usages of ‘sort()’ simpler. All of these parameters are optional.
For the 'cmp' parameter, the value should be a comparison function
that takes two parameters and returns -1, 0, or +1 depending on how
the parameters compare. This function will then be used to sort
the list. Previously this was the only parameter that could be
provided to ‘sort()’.
'key' should be a single-parameter function that takes a list
element and returns a comparison key for the element. The list is
then sorted using the comparison keys. The following example sorts
a list case-insensitively:
>>> L = ['A', 'b', 'c', 'D']
>>> L.sort() # Case-sensitive sort
>>> L
['A', 'D', 'b', 'c']
>>> # Using 'key' parameter to sort list
>>> L.sort(key=lambda x: x.lower())
>>> L
['A', 'b', 'c', 'D']
>>> # Old-fashioned way
>>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
>>> L
['A', 'b', 'c', 'D']
The last example, which uses the 'cmp' parameter, is the old way to
perform a case-insensitive sort. It works but is slower than using
a 'key' parameter. Using 'key' calls ‘lower()’ method once for
each element in the list while using 'cmp' will call it twice for
each comparison, so using 'key' saves on invocations of the
‘lower()’ method.
For simple key functions and comparison functions, it is often
possible to avoid a *note lambda: 128f. expression by using an
unbound method instead. For example, the above case-insensitive
sort is best written as:
>>> L.sort(key=str.lower)
>>> L
['A', 'b', 'c', 'D']
Finally, the 'reverse' parameter takes a Boolean value. If the
value is true, the list will be sorted into reverse order. Instead
of ‘L.sort(); L.reverse()’, you can now write
‘L.sort(reverse=True)’.
The results of sorting are now guaranteed to be stable. This means
that two entries with equal keys will be returned in the same order
as they were input. For example, you can sort a list of people by
name, and then sort the list by age, resulting in a list sorted by
age where people with the same age are in name-sorted order.
(All changes to ‘sort()’ contributed by Raymond Hettinger.)
* There is a new built-in function ‘sorted(iterable)’ that works like
the in-place *note list.sort(): bcf. method but can be used in
expressions. The differences are:
* the input may be any iterable;
* a newly formed copy is sorted, leaving the original intact; and
* the expression returns the new sorted copy
>>> L = [9,7,8,3,2,4,1,6,5]
>>> [10+i for i in sorted(L)] # usable in a list comprehension
[11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> L # original is left unchanged
[9,7,8,3,2,4,1,6,5]
>>> sorted('Monty Python') # any iterable may be an input
[' ', 'M', 'P', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y', 'y']
>>> # List the contents of a dict sorted by key values
>>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)
>>> for k, v in sorted(colormap.iteritems()):
... print k, v
...
black 4
blue 2
green 3
red 1
yellow 5
(Contributed by Raymond Hettinger.)
* Integer operations will no longer trigger an ‘OverflowWarning’.
The ‘OverflowWarning’ warning will disappear in Python 2.5.
* The interpreter gained a new switch, *note -m: 5dd, that takes a
name, searches for the corresponding module on ‘sys.path’, and runs
the module as a script. For example, you can now run the Python
profiler with ‘python -m profile’. (Contributed by Nick Coghlan.)
* The ‘eval(expr, globals, locals)’ and ‘execfile(filename, globals,
locals)’ functions and the ‘exec’ statement now accept any mapping
type for the 'locals' parameter. Previously this had to be a
regular Python dictionary. (Contributed by Raymond Hettinger.)
* The *note zip(): 7cb. built-in function and ‘itertools.izip()’ now
return an empty list if called with no arguments. Previously they
raised a *note TypeError: 534. exception. This makes them more
suitable for use with variable length argument lists:
>>> def transpose(array):
... return zip(*array)
...
>>> transpose([(1,2,3), (4,5,6)])
[(1, 4), (2, 5), (3, 6)]
>>> transpose([])
[]
(Contributed by Raymond Hettinger.)
* Encountering a failure while importing a module no longer leaves a
partially initialized module object in ‘sys.modules’. The
incomplete module object left behind would fool further imports of
the same module into succeeding, leading to confusing errors.
(Fixed by Tim Peters.)
* *note None: 671. is now a constant; code that binds a new value to
the name ‘None’ is now a syntax error. (Contributed by Raymond
Hettinger.)
* Menu:
* Optimizations: Optimizations<16>.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0318/
(2) https://peps.python.org/pep-0218/
(3) https://peps.python.org/pep-0322/
(4) https://peps.python.org/pep-0289/
(5) https://peps.python.org/pep-0237/
(6) https://peps.python.org/pep-0328/
File: python.info, Node: Optimizations<16>, Up: Other Language Changes<17>
1.18.11.1 Optimizations
.......................
* The inner loops for list and tuple slicing were optimized and now
run about one-third faster. The inner loops for dictionaries were
also optimized, resulting in performance boosts for ‘keys()’,
‘values()’, ‘items()’, ‘iterkeys()’, ‘itervalues()’, and
‘iteritems()’. (Contributed by Raymond Hettinger.)
* The machinery for growing and shrinking lists was optimized for
speed and for space efficiency. Appending and popping from lists
now runs faster due to more efficient code paths and less frequent
use of the underlying system ‘realloc()’. List comprehensions also
benefit. ‘list.extend()’ was also optimized and no longer converts
its argument into a temporary list before extending the base list.
(Contributed by Raymond Hettinger.)
* *note list(): 60d, *note tuple(): 36b, *note map(): 860, *note
filter(): 861, and *note zip(): 7cb. now run several times faster
with non-sequence arguments that supply a ‘__len__()’ method.
(Contributed by Raymond Hettinger.)
* The methods ‘list.__getitem__()’, ‘dict.__getitem__()’, and
‘dict.__contains__()’ are now implemented as ‘method_descriptor’
objects rather than ‘wrapper_descriptor’ objects. This form of
access doubles their performance and makes them more suitable for
use as arguments to functionals: ‘map(mydict.__getitem__,
keylist)’. (Contributed by Raymond Hettinger.)
* Added a new opcode, ‘LIST_APPEND’, that simplifies the generated
bytecode for list comprehensions and speeds them up by about a
third. (Contributed by Raymond Hettinger.)
* The peephole bytecode optimizer has been improved to produce
shorter, faster bytecode; remarkably, the resulting bytecode is
more readable. (Enhanced by Raymond Hettinger.)
* String concatenations in statements of the form ‘s = s + "abc"’ and
‘s += "abc"’ are now performed more efficiently in certain
circumstances. This optimization won’t be present in other Python
implementations such as Jython, so you shouldn’t rely on it; using
the ‘join()’ method of strings is still recommended when you want
to efficiently glue a large number of strings together.
(Contributed by Armin Rigo.)
The net result of the 2.4 optimizations is that Python 2.4 runs the
pystone benchmark around 5% faster than Python 2.3 and 35% faster than
Python 2.2. (pystone is not a particularly good benchmark, but it’s the
most commonly used measurement of Python’s performance. Your own
applications may show greater or smaller benefits from Python 2.4.)
File: python.info, Node: New Improved and Deprecated Modules<3>, Next: Build and C API Changes<11>, Prev: Other Language Changes<17>, Up: What’s New in Python 2 4
1.18.12 New, Improved, and Deprecated Modules
---------------------------------------------
As usual, Python’s standard library received a number of enhancements
and bug fixes. Here’s a partial list of the most notable changes,
sorted alphabetically by module name. Consult the ‘Misc/NEWS’ file in
the source tree for a more complete list of changes, or look through the
CVS logs for all the details.
* The ‘asyncore’ module’s ‘loop()’ function now has a 'count'
parameter that lets you perform a limited number of passes through
the polling loop. The default is still to loop forever.
* The *note base64: e. module now has more complete RFC 3548(1)
support for Base64, Base32, and Base16 encoding and decoding,
including optional case folding and optional alternative alphabets.
(Contributed by Barry Warsaw.)
* The *note bisect: 11. module now has an underlying C implementation
for improved performance. (Contributed by Dmitry Vasiliev.)
* The CJKCodecs collections of East Asian codecs, maintained by
Hye-Shik Chang, was integrated into 2.4. The new encodings are:
* Chinese (PRC): gb2312, gbk, gb18030, big5hkscs, hz
* Chinese (ROC): big5, cp950
*
Japanese: cp932, euc-jis-2004, euc-jp, euc-jisx0213, iso-2022-jp,
iso-2022-jp-1, iso-2022-jp-2, iso-2022-jp-3, iso-2022-jp-ext,
iso-2022-jp-2004, shift-jis, shift-jisx0213, shift-jis-2004
* Korean: cp949, euc-kr, johab, iso-2022-kr
* Some other new encodings were added: HP Roman8, ISO_8859-11,
ISO_8859-16, PCTP-154, and TIS-620.
* The UTF-8 and UTF-16 codecs now cope better with receiving partial
input. Previously the ‘StreamReader’ class would try to read more
data, making it impossible to resume decoding from the stream. The
‘read()’ method will now return as much data as it can and future
calls will resume decoding where previous ones left off.
(Implemented by Walter Dörwald.)
* There is a new *note collections: 1d. module for various
specialized collection datatypes. Currently it contains just one
type, ‘deque’, a double-ended queue that supports efficiently
adding and removing elements from either end:
>>> from collections import deque
>>> d = deque('ghi') # make a new deque with three items
>>> d.append('j') # add a new entry to the right side
>>> d.appendleft('f') # add a new entry to the left side
>>> d # show the representation of the deque
deque(['f', 'g', 'h', 'i', 'j'])
>>> d.pop() # return and remove the rightmost item
'j'
>>> d.popleft() # return and remove the leftmost item
'f'
>>> list(d) # list the contents of the deque
['g', 'h', 'i']
>>> 'h' in d # search the deque
True
Several modules, such as the ‘Queue’ and *note threading: ed.
modules, now take advantage of *note collections.deque: 5d8. for
improved performance. (Contributed by Raymond Hettinger.)
* The *note ConfigParser: 22. classes have been enhanced slightly.
The *note read(): 1431. method now returns a list of the files that
were successfully parsed, and the *note set(): 1260. method raises
*note TypeError: 534. if passed a 'value' argument that isn’t a
string. (Contributed by John Belmonte and David Goodger.)
* The *note curses: 2b. module now supports the ncurses extension
‘use_default_colors()’. On platforms where the terminal supports
transparency, this makes it possible to use a transparent
background. (Contributed by Jörg Lehmann.)
* The *note difflib: 37. module now includes an ‘HtmlDiff’ class that
creates an HTML table showing a side by side comparison of two
versions of a text. (Contributed by Dan Gass.)
* The *note email: 3b. package was updated to version 3.0, which
dropped various deprecated APIs and removes support for Python
versions earlier than 2.3. The 3.0 version of the package uses a
new incremental parser for MIME messages, available in the
‘email.FeedParser’ module. The new parser doesn’t require reading
the entire message into memory, and doesn’t raise exceptions if a
message is malformed; instead it records any problems in the
‘defect’ attribute of the message. (Developed by Anthony Baxter,
Barry Warsaw, Thomas Wouters, and others.)
* The *note heapq: 69. module has been converted to C. The resulting
tenfold improvement in speed makes the module suitable for handling
high volumes of data. In addition, the module has two new
functions ‘nlargest()’ and ‘nsmallest()’ that use heaps to find the
N largest or smallest values in a dataset without the expense of a
full sort. (Contributed by Raymond Hettinger.)
* The *note httplib: 6e. module now contains constants for HTTP
status codes defined in various HTTP-related RFC documents.
Constants have names such as ‘OK’, ‘CREATED’, ‘CONTINUE’, and
‘MOVED_PERMANENTLY’; use pydoc to get a full list. (Contributed by
Andrew Eland.)
* The *note imaplib: 74. module now supports IMAP’s THREAD command
(contributed by Yves Dionne) and new ‘deleteacl()’ and ‘myrights()’
methods (contributed by Arnaud Mazin).
* The *note itertools: 81. module gained a ‘groupby(iterable[,
*func*])’ function. 'iterable' is something that can be iterated
over to return a stream of elements, and the optional 'func'
parameter is a function that takes an element and returns a key
value; if omitted, the key is simply the element itself.
‘groupby()’ then groups the elements into subsequences which have
matching values of the key, and returns a series of 2-tuples
containing the key value and an iterator over the subsequence.
Here’s an example to make this clearer. The 'key' function simply
returns whether a number is even or odd, so the result of
‘groupby()’ is to return consecutive runs of odd or even numbers.
>>> import itertools
>>> L = [2, 4, 6, 7, 8, 9, 11, 12, 14]
>>> for key_val, it in itertools.groupby(L, lambda x: x % 2):
... print key_val, list(it)
...
0 [2, 4, 6]
1 [7]
0 [8]
1 [9, 11]
0 [12, 14]
>>>
‘groupby()’ is typically used with sorted input. The logic for
‘groupby()’ is similar to the Unix ‘uniq’ filter which makes it
handy for eliminating, counting, or identifying duplicate elements:
>>> word = 'abracadabra'
>>> letters = sorted(word) # Turn string into a sorted list of letters
>>> letters
['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']
>>> for k, g in itertools.groupby(letters):
... print k, list(g)
...
a ['a', 'a', 'a', 'a', 'a']
b ['b', 'b']
c ['c']
d ['d']
r ['r', 'r']
>>> # List unique letters
>>> [k for k, g in groupby(letters)]
['a', 'b', 'c', 'd', 'r']
>>> # Count letter occurrences
>>> [(k, len(list(g))) for k, g in groupby(letters)]
[('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]
(Contributed by Hye-Shik Chang.)
* *note itertools: 81. also gained a function named ‘tee(iterator,
N)’ that returns 'N' independent iterators that replicate
'iterator'. If 'N' is omitted, the default is 2.
>>> L = [1,2,3]
>>> i1, i2 = itertools.tee(L)
>>> i1,i2
(, )
>>> list(i1) # Run the first iterator to exhaustion
[1, 2, 3]
>>> list(i2) # Run the second iterator to exhaustion
[1, 2, 3]
Note that ‘tee()’ has to keep copies of the values returned by the
iterator; in the worst case, it may need to keep all of them. This
should therefore be used carefully if the leading iterator can run
far ahead of the trailing iterator in a long stream of inputs. If
the separation is large, then you might as well use *note list():
60d. instead. When the iterators track closely with one another,
‘tee()’ is ideal. Possible applications include bookmarking,
windowing, or lookahead iterators. (Contributed by Raymond
Hettinger.)
* A number of functions were added to the *note locale: 86. module,
such as ‘bind_textdomain_codeset()’ to specify a particular
encoding and a family of ‘l*gettext()’ functions that return
messages in the chosen encoding. (Contributed by Gustavo
Niemeyer.)
* Some keyword arguments were added to the *note logging: 87.
package’s ‘basicConfig()’ function to simplify log configuration.
The default behavior is to log messages to standard error, but
various keyword arguments can be specified to log to a particular
file, change the logging format, or set the logging level. For
example:
import logging
logging.basicConfig(filename='/var/log/application.log',
level=0, # Log all messages
format='%(levelname):%(process):%(thread):%(message)')
Other additions to the *note logging: 87. package include a
‘log(level, msg)’ convenience method, as well as a
‘TimedRotatingFileHandler’ class that rotates its log files at a
timed interval. The module already had ‘RotatingFileHandler’,
which rotated logs once the file exceeded a certain size. Both
classes derive from a new ‘BaseRotatingHandler’ class that can be
used to implement other rotating handlers.
(Changes implemented by Vinay Sajip.)
* The *note marshal: 8d. module now shares interned strings on
unpacking a data structure. This may shrink the size of certain
pickle strings, but the primary effect is to make ‘.pyc’ files
significantly smaller. (Contributed by Martin von Löwis.)
* The ‘nntplib’ module’s ‘NNTP’ class gained ‘description()’ and
‘descriptions()’ methods to retrieve newsgroup descriptions for a
single group or for a range of groups. (Contributed by Jürgen A.
Erhard.)
* Two new functions were added to the *note operator: 9f. module,
‘attrgetter(attr)’ and ‘itemgetter(index)’. Both functions return
callables that take a single argument and return the corresponding
attribute or item; these callables make excellent data extractors
when used with *note map(): 860. or *note sorted(): bce. For
example:
>>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)]
>>> map(operator.itemgetter(0), L)
['c', 'd', 'a', 'b']
>>> map(operator.itemgetter(1), L)
[2, 1, 4, 3]
>>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item
[('d', 1), ('c', 2), ('b', 3), ('a', 4)]
(Contributed by Raymond Hettinger.)
* The *note optparse: a0. module was updated in various ways. The
module now passes its messages through *note gettext.gettext():
1432, making it possible to internationalize Optik’s help and error
messages. Help messages for options can now include the string
‘'%default'’, which will be replaced by the option’s default value.
(Contributed by Greg Ward.)
* The long-term plan is to deprecate the ‘rfc822’ module in some
future Python release in favor of the *note email: 3b. package. To
this end, the *note email.Utils.formatdate: 1433. function has been
changed to make it usable as a replacement for
‘rfc822.formatdate()’. You may want to write new e-mail processing
code with this in mind. (Change implemented by Anthony Baxter.)
* A new ‘urandom(n)’ function was added to the *note os: a1. module,
returning a string containing 'n' bytes of random data. This
function provides access to platform-specific sources of randomness
such as ‘/dev/urandom’ on Linux or the Windows CryptoAPI.
(Contributed by Trevor Perrin.)
* Another new function: ‘os.path.lexists(path)’ returns true if the
file specified by 'path' exists, whether or not it’s a symbolic
link. This differs from the existing ‘os.path.exists(path)’
function, which returns false if 'path' is a symlink that points to
a destination that doesn’t exist. (Contributed by Beni
Cherniavsky.)
* A new ‘getsid()’ function was added to the *note posix: ad. module
that underlies the *note os: a1. module. (Contributed by J.
Raynor.)
* The *note poplib: ac. module now supports POP over SSL.
(Contributed by Hector Urtubia.)
* The *note profile: af. module can now profile C extension
functions. (Contributed by Nick Bastin.)
* The *note random: b8. module has a new method called
‘getrandbits(N)’ that returns a long integer 'N' bits in length.
The existing ‘randrange()’ method now uses ‘getrandbits()’ where
appropriate, making generation of arbitrarily large random numbers
more efficient. (Contributed by Raymond Hettinger.)
* The regular expression language accepted by the *note re: b9.
module was extended with simple conditional expressions, written as
‘(?(group)A|B)’. 'group' is either a numeric group ID or a group
name defined with ‘(?P...)’ earlier in the expression. If
the specified group matched, the regular expression pattern 'A'
will be tested against the string; if the group didn’t match, the
pattern 'B' will be used instead. (Contributed by Gustavo
Niemeyer.)
* The *note re: b9. module is also no longer recursive, thanks to a
massive amount of work by Gustavo Niemeyer. In a recursive regular
expression engine, certain patterns result in a large amount of C
stack space being consumed, and it was possible to overflow the
stack. For example, if you matched a 30000-byte string of ‘a’
characters against the expression ‘(a|b)+’, one stack frame was
consumed per character. Python 2.3 tried to check for stack
overflow and raise a *note RuntimeError: 195. exception, but
certain patterns could sidestep the checking and if you were
unlucky Python could segfault. Python 2.4’s regular expression
engine can match this pattern without problems.
* The *note signal: c6. module now performs tighter error-checking on
the parameters to the *note signal.signal(): 1128. function. For
example, you can’t set a handler on the ‘SIGKILL’ signal; previous
versions of Python would quietly accept this, but 2.4 will raise a
*note RuntimeError: 195. exception.
* Two new functions were added to the *note socket: cc. module.
‘socketpair()’ returns a pair of connected sockets and
‘getservbyport(port)’ looks up the service name for a given port
number. (Contributed by Dave Cole and Barry Warsaw.)
* The ‘sys.exitfunc()’ function has been deprecated. Code should be
using the existing *note atexit: c. module, which correctly handles
calling multiple exit functions. Eventually ‘sys.exitfunc()’ will
become a purely internal interface, accessed only by *note atexit:
c.
* The *note tarfile: de. module now generates GNU-format tar files by
default. (Contributed by Lars Gustäbel.)
* The *note threading: ed. module now has an elegantly simple way to
support thread-local data. The module contains a ‘local’ class
whose attribute values are local to different threads.
import threading
data = threading.local()
data.number = 42
data.url = ('www.python.org', 80)
Other threads can assign and retrieve their own values for the
‘number’ and ‘url’ attributes. You can subclass ‘local’ to
initialize attributes or to add methods. (Contributed by Jim
Fulton.)
* The *note timeit: ef. module now automatically disables periodic
garbage collection during the timing loop. This change makes
consecutive timings more comparable. (Contributed by Raymond
Hettinger.)
* The *note weakref: 114. module now supports a wider variety of
objects including Python functions, class instances, sets,
frozensets, deques, arrays, files, sockets, and regular expression
pattern objects. (Contributed by Raymond Hettinger.)
* The *note xmlrpclib: 12e. module now supports a multi-call
extension for transmitting multiple XML-RPC calls in a single HTTP
operation. (Contributed by Brian Quinlan.)
* The ‘mpz’, ‘rotor’, and ‘xreadlines’ modules have been removed.
* Menu:
* cookielib::
* doctest: doctest<5>.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc3548.html
File: python.info, Node: cookielib, Next: doctest<5>, Up: New Improved and Deprecated Modules<3>
1.18.12.1 cookielib
...................
The *note cookielib: 70. library supports client-side handling for HTTP
cookies, mirroring the *note Cookie: 71. module’s server-side cookie
support. Cookies are stored in cookie jars; the library transparently
stores cookies offered by the web server in the cookie jar, and fetches
the cookie from the jar when connecting to the server. As in web
browsers, policy objects control whether cookies are accepted or not.
In order to store cookies across sessions, two implementations of cookie
jars are provided: one that stores cookies in the Netscape format so
applications can use the Mozilla or Lynx cookie files, and one that
stores cookies in the same format as the Perl libwww library.
*note urllib2: 10b. has been changed to interact with *note cookielib:
70.: ‘HTTPCookieProcessor’ manages a cookie jar that is used when
accessing URLs.
This module was contributed by John J. Lee.
File: python.info, Node: doctest<5>, Prev: cookielib, Up: New Improved and Deprecated Modules<3>
1.18.12.2 doctest
.................
The *note doctest: 3a. module underwent considerable refactoring thanks
to Edward Loper and Tim Peters. Testing can still be as simple as
running *note doctest.testmod(): 1436, but the refactorings allow
customizing the module’s operation in various ways
The new ‘DocTestFinder’ class extracts the tests from a given object’s
docstrings:
def f (x, y):
""">>> f(2,2)
4
>>> f(3,2)
6
"""
return x*y
finder = doctest.DocTestFinder()
# Get list of DocTest instances
tests = finder.find(f)
The new ‘DocTestRunner’ class then runs individual tests and can produce
a summary of the results:
runner = doctest.DocTestRunner()
for t in tests:
tried, failed = runner.run(t)
runner.summarize(verbose=1)
The above example produces the following output:
1 items passed all tests:
2 tests in f
2 tests in 1 items.
2 passed and 0 failed.
Test passed.
‘DocTestRunner’ uses an instance of the ‘OutputChecker’ class to compare
the expected output with the actual output. This class takes a number
of different flags that customize its behaviour; ambitious users can
also write a completely new subclass of ‘OutputChecker’.
The default output checker provides a number of handy features. For
example, with the *note doctest.ELLIPSIS: 1437. option flag, an ellipsis
(‘...’) in the expected output matches any substring, making it easier
to accommodate outputs that vary in minor ways:
def o (n):
""">>> o(1)
<__main__.C instance at 0x...>
>>>
"""
Another special string, ‘’, matches a blank line:
def p (n):
""">>> p(1)
>>>
"""
Another new capability is producing a diff-style display of the output
by specifying the *note doctest.REPORT_UDIFF: 1438. (unified diffs),
*note doctest.REPORT_CDIFF: 1439. (context diffs), or *note
doctest.REPORT_NDIFF: 143a. (delta-style) option flags. For example:
def g (n):
""">>> g(4)
here
is
a
lengthy
>>>"""
L = 'here is a rather lengthy list of words'.split()
for word in L[:n]:
print word
Running the above function’s tests with *note doctest.REPORT_UDIFF:
1438. specified, you get the following output:
**********************************************************************
File "t.py", line 15, in g
Failed example:
g(4)
Differences (unified diff with -expected +actual):
@@ -2,3 +2,3 @@
is
a
-lengthy
+rather
**********************************************************************
File: python.info, Node: Build and C API Changes<11>, Next: Porting to Python 2 4, Prev: New Improved and Deprecated Modules<3>, Up: What’s New in Python 2 4
1.18.13 Build and C API Changes
-------------------------------
Some of the changes to Python’s build process and to the C API are:
* Three new convenience macros were added for common return values
from extension functions: *note Py_RETURN_NONE: 143c, *note
Py_RETURN_TRUE: 143d, and *note Py_RETURN_FALSE: 143e.
(Contributed by Brett Cannon.)
* Another new macro, *note Py_CLEAR: 575, decreases the reference
count of 'obj' and sets 'obj' to the null pointer. (Contributed by
Jim Fulton.)
* A new function, ‘PyTuple_Pack(N, obj1, obj2, ..., objN)’,
constructs tuples from a variable length argument list of Python
objects. (Contributed by Raymond Hettinger.)
* A new function, ‘PyDict_Contains(d, k)’, implements fast dictionary
lookups without masking exceptions raised during the look-up
process. (Contributed by Raymond Hettinger.)
* The Py_IS_NAN(X) macro returns 1 if its float or double argument
'X' is a NaN. (Contributed by Tim Peters.)
* C code can avoid unnecessary locking by using the new
‘PyEval_ThreadsInitialized()’ function to tell if any thread
operations have been performed. If this function returns false, no
lock operations are needed. (Contributed by Nick Coghlan.)
* A new function, *note PyArg_VaParseTupleAndKeywords(): 37f, is the
same as *note PyArg_ParseTupleAndKeywords(): 37e. but takes a
‘va_list’ instead of a number of arguments. (Contributed by Greg
Chapman.)
* A new method flag, *note METH_COEXIST: 143f, allows a function
defined in slots to co-exist with a *note PyCFunction: 1440. having
the same name. This can halve the access time for a method such as
‘set.__contains__()’. (Contributed by Raymond Hettinger.)
* Python can now be built with additional profiling for the
interpreter itself, intended as an aid to people developing the
Python core. Providing ‘--enable-profiling’ to the ‘configure’
script will let you profile the interpreter with ‘gprof’, and
providing the ‘--with-tsc’ switch enables profiling using the
Pentium’s Time-Stamp-Counter register. Note that the ‘--with-tsc’
switch is slightly misnamed, because the profiling feature also
works on the PowerPC platform, though that processor architecture
doesn’t call that register “the TSC register”. (Contributed by
Jeremy Hylton.)
* The ‘tracebackobject’ type has been renamed to ‘PyTracebackObject’.
* Menu:
* Port-Specific Changes: Port-Specific Changes<2>.
File: python.info, Node: Port-Specific Changes<2>, Up: Build and C API Changes<11>
1.18.13.1 Port-Specific Changes
...............................
* The Windows port now builds under MSVC++ 7.1 as well as version 6.
(Contributed by Martin von Löwis.)
File: python.info, Node: Porting to Python 2 4, Next: Acknowledgements<4>, Prev: Build and C API Changes<11>, Up: What’s New in Python 2 4
1.18.14 Porting to Python 2.4
-----------------------------
This section lists previously described changes that may require changes
to your code:
* Left shifts and hexadecimal/octal constants that are too large no
longer trigger a *note FutureWarning: 411. and return a value
limited to 32 or 64 bits; instead they return a long integer.
* Integer operations will no longer trigger an ‘OverflowWarning’.
The ‘OverflowWarning’ warning will disappear in Python 2.5.
* The *note zip(): 7cb. built-in function and ‘itertools.izip()’ now
return an empty list instead of raising a *note TypeError: 534.
exception if called with no arguments.
* You can no longer compare the ‘date’ and *note datetime: 1cc.
instances provided by the *note datetime: 30. module. Two
instances of different classes will now always be unequal, and
relative comparisons (‘<’, ‘>’) will raise a *note TypeError: 534.
* ‘dircache.listdir()’ now passes exceptions to the caller instead of
returning empty lists.
* ‘LexicalHandler.startDTD()’ used to receive the public and system
IDs in the wrong order. This has been corrected; applications
relying on the wrong order need to be fixed.
* *note fcntl.ioctl(): 1443. now warns if the 'mutate' argument is
omitted and relevant.
* The *note tarfile: de. module now generates GNU-format tar files by
default.
* Encountering a failure while importing a module no longer leaves a
partially initialized module object in ‘sys.modules’.
* *note None: 671. is now a constant; code that binds a new value to
the name ‘None’ is now a syntax error.
* The ‘signals.signal()’ function now raises a *note RuntimeError:
195. exception for certain illegal values; previously these errors
would pass silently. For example, you can no longer set a handler
on the ‘SIGKILL’ signal.
File: python.info, Node: Acknowledgements<4>, Prev: Porting to Python 2 4, Up: What’s New in Python 2 4
1.18.15 Acknowledgements
------------------------
The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Koray Can, Hye-Shik Chang, Michael Dyck, Raymond Hettinger,
Brian Hurt, Hamish Lawson, Fredrik Lundh, Sean Reifschneider, Sadruddin
Rejeb.
File: python.info, Node: What’s New in Python 2 3, Next: What’s New in Python 2 2, Prev: What’s New in Python 2 4, Up: What’s New in Python
1.19 What’s New in Python 2.3
=============================
Author: A.M. Kuchling
This article explains the new features in Python 2.3. Python 2.3 was
released on July 29, 2003.
The main themes for Python 2.3 are polishing some of the features added
in 2.2, adding various small but useful enhancements to the core
language, and expanding the standard library. The new object model
introduced in the previous version has benefited from 18 months of
bugfixes and from optimization efforts that have improved the
performance of new-style classes. A few new built-in functions have
been added such as *note sum(): 466. and *note enumerate(): 1448. The
*note in: 2ee. operator can now be used for substring searches (e.g.
‘"ab" in "abc"’ returns *note True: c0d.).
Some of the many new library features include Boolean, set, heap, and
date/time data types, the ability to import modules from ZIP-format
archives, metadata support for the long-awaited Python catalog, an
updated version of IDLE, and modules for logging messages, wrapping
text, parsing CSV files, processing command-line options, using
BerkeleyDB databases… the list of new and enhanced modules is lengthy.
This article doesn’t attempt to provide a complete specification of the
new features, but instead provides a convenient overview. For full
details, you should refer to the documentation for Python 2.3, such as
the Python Library Reference and the Python Reference Manual. If you
want to understand the complete implementation and design rationale,
refer to the PEP for a particular new feature.
* Menu:
* PEP 218; A Standard Set Datatype: PEP 218 A Standard Set Datatype.
* PEP 255; Simple Generators: PEP 255 Simple Generators.
* PEP 263; Source Code Encodings: PEP 263 Source Code Encodings.
* PEP 273; Importing Modules from ZIP Archives: PEP 273 Importing Modules from ZIP Archives.
* PEP 277; Unicode file name support for Windows NT: PEP 277 Unicode file name support for Windows NT.
* PEP 278; Universal Newline Support: PEP 278 Universal Newline Support.
* PEP 279; enumerate(): PEP 279 enumerate.
* PEP 282; The logging Package: PEP 282 The logging Package.
* PEP 285; A Boolean Type: PEP 285 A Boolean Type.
* PEP 293; Codec Error Handling Callbacks: PEP 293 Codec Error Handling Callbacks.
* PEP 301; Package Index and Metadata for Distutils: PEP 301 Package Index and Metadata for Distutils.
* PEP 302; New Import Hooks: PEP 302 New Import Hooks.
* PEP 305; Comma-separated Files: PEP 305 Comma-separated Files.
* PEP 307; Pickle Enhancements: PEP 307 Pickle Enhancements.
* Extended Slices::
* Other Language Changes: Other Language Changes<18>.
* New, Improved, and Deprecated Modules: New Improved and Deprecated Modules<4>.
* Pymalloc; A Specialized Object Allocator: Pymalloc A Specialized Object Allocator.
* Build and C API Changes: Build and C API Changes<12>.
* Other Changes and Fixes: Other Changes and Fixes<2>.
* Porting to Python 2.3: Porting to Python 2 3.
* Acknowledgements: Acknowledgements<5>.
File: python.info, Node: PEP 218 A Standard Set Datatype, Next: PEP 255 Simple Generators, Up: What’s New in Python 2 3
1.19.1 PEP 218: A Standard Set Datatype
---------------------------------------
The new ‘sets’ module contains an implementation of a set datatype. The
‘Set’ class is for mutable sets, sets that can have members added and
removed. The ‘ImmutableSet’ class is for sets that can’t be modified,
and instances of ‘ImmutableSet’ can therefore be used as dictionary
keys. Sets are built on top of dictionaries, so the elements within a
set must be hashable.
Here’s a simple example:
>>> import sets
>>> S = sets.Set([1,2,3])
>>> S
Set([1, 2, 3])
>>> 1 in S
True
>>> 0 in S
False
>>> S.add(5)
>>> S.remove(3)
>>> S
Set([1, 2, 5])
>>>
The union and intersection of sets can be computed with the *note
union(): 144a. and *note intersection(): 144b. methods; an alternative
notation uses the bitwise operators ‘&’ and ‘|’. Mutable sets also have
in-place versions of these methods, ‘union_update()’ and *note
intersection_update(): 144c.
>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([4,5,6])
>>> S1.union(S2)
Set([1, 2, 3, 4, 5, 6])
>>> S1 | S2 # Alternative notation
Set([1, 2, 3, 4, 5, 6])
>>> S1.intersection(S2)
Set([])
>>> S1 & S2 # Alternative notation
Set([])
>>> S1.union_update(S2)
>>> S1
Set([1, 2, 3, 4, 5, 6])
>>>
It’s also possible to take the symmetric difference of two sets. This
is the set of all elements in the union that aren’t in the intersection.
Another way of putting it is that the symmetric difference contains all
elements that are in exactly one set. Again, there’s an alternative
notation (‘^’), and an in-place version with the ungainly name *note
symmetric_difference_update(): 144d.
>>> S1 = sets.Set([1,2,3,4])
>>> S2 = sets.Set([3,4,5,6])
>>> S1.symmetric_difference(S2)
Set([1, 2, 5, 6])
>>> S1 ^ S2
Set([1, 2, 5, 6])
>>>
There are also ‘issubset()’ and ‘issuperset()’ methods for checking
whether one set is a subset or superset of another:
>>> S1 = sets.Set([1,2,3])
>>> S2 = sets.Set([2,3])
>>> S2.issubset(S1)
True
>>> S1.issubset(S2)
False
>>> S1.issuperset(S2)
True
>>>
See also
........
PEP 218(1) - Adding a Built-In Set Object Type
PEP written by Greg V. Wilson. Implemented by Greg V. Wilson, Alex
Martelli, and GvR.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0218/
File: python.info, Node: PEP 255 Simple Generators, Next: PEP 263 Source Code Encodings, Prev: PEP 218 A Standard Set Datatype, Up: What’s New in Python 2 3
1.19.2 PEP 255: Simple Generators
---------------------------------
In Python 2.2, generators were added as an optional feature, to be
enabled by a ‘from __future__ import generators’ directive. In 2.3
generators no longer need to be specially enabled, and are now always
present; this means that *note yield: 9cd. is now always a keyword. The
rest of this section is a copy of the description of generators from the
“What’s New in Python 2.2” document; if you read it back when Python 2.2
came out, you can skip the rest of this section.
You’re doubtless familiar with how function calls work in Python or C.
When you call a function, it gets a private namespace where its local
variables are created. When the function reaches a *note return: 9ce.
statement, the local variables are destroyed and the resulting value is
returned to the caller. A later call to the same function will get a
fresh new set of local variables. But, what if the local variables
weren’t thrown away on exiting a function? What if you could later
resume the function where it left off? This is what generators provide;
they can be thought of as resumable functions.
Here’s the simplest example of a generator function:
def generate_ints(N):
for i in range(N):
yield i
A new keyword, *note yield: 9cd, was introduced for generators. Any
function containing a ‘yield’ statement is a generator function; this is
detected by Python’s bytecode compiler which compiles the function
specially as a result.
When you call a generator function, it doesn’t return a single value;
instead it returns a generator object that supports the iterator
protocol. On executing the *note yield: 9cd. statement, the generator
outputs the value of ‘i’, similar to a *note return: 9ce. statement.
The big difference between ‘yield’ and a ‘return’ statement is that on
reaching a ‘yield’ the generator’s state of execution is suspended and
local variables are preserved. On the next call to the generator’s
‘.next()’ method, the function will resume executing immediately after
the ‘yield’ statement. (For complicated reasons, the ‘yield’ statement
isn’t allowed inside the *note try: 6e4. block of a ‘try’…‘finally’
statement; read PEP 255(1) for a full explanation of the interaction
between ‘yield’ and exceptions.)
Here’s a sample usage of the ‘generate_ints()’ generator:
>>> gen = generate_ints(3)
>>> gen
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
File "stdin", line 1, in ?
File "stdin", line 2, in generate_ints
StopIteration
You could equally write ‘for i in generate_ints(5)’, or ‘a,b,c =
generate_ints(3)’.
Inside a generator function, the *note return: 9ce. statement can only
be used without a value, and signals the end of the procession of
values; afterwards the generator cannot return any further values.
‘return’ with a value, such as ‘return 5’, is a syntax error inside a
generator function. The end of the generator’s results can also be
indicated by raising *note StopIteration: bfa. manually, or by just
letting the flow of execution fall off the bottom of the function.
You could achieve the effect of generators manually by writing your own
class and storing all the local variables of the generator as instance
variables. For example, returning a list of integers could be done by
setting ‘self.count’ to 0, and having the *note next(): 7d3. method
increment ‘self.count’ and return it. However, for a moderately
complicated generator, writing a corresponding class would be much
messier. ‘Lib/test/test_generators.py’ contains a number of more
interesting examples. The simplest one implements an in-order traversal
of a tree using generators recursively.
# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
if t:
for x in inorder(t.left):
yield x
yield t.label
for x in inorder(t.right):
yield x
Two other examples in ‘Lib/test/test_generators.py’ produce solutions
for the N-Queens problem (placing $N$ queens on an $NxN$ chess board so
that no queen threatens another) and the Knight’s Tour (a route that
takes a knight to every square of an $NxN$ chessboard without visiting
any square twice).
The idea of generators comes from other programming languages,
especially Icon (‘https://www2.cs.arizona.edu/icon/’), where the idea of
generators is central. In Icon, every expression and function call
behaves like a generator. One example from “An Overview of the Icon
Programming Language” at
‘https://www2.cs.arizona.edu/icon/docs/ipd266.htm’ gives an idea of what
this looks like:
sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)
In Icon the ‘find()’ function returns the indexes at which the substring
“or” is found: 3, 23, 33. In the *note if: 2ed. statement, ‘i’ is first
assigned a value of 3, but 3 is less than 5, so the comparison fails,
and Icon retries it with the second value of 23. 23 is greater than 5,
so the comparison now succeeds, and the code prints the value 23 to the
screen.
Python doesn’t go nearly as far as Icon in adopting generators as a
central concept. Generators are considered part of the core Python
language, but learning or using them isn’t compulsory; if they don’t
solve any problems that you have, feel free to ignore them. One novel
feature of Python’s interface as compared to Icon’s is that a
generator’s state is represented as a concrete object (the iterator)
that can be passed around to other functions or stored in a data
structure.
See also
........
PEP 255(2) - Simple Generators
Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland.
Implemented mostly by Neil Schemenauer and Tim Peters, with other
fixes from the Python Labs crew.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0255/
(2) https://peps.python.org/pep-0255/
File: python.info, Node: PEP 263 Source Code Encodings, Next: PEP 273 Importing Modules from ZIP Archives, Prev: PEP 255 Simple Generators, Up: What’s New in Python 2 3
1.19.3 PEP 263: Source Code Encodings
-------------------------------------
Python source files can now be declared as being in different character
set encodings. Encodings are declared by including a specially
formatted comment in the first or second line of the source file. For
example, a UTF-8 file can be declared with:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
Without such an encoding declaration, the default encoding used is 7-bit
ASCII. Executing or importing modules that contain string literals with
8-bit characters and have no encoding declaration will result in a *note
DeprecationWarning: 1a5. being signalled by Python 2.3; in 2.4 this will
be a syntax error.
The encoding declaration only affects Unicode string literals, which
will be converted to Unicode using the specified encoding. Note that
Python identifiers are still restricted to ASCII characters, so you
can’t have variable names that use characters outside of the usual
alphanumerics.
See also
........
PEP 263(1) - Defining Python Source Code Encodings
Written by Marc-André Lemburg and Martin von Löwis; implemented by
Suzuki Hisao and Martin von Löwis.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0263/
File: python.info, Node: PEP 273 Importing Modules from ZIP Archives, Next: PEP 277 Unicode file name support for Windows NT, Prev: PEP 263 Source Code Encodings, Up: What’s New in Python 2 3
1.19.4 PEP 273: Importing Modules from ZIP Archives
---------------------------------------------------
The new *note zipimport: 132. module adds support for importing modules
from a ZIP-format archive. You don’t need to import the module
explicitly; it will be automatically imported if a ZIP archive’s
filename is added to ‘sys.path’. For example:
amk@nyman:~/src/python$ unzip -l /tmp/example.zip
Archive: /tmp/example.zip
Length Date Time Name
-------- ---- ---- ----
8467 11-26-02 22:30 jwzthreading.py
-------- -------
8467 1 file
amk@nyman:~/src/python$ ./python
Python 2.3 (#1, Aug 1 2003, 19:54:32)
>>> import sys
>>> sys.path.insert(0, '/tmp/example.zip') # Add .zip file to front of path
>>> import jwzthreading
>>> jwzthreading.__file__
'/tmp/example.zip/jwzthreading.py'
>>>
An entry in ‘sys.path’ can now be the filename of a ZIP archive. The
ZIP archive can contain any kind of files, but only files named ‘*.py’,
‘*.pyc’, or ‘*.pyo’ can be imported. If an archive only contains ‘*.py’
files, Python will not attempt to modify the archive by adding the
corresponding ‘*.pyc’ file, meaning that if a ZIP archive doesn’t
contain ‘*.pyc’ files, importing may be rather slow.
A path within the archive can also be specified to only import from a
subdirectory; for example, the path ‘/tmp/example.zip/lib/’ would only
import from the ‘lib/’ subdirectory within the archive.
See also
........
PEP 273(1) - Import Modules from Zip Archives
Written by James C. Ahlstrom, who also provided an implementation.
Python 2.3 follows the specification in PEP 273(2), but uses an
implementation written by Just van Rossum that uses the import
hooks described in PEP 302(3). See section *note PEP 302; New
Import Hooks: 1453. for a description of the new import hooks.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0273/
(2) https://peps.python.org/pep-0273/
(3) https://peps.python.org/pep-0302/
File: python.info, Node: PEP 277 Unicode file name support for Windows NT, Next: PEP 278 Universal Newline Support, Prev: PEP 273 Importing Modules from ZIP Archives, Up: What’s New in Python 2 3
1.19.5 PEP 277: Unicode file name support for Windows NT
--------------------------------------------------------
On Windows NT, 2000, and XP, the system stores file names as Unicode
strings. Traditionally, Python has represented file names as byte
strings, which is inadequate because it renders some file names
inaccessible.
Python now allows using arbitrary Unicode strings (within the
limitations of the file system) for all functions that expect file
names, most notably the *note open(): 517. built-in function. If a
Unicode string is passed to *note os.listdir(): 10ee, Python now returns
a list of Unicode strings. A new function, ‘os.getcwdu()’, returns the
current directory as a Unicode string.
Byte strings still work as file names, and on Windows Python will
transparently convert them to Unicode using the ‘mbcs’ encoding.
Other systems also allow Unicode strings as file names but convert them
to byte strings before passing them to the system, which can cause a
*note UnicodeError: 1296. to be raised. Applications can test whether
arbitrary Unicode strings are supported as file names by checking *note
os.path.supports_unicode_filenames: 1455, a Boolean value.
Under MacOS, *note os.listdir(): 10ee. may now return Unicode filenames.
See also
........
PEP 277(1) - Unicode file name support for Windows NT
Written by Neil Hodgson; implemented by Neil Hodgson, Martin von
Löwis, and Mark Hammond.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0277/
File: python.info, Node: PEP 278 Universal Newline Support, Next: PEP 279 enumerate, Prev: PEP 277 Unicode file name support for Windows NT, Up: What’s New in Python 2 3
1.19.6 PEP 278: Universal Newline Support
-----------------------------------------
The three major operating systems used today are Microsoft Windows,
Apple’s Macintosh OS, and the various Unix derivatives. A minor
irritation of cross-platform work is that these three platforms all use
different characters to mark the ends of lines in text files. Unix uses
the linefeed (ASCII character 10), MacOS uses the carriage return (ASCII
character 13), and Windows uses a two-character sequence of a carriage
return plus a newline.
Python’s file objects can now support end of line conventions other than
the one followed by the platform on which Python is running. Opening a
file with the mode ‘'U'’ or ‘'rU'’ will open a file for reading in *note
universal newlines: d3a. mode. All three line ending conventions will
be translated to a ‘'\n'’ in the strings returned by the various file
methods such as ‘read()’ and ‘readline()’.
Universal newline support is also used when importing modules and when
executing a file with the ‘execfile()’ function. This means that Python
modules can be shared between all three operating systems without
needing to convert the line-endings.
This feature can be disabled when compiling Python by specifying the
‘--without-universal-newlines’ switch when running Python’s ‘configure’
script.
See also
........
PEP 278(1) - Universal Newline Support
Written and implemented by Jack Jansen.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0278/
File: python.info, Node: PEP 279 enumerate, Next: PEP 282 The logging Package, Prev: PEP 278 Universal Newline Support, Up: What’s New in Python 2 3
1.19.7 PEP 279: enumerate()
---------------------------
A new built-in function, *note enumerate(): 1448, will make certain
loops a bit clearer. ‘enumerate(thing)’, where 'thing' is either an
iterator or a sequence, returns an iterator that will return ‘(0,
thing[0])’, ‘(1, thing[1])’, ‘(2, thing[2])’, and so forth.
A common idiom to change every element of a list looks like this:
for i in range(len(L)):
item = L[i]
# ... compute some result based on item ...
L[i] = result
This can be rewritten using *note enumerate(): 1448. as:
for i, item in enumerate(L):
# ... compute some result based on item ...
L[i] = result
See also
........
PEP 279(1) - The enumerate() built-in function
Written and implemented by Raymond D. Hettinger.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0279/
File: python.info, Node: PEP 282 The logging Package, Next: PEP 285 A Boolean Type, Prev: PEP 279 enumerate, Up: What’s New in Python 2 3
1.19.8 PEP 282: The logging Package
-----------------------------------
A standard package for writing logs, *note logging: 87, has been added
to Python 2.3. It provides a powerful and flexible mechanism for
generating logging output which can then be filtered and processed in
various ways. A configuration file written in a standard format can be
used to control the logging behavior of a program. Python includes
handlers that will write log records to standard error or to a file or
socket, send them to the system log, or even e-mail them to a particular
address; of course, it’s also possible to write your own handler
classes.
The *note Logger: b4f. class is the primary class. Most application
code will deal with one or more *note Logger: b4f. objects, each one
used by a particular subsystem of the application. Each *note Logger:
b4f. is identified by a name, and names are organized into a hierarchy
using ‘.’ as the component separator. For example, you might have *note
Logger: b4f. instances named ‘server’, ‘server.auth’ and
‘server.network’. The latter two instances are below ‘server’ in the
hierarchy. This means that if you turn up the verbosity for ‘server’ or
direct ‘server’ messages to a different handler, the changes will also
apply to records logged to ‘server.auth’ and ‘server.network’. There’s
also a root *note Logger: b4f. that’s the parent of all other loggers.
For simple uses, the *note logging: 87. package contains some
convenience functions that always use the root log:
import logging
logging.debug('Debugging information')
logging.info('Informational message')
logging.warning('Warning:config file %s not found', 'server.conf')
logging.error('Error occurred')
logging.critical('Critical error -- shutting down')
This produces the following output:
WARNING:root:Warning:config file server.conf not found
ERROR:root:Error occurred
CRITICAL:root:Critical error -- shutting down
In the default configuration, informational and debugging messages are
suppressed and the output is sent to standard error. You can enable the
display of informational and debugging messages by calling the *note
setLevel(): 145a. method on the root logger.
Notice the *note warning(): 2f9. call’s use of string formatting
operators; all of the functions for logging messages take the arguments
‘(msg, arg1, arg2, ...)’ and log the string resulting from ‘msg % (arg1,
arg2, ...)’.
There’s also an *note exception(): 145b. function that records the most
recent traceback. Any of the other functions will also record the
traceback if you specify a true value for the keyword argument
'exc_info'.
def f():
try: 1/0
except: logging.exception('Problem recorded')
f()
This produces the following output:
ERROR:root:Problem recorded
Traceback (most recent call last):
File "t.py", line 6, in f
1/0
ZeroDivisionError: integer division or modulo by zero
Slightly more advanced programs will use a logger other than the root
logger. The ‘getLogger(name)’ function is used to get a particular log,
creating it if it doesn’t exist yet. ‘getLogger(None)’ returns the root
logger.
log = logging.getLogger('server')
...
log.info('Listening on port %i', port)
...
log.critical('Disk full')
...
Log records are usually propagated up the hierarchy, so a message logged
to ‘server.auth’ is also seen by ‘server’ and ‘root’, but a *note
Logger: b4f. can prevent this by setting its *note propagate: 145c.
attribute to *note False: b37.
There are more classes provided by the *note logging: 87. package that
can be customized. When a *note Logger: b4f. instance is told to log a
message, it creates a *note LogRecord: fe1. instance that is sent to any
number of different *note Handler: 145d. instances. Loggers and
handlers can also have an attached list of filters, and each filter can
cause the *note LogRecord: fe1. to be ignored or can modify the record
before passing it along. When they’re finally output, *note LogRecord:
fe1. instances are converted to text by a *note Formatter: 145e. class.
All of these classes can be replaced by your own specially written
classes.
With all of these features the *note logging: 87. package should provide
enough flexibility for even the most complicated applications. This is
only an incomplete overview of its features, so please see the package’s
reference documentation for all of the details. Reading PEP 282(1) will
also be helpful.
See also
........
PEP 282(2) - A Logging System
Written by Vinay Sajip and Trent Mick; implemented by Vinay Sajip.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0282/
(2) https://peps.python.org/pep-0282/
File: python.info, Node: PEP 285 A Boolean Type, Next: PEP 293 Codec Error Handling Callbacks, Prev: PEP 282 The logging Package, Up: What’s New in Python 2 3
1.19.9 PEP 285: A Boolean Type
------------------------------
A Boolean type was added to Python 2.3. Two new constants were added to
the ‘__builtin__’ module, *note True: c0d. and *note False: b37. (*note
True: c0d. and *note False: b37. constants were added to the built-ins
in Python 2.2.1, but the 2.2.1 versions are simply set to integer values
of 1 and 0 and aren’t a different type.)
The type object for this new type is named *note bool: 463.; the
constructor for it takes any Python value and converts it to *note True:
c0d. or *note False: b37.
>>> bool(1)
True
>>> bool(0)
False
>>> bool([])
False
>>> bool( (1,) )
True
Most of the standard library modules and built-in functions have been
changed to return Booleans.
>>> obj = []
>>> hasattr(obj, 'append')
True
>>> isinstance(obj, list)
True
>>> isinstance(obj, tuple)
False
Python’s Booleans were added with the primary goal of making code
clearer. For example, if you’re reading a function and encounter the
statement ‘return 1’, you might wonder whether the ‘1’ represents a
Boolean truth value, an index, or a coefficient that multiplies some
other quantity. If the statement is ‘return True’, however, the meaning
of the return value is quite clear.
Python’s Booleans were 'not' added for the sake of strict type-checking.
A very strict language such as Pascal would also prevent you performing
arithmetic with Booleans, and would require that the expression in an
*note if: 2ed. statement always evaluate to a Boolean result. Python is
not this strict and never will be, as PEP 285(1) explicitly says. This
means you can still use any expression in an ‘if’ statement, even ones
that evaluate to a list or tuple or some random object. The Boolean
type is a subclass of the *note int: 259. class so that arithmetic using
a Boolean still works.
>>> True + 1
2
>>> False + 1
1
>>> False * 75
0
>>> True * 75
75
To sum up *note True: c0d. and *note False: b37. in a sentence: they’re
alternative ways to spell the integer values 1 and 0, with the single
difference that *note str(): 447. and *note repr(): 7f9. return the
strings ‘'True'’ and ‘'False'’ instead of ‘'1'’ and ‘'0'’.
See also
........
PEP 285(2) - Adding a bool type
Written and implemented by GvR.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0285/
(2) https://peps.python.org/pep-0285/
File: python.info, Node: PEP 293 Codec Error Handling Callbacks, Next: PEP 301 Package Index and Metadata for Distutils, Prev: PEP 285 A Boolean Type, Up: What’s New in Python 2 3
1.19.10 PEP 293: Codec Error Handling Callbacks
-----------------------------------------------
When encoding a Unicode string into a byte string, unencodable
characters may be encountered. So far, Python has allowed specifying
the error processing as either “strict” (raising *note UnicodeError:
1296.), “ignore” (skipping the character), or “replace” (using a
question mark in the output string), with “strict” being the default
behavior. It may be desirable to specify alternative processing of such
errors, such as inserting an XML character reference or HTML entity
reference into the converted string.
Python now has a flexible framework to add different processing
strategies. New error handlers can be added with *note
codecs.register_error(): 1462, and codecs then can access the error
handler with *note codecs.lookup_error(): 1463. An equivalent C API has
been added for codecs written in C. The error handler gets the necessary
state information such as the string being converted, the position in
the string where the error was detected, and the target encoding. The
handler can then either raise an exception or return a replacement
string.
Two additional error handlers have been implemented using this
framework: “backslashreplace” uses Python backslash quoting to represent
unencodable characters and “xmlcharrefreplace” emits XML character
references.
See also
........
PEP 293(1) - Codec Error Handling Callbacks
Written and implemented by Walter Dörwald.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0293/
File: python.info, Node: PEP 301 Package Index and Metadata for Distutils, Next: PEP 302 New Import Hooks, Prev: PEP 293 Codec Error Handling Callbacks, Up: What’s New in Python 2 3
1.19.11 PEP 301: Package Index and Metadata for Distutils
---------------------------------------------------------
Support for the long-requested Python catalog makes its first appearance
in 2.3.
The heart of the catalog is the new Distutils ‘register’ command.
Running ‘python setup.py register’ will collect the metadata describing
a package, such as its name, version, maintainer, description, &c., and
send it to a central catalog server. The resulting catalog is available
from ‘https://pypi.org’.
To make the catalog a bit more useful, a new optional 'classifiers'
keyword argument has been added to the Distutils ‘setup()’ function. A
list of Trove(1)-style strings can be supplied to help classify the
software.
Here’s an example ‘setup.py’ with classifiers, written to be compatible
with older versions of the Distutils:
from distutils import core
kw = {'name': "Quixote",
'version': "0.5.1",
'description': "A highly Pythonic Web application framework",
# ...
}
if (hasattr(core, 'setup_keywords') and
'classifiers' in core.setup_keywords):
kw['classifiers'] = \
['Topic :: Internet :: WWW/HTTP :: Dynamic Content',
'Environment :: No Input/Output (Daemon)',
'Intended Audience :: Developers'],
core.setup(**kw)
The full list of classifiers can be obtained by running ‘python setup.py
register --list-classifiers’.
See also
........
PEP 301(2) - Package Index and Metadata for Distutils
Written and implemented by Richard Jones.
---------- Footnotes ----------
(1) http://catb.org/~esr/trove/
(2) https://peps.python.org/pep-0301/
File: python.info, Node: PEP 302 New Import Hooks, Next: PEP 305 Comma-separated Files, Prev: PEP 301 Package Index and Metadata for Distutils, Up: What’s New in Python 2 3
1.19.12 PEP 302: New Import Hooks
---------------------------------
While it’s been possible to write custom import hooks ever since the
‘ihooks’ module was introduced in Python 1.3, no one has ever been
really happy with it because writing new import hooks is difficult and
messy. There have been various proposed alternatives such as the
‘imputil’ and ‘iu’ modules, but none of them has ever gained much
acceptance, and none of them were easily usable from C code.
PEP 302(1) borrows ideas from its predecessors, especially from Gordon
McMillan’s ‘iu’ module. Three new items are added to the *note sys: d9.
module:
* ‘sys.path_hooks’ is a list of callable objects; most often they’ll
be classes. Each callable takes a string containing a path and
either returns an importer object that will handle imports from
this path or raises an *note ImportError: 415. exception if it
can’t handle this path.
* ‘sys.path_importer_cache’ caches importer objects for each path, so
‘sys.path_hooks’ will only need to be traversed once for each path.
* ‘sys.meta_path’ is a list of importer objects that will be
traversed before ‘sys.path’ is checked. This list is initially
empty, but user code can add objects to it. Additional built-in
and frozen modules can be imported by an object added to this list.
Importer objects must have a single method, ‘find_module(fullname,
path=None)’. 'fullname' will be a module or package name, e.g.
‘string’ or ‘distutils.core’. ‘find_module()’ must return a loader
object that has a single method, ‘load_module(fullname)’, that creates
and returns the corresponding module object.
Pseudo-code for Python’s new import logic, therefore, looks something
like this (simplified a bit; see PEP 302(2) for the full details):
for mp in sys.meta_path:
loader = mp(fullname)
if loader is not None:
= loader.load_module(fullname)
for path in sys.path:
for hook in sys.path_hooks:
try:
importer = hook(path)
except ImportError:
# ImportError, so try the other path hooks
pass
else:
loader = importer.find_module(fullname)
= loader.load_module(fullname)
# Not found!
raise ImportError
See also
........
PEP 302(3) - New Import Hooks
Written by Just van Rossum and Paul Moore. Implemented by Just van
Rossum.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0302/
(2) https://peps.python.org/pep-0302/
(3) https://peps.python.org/pep-0302/
File: python.info, Node: PEP 305 Comma-separated Files, Next: PEP 307 Pickle Enhancements, Prev: PEP 302 New Import Hooks, Up: What’s New in Python 2 3
1.19.13 PEP 305: Comma-separated Files
--------------------------------------
Comma-separated files are a format frequently used for exporting data
from databases and spreadsheets. Python 2.3 adds a parser for
comma-separated files.
Comma-separated format is deceptively simple at first glance:
Costs,150,200,3.95
Read a line and call ‘line.split(',')’: what could be simpler? But toss
in string data that can contain commas, and things get more complicated:
"Costs",150,200,3.95,"Includes taxes, shipping, and sundry items"
A big ugly regular expression can parse this, but using the new *note
csv: 29. package is much simpler:
import csv
input = open('datafile', 'rb')
reader = csv.reader(input)
for line in reader:
print line
The *note reader(): 482. function takes a number of different options.
The field separator isn’t limited to the comma and can be changed to any
character, and so can the quoting and line-ending characters.
Different dialects of comma-separated files can be defined and
registered; currently there are two dialects, both used by Microsoft
Excel. A separate *note csv.writer: 483. class will generate
comma-separated files from a succession of tuples or lists, quoting
strings that contain the delimiter.
See also
........
PEP 305(1) - CSV File API
Written and implemented by Kevin Altis, Dave Cole, Andrew McNamara,
Skip Montanaro, Cliff Wells.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0305/
File: python.info, Node: PEP 307 Pickle Enhancements, Next: Extended Slices, Prev: PEP 305 Comma-separated Files, Up: What’s New in Python 2 3
1.19.14 PEP 307: Pickle Enhancements
------------------------------------
The *note pickle: a6. and ‘cPickle’ modules received some attention
during the 2.3 development cycle. In 2.2, new-style classes could be
pickled without difficulty, but they weren’t pickled very compactly; PEP
307(1) quotes a trivial example where a new-style class results in a
pickled string three times longer than that for a classic class.
The solution was to invent a new pickle protocol. The *note
pickle.dumps(): 146b. function has supported a text-or-binary flag for a
long time. In 2.3, this flag is redefined from a Boolean to an integer:
0 is the old text-mode pickle format, 1 is the old binary format, and
now 2 is a new 2.3-specific format. A new constant, *note
pickle.HIGHEST_PROTOCOL: 146c, can be used to select the fanciest
protocol available.
Unpickling is no longer considered a safe operation. 2.2’s *note
pickle: a6. provided hooks for trying to prevent unsafe classes from
being unpickled (specifically, a ‘__safe_for_unpickling__’ attribute),
but none of this code was ever audited and therefore it’s all been
ripped out in 2.3. You should not unpickle untrusted data in any
version of Python.
To reduce the pickling overhead for new-style classes, a new interface
for customizing pickling was added using three special methods: *note
__getstate__(): 5d4, *note __setstate__(): 146d, and *note
__getnewargs__(): 146e. Consult PEP 307(2) for the full semantics of
these methods.
As a way to compress pickles yet further, it’s now possible to use
integer codes instead of long strings to identify pickled classes. The
Python Software Foundation will maintain a list of standardized codes;
there’s also a range of codes for private use. Currently no codes have
been specified.
See also
........
PEP 307(3) - Extensions to the pickle protocol
Written and implemented by Guido van Rossum and Tim Peters.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0307/
(2) https://peps.python.org/pep-0307/
(3) https://peps.python.org/pep-0307/
File: python.info, Node: Extended Slices, Next: Other Language Changes<18>, Prev: PEP 307 Pickle Enhancements, Up: What’s New in Python 2 3
1.19.15 Extended Slices
-----------------------
Ever since Python 1.4, the slicing syntax has supported an optional
third “step” or “stride” argument. For example, these are all legal
Python syntax: ‘L[1:10:2]’, ‘L[:-1:1]’, ‘L[::-1]’. This was added to
Python at the request of the developers of Numerical Python, which uses
the third argument extensively. However, Python’s built-in list, tuple,
and string sequence types have never supported this feature, raising a
*note TypeError: 534. if you tried it. Michael Hudson contributed a
patch to fix this shortcoming.
For example, you can now easily extract the elements of a list that have
even indexes:
>>> L = range(10)
>>> L[::2]
[0, 2, 4, 6, 8]
Negative values also work to make a copy of the same list in reverse
order:
>>> L[::-1]
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
This also works for tuples, arrays, and strings:
>>> s='abcd'
>>> s[::2]
'ac'
>>> s[::-1]
'dcba'
If you have a mutable sequence such as a list or an array you can assign
to or delete an extended slice, but there are some differences between
assignment to extended and regular slices. Assignment to a regular
slice can be used to change the length of the sequence:
>>> a = range(3)
>>> a
[0, 1, 2]
>>> a[1:3] = [4, 5, 6]
>>> a
[0, 4, 5, 6]
Extended slices aren’t this flexible. When assigning to an extended
slice, the list on the right hand side of the statement must contain the
same number of items as the slice it is replacing:
>>> a = range(4)
>>> a
[0, 1, 2, 3]
>>> a[::2]
[0, 2]
>>> a[::2] = [0, -1]
>>> a
[0, 1, -1, 3]
>>> a[::2] = [0,1,2]
Traceback (most recent call last):
File "", line 1, in ?
ValueError: attempt to assign sequence of size 3 to extended slice of size 2
Deletion is more straightforward:
>>> a = range(4)
>>> a
[0, 1, 2, 3]
>>> a[::2]
[0, 2]
>>> del a[::2]
>>> a
[1, 3]
One can also now pass slice objects to the *note __getitem__(): 285.
methods of the built-in sequences:
>>> range(10).__getitem__(slice(0, 5, 2))
[0, 2, 4]
Or use slice objects directly in subscripts:
>>> range(10)[slice(0, 5, 2)]
[0, 2, 4]
To simplify implementing sequences that support extended slicing, slice
objects now have a method ‘indices(length)’ which, given the length of a
sequence, returns a ‘(start, stop, step)’ tuple that can be passed
directly to *note range(): 943. ‘indices()’ handles omitted and
out-of-bounds indices in a manner consistent with regular slices (and
this innocuous phrase hides a welter of confusing details!). The method
is intended to be used like this:
class FakeSeq:
...
def calc_item(self, i):
...
def __getitem__(self, item):
if isinstance(item, slice):
indices = item.indices(len(self))
return FakeSeq([self.calc_item(i) for i in range(*indices)])
else:
return self.calc_item(i)
From this example you can also see that the built-in *note slice: 465.
object is now the type object for the slice type, and is no longer a
function. This is consistent with Python 2.2, where *note int: 259,
*note str: 447, etc., underwent the same change.
File: python.info, Node: Other Language Changes<18>, Next: New Improved and Deprecated Modules<4>, Prev: Extended Slices, Up: What’s New in Python 2 3
1.19.16 Other Language Changes
------------------------------
Here are all of the changes that Python 2.3 makes to the core Python
language.
* The *note yield: 9cd. statement is now always a keyword, as
described in section *note PEP 255; Simple Generators: 144f. of
this document.
* A new built-in function *note enumerate(): 1448. was added, as
described in section *note PEP 279; enumerate(): 1458. of this
document.
* Two new constants, *note True: c0d. and *note False: b37. were
added along with the built-in *note bool: 463. type, as described
in section *note PEP 285; A Boolean Type: 1460. of this document.
* The *note int(): 259. type constructor will now return a long
integer instead of raising an *note OverflowError: 87f. when a
string or floating-point number is too large to fit into an
integer. This can lead to the paradoxical result that
‘isinstance(int(expression), int)’ is false, but that seems
unlikely to cause problems in practice.
* Built-in types now support the extended slicing syntax, as
described in section *note Extended Slices: 1470. of this document.
* A new built-in function, ‘sum(iterable, start=0)’, adds up the
numeric items in the iterable object and returns their sum. *note
sum(): 466. only accepts numbers, meaning that you can’t use it to
concatenate a bunch of strings. (Contributed by Alex Martelli.)
* ‘list.insert(pos, value)’ used to insert 'value' at the front of
the list when 'pos' was negative. The behaviour has now been
changed to be consistent with slice indexing, so when 'pos' is -1
the value will be inserted before the last element, and so forth.
* ‘list.index(value)’, which searches for 'value' within the list and
returns its index, now takes optional 'start' and 'stop' arguments
to limit the search to only part of the list.
* Dictionaries have a new method, ‘pop(key[, *default*])’, that
returns the value corresponding to 'key' and removes that key/value
pair from the dictionary. If the requested key isn’t present in
the dictionary, 'default' is returned if it’s specified and *note
KeyError: 33f. raised if it isn’t.
>>> d = {1:2}
>>> d
{1: 2}
>>> d.pop(4)
Traceback (most recent call last):
File "stdin", line 1, in ?
KeyError: 4
>>> d.pop(1)
2
>>> d.pop(1)
Traceback (most recent call last):
File "stdin", line 1, in ?
KeyError: 'pop(): dictionary is empty'
>>> d
{}
>>>
There’s also a new class method, ‘dict.fromkeys(iterable, value)’,
that creates a dictionary with keys taken from the supplied
iterator 'iterable' and all values set to 'value', defaulting to
‘None’.
(Patches contributed by Raymond Hettinger.)
Also, the *note dict(): 258. constructor now accepts keyword
arguments to simplify creating small dictionaries:
>>> dict(red=1, blue=2, green=3, black=4)
{'blue': 2, 'black': 4, 'green': 3, 'red': 1}
(Contributed by Just van Rossum.)
* The *note assert: 968. statement no longer checks the ‘__debug__’
flag, so you can no longer disable assertions by assigning to
‘__debug__’. Running Python with the *note -O: db4. switch will
still generate code that doesn’t execute any assertions.
* Most type objects are now callable, so you can use them to create
new objects such as functions, classes, and modules. (This means
that the ‘new’ module can be deprecated in a future Python version,
because you can now use the type objects available in the *note
types: 103. module.) For example, you can create a new module
object with the following code:
>>> import types
>>> m = types.ModuleType('abc','docstring')
>>> m
>>> m.__doc__
'docstring'
* A new warning, *note PendingDeprecationWarning: 8c7. was added to
indicate features which are in the process of being deprecated.
The warning will 'not' be printed by default. To check for use of
features that will be deprecated in the future, supply *note
-Walways;;PendingDeprecationWarning;;: 8c6. on the command line or
use *note warnings.filterwarnings(): 1472.
* The process of deprecating string-based exceptions, as in ‘raise
"Error occurred"’, has begun. Raising a string will now trigger
*note PendingDeprecationWarning: 8c7.
* Using ‘None’ as a variable name will now result in a *note
SyntaxWarning: 461. warning. In a future version of Python, ‘None’
may finally become a keyword.
* The ‘xreadlines()’ method of file objects, introduced in Python
2.1, is no longer necessary because files now behave as their own
iterator. ‘xreadlines()’ was originally introduced as a faster way
to loop over all the lines in a file, but now you can simply write
‘for line in file_obj’. File objects also have a new read-only
‘encoding’ attribute that gives the encoding used by the file;
Unicode strings written to the file will be automatically converted
to bytes using the given encoding.
* The method resolution order used by new-style classes has changed,
though you’ll only notice the difference if you have a really
complicated inheritance hierarchy. Classic classes are unaffected
by this change. Python 2.2 originally used a topological sort of a
class’s ancestors, but 2.3 now uses the C3 algorithm as described
in the paper "A Monotonic Superclass Linearization for Dylan"(1).
To understand the motivation for this change, read Michele
Simionato’s article *note The Python 2.3 Method Resolution Order:
1473, or read the thread on python-dev starting with the message at
‘https://mail.python.org/pipermail/python-dev/2002-October/029035.html’.
Samuele Pedroni first pointed out the problem and also implemented
the fix by coding the C3 algorithm.
* Python runs multithreaded programs by switching between threads
after executing N bytecodes. The default value for N has been
increased from 10 to 100 bytecodes, speeding up single-threaded
applications by reducing the switching overhead. Some
multithreaded applications may suffer slower response time, but
that’s easily fixed by setting the limit back to a lower number
using ‘sys.setcheckinterval(N)’. The limit can be retrieved with
the new ‘sys.getcheckinterval()’ function.
* One minor but far-reaching change is that the names of extension
types defined by the modules included with Python now contain the
module and a ‘'.'’ in front of the type name. For example, in
Python 2.2, if you created a socket and printed its ‘__class__’,
you’d get this output:
>>> s = socket.socket()
>>> s.__class__
In 2.3, you get this:
>>> s.__class__
* One of the noted incompatibilities between old- and new-style
classes has been removed: you can now assign to the *note __name__:
1474. and *note __bases__: 1475. attributes of new-style classes.
There are some restrictions on what can be assigned to ‘__bases__’
along the lines of those relating to assigning to an instance’s
*note __class__: 1476. attribute.
* Menu:
* String Changes::
* Optimizations: Optimizations<17>.
---------- Footnotes ----------
(1) https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.19.3910
File: python.info, Node: String Changes, Next: Optimizations<17>, Up: Other Language Changes<18>
1.19.16.1 String Changes
........................
* The *note in: 2ee. operator now works differently for strings.
Previously, when evaluating ‘X in Y’ where 'X' and 'Y' are strings,
'X' could only be a single character. That’s now changed; 'X' can
be a string of any length, and ‘X in Y’ will return *note True:
c0d. if 'X' is a substring of 'Y'. If 'X' is the empty string, the
result is always *note True: c0d.
>>> 'ab' in 'abcd'
True
>>> 'ad' in 'abcd'
False
>>> '' in 'abcd'
True
Note that this doesn’t tell you where the substring starts; if you
need that information, use the *note find(): ea1. string method.
* The *note strip(): 1478, *note lstrip(): 1479, and *note rstrip():
147a. string methods now have an optional argument for specifying
the characters to strip. The default is still to remove all
whitespace characters:
>>> ' abc '.strip()
'abc'
>>> '><><><>'.strip('<>')
'abc'
>>> '><><><>\n'.strip('<>')
'abc<><><>\n'
>>> u'\u4000\u4001abc\u4000'.strip(u'\u4000')
u'\u4001abc'
>>>
(Suggested by Simon Brunning and implemented by Walter Dörwald.)
* The *note startswith(): ec4. and *note endswith(): ec5. string
methods now accept negative numbers for the 'start' and 'end'
parameters.
* Another new string method is *note zfill(): 147b, originally a
function in the *note string: d3. module. *note zfill(): 147b.
pads a numeric string with zeros on the left until it’s the
specified width. Note that the ‘%’ operator is still more flexible
and powerful than *note zfill(): 147b.
>>> '45'.zfill(4)
'0045'
>>> '12345'.zfill(4)
'12345'
>>> 'goofy'.zfill(6)
'0goofy'
(Contributed by Walter Dörwald.)
* A new type object, ‘basestring’, has been added. Both 8-bit
strings and Unicode strings inherit from this type, so
‘isinstance(obj, basestring)’ will return *note True: c0d. for
either kind of string. It’s a completely abstract type, so you
can’t create ‘basestring’ instances.
* Interned strings are no longer immortal and will now be
garbage-collected in the usual way when the only reference to them
is from the internal dictionary of interned strings. (Implemented
by Oren Tirosh.)
File: python.info, Node: Optimizations<17>, Prev: String Changes, Up: Other Language Changes<18>
1.19.16.2 Optimizations
.......................
* The creation of new-style class instances has been made much
faster; they’re now faster than classic classes!
* The *note sort(): bcf. method of list objects has been extensively
rewritten by Tim Peters, and the implementation is significantly
faster.
* Multiplication of large long integers is now much faster thanks to
an implementation of Karatsuba multiplication, an algorithm that
scales better than the 'O'('n'^2) required for the grade-school
multiplication algorithm. (Original patch by Christopher A. Craig,
and significantly reworked by Tim Peters.)
* The ‘SET_LINENO’ opcode is now gone. This may provide a small
speed increase, depending on your compiler’s idiosyncrasies. See
section *note Other Changes and Fixes: 147d. for a longer
explanation. (Removed by Michael Hudson.)
* ‘xrange()’ objects now have their own iterator, making ‘for i in
xrange(n)’ slightly faster than ‘for i in range(n)’. (Patch by
Raymond Hettinger.)
* A number of small rearrangements have been made in various hotspots
to improve performance, such as inlining a function or removing
some code. (Implemented mostly by GvR, but lots of people have
contributed single changes.)
The net result of the 2.3 optimizations is that Python 2.3 runs the
pystone benchmark around 25% faster than Python 2.2.
File: python.info, Node: New Improved and Deprecated Modules<4>, Next: Pymalloc A Specialized Object Allocator, Prev: Other Language Changes<18>, Up: What’s New in Python 2 3
1.19.17 New, Improved, and Deprecated Modules
---------------------------------------------
As usual, Python’s standard library received a number of enhancements
and bug fixes. Here’s a partial list of the most notable changes,
sorted alphabetically by module name. Consult the ‘Misc/NEWS’ file in
the source tree for a more complete list of changes, or look through the
CVS logs for all the details.
* The *note array: 7. module now supports arrays of Unicode
characters using the ‘'u'’ format character. Arrays also now
support using the ‘+=’ assignment operator to add another array’s
contents, and the ‘*=’ assignment operator to repeat an array.
(Contributed by Jason Orendorff.)
* The ‘bsddb’ module has been replaced by version 4.1.6 of the
PyBSDDB(1) package, providing a more complete interface to the
transactional features of the BerkeleyDB library.
The old version of the module has been renamed to ‘bsddb185’ and is
no longer built automatically; you’ll have to edit ‘Modules/Setup’
to enable it. Note that the new ‘bsddb’ package is intended to be
compatible with the old module, so be sure to file bugs if you
discover any incompatibilities. When upgrading to Python 2.3, if
the new interpreter is compiled with a new version of the
underlying BerkeleyDB library, you will almost certainly have to
convert your database files to the new version. You can do this
fairly easily with the new scripts ‘db2pickle.py’ and
‘pickle2db.py’ which you will find in the distribution’s
‘Tools/scripts’ directory. If you’ve already been using the
PyBSDDB package and importing it as ‘bsddb3’, you will have to
change your ‘import’ statements to import it as ‘bsddb’.
* The new *note bz2: 13. module is an interface to the bz2 data
compression library. bz2-compressed data is usually smaller than
corresponding *note zlib: 133.-compressed data. (Contributed by
Gustavo Niemeyer.)
* A set of standard date/time types has been added in the new *note
datetime: 30. module. See the following section for more details.
* The Distutils ‘Extension’ class now supports an extra constructor
argument named 'depends' for listing additional source files that
an extension depends on. This lets Distutils recompile the module
if any of the dependency files are modified. For example, if
‘sampmodule.c’ includes the header file ‘sample.h’, you would
create the ‘Extension’ object like this:
ext = Extension("samp",
sources=["sampmodule.c"],
depends=["sample.h"])
Modifying ‘sample.h’ would then cause the module to be recompiled.
(Contributed by Jeremy Hylton.)
* Other minor changes to Distutils: it now checks for the *note CC:
147f, *note CFLAGS: 1480, ‘CPP’, *note LDFLAGS: 1481, and *note
CPPFLAGS: 1482. environment variables, using them to override the
settings in Python’s configuration (contributed by Robert Weber).
* Previously the *note doctest: 3a. module would only search the
docstrings of public methods and functions for test cases, but it
now also examines private ones as well. The *note DocTestSuite():
df2. function creates a *note unittest.TestSuite: df3. object from
a set of *note doctest: 3a. tests.
* The new ‘gc.get_referents(object)’ function returns a list of all
the objects referenced by 'object'.
* The *note getopt: 61. module gained a new function, *note
gnu_getopt(): 1483, that supports the same arguments as the
existing *note getopt(): 1484. function but uses GNU-style scanning
mode. The existing *note getopt(): 1484. stops processing options
as soon as a non-option argument is encountered, but in GNU-style
mode processing continues, meaning that options and arguments can
be mixed. For example:
>>> getopt.getopt(['-f', 'filename', 'output', '-v'], 'f:v')
([('-f', 'filename')], ['output', '-v'])
>>> getopt.gnu_getopt(['-f', 'filename', 'output', '-v'], 'f:v')
([('-f', 'filename'), ('-v', '')], ['output'])
(Contributed by Peter Åstrand.)
* The *note grp: 66, *note pwd: b2, and *note resource: bc. modules
now return enhanced tuples:
>>> import grp
>>> g = grp.getgrnam('amk')
>>> g.gr_name, g.gr_gid
('amk', 500)
* The *note gzip: 67. module can now handle files exceeding 2 GiB.
* The new *note heapq: 69. module contains an implementation of a
heap queue algorithm. A heap is an array-like data structure that
keeps items in a partially sorted order such that, for every index
'k', ‘heap[k] <= heap[2*k+1]’ and ‘heap[k] <= heap[2*k+2]’. This
makes it quick to remove the smallest item, and inserting a new
item while maintaining the heap property is 'O'(log 'n'). (See
‘https://xlinux.nist.gov/dads//HTML/priorityque.html’ for more
information about the priority queue data structure.)
The *note heapq: 69. module provides *note heappush(): 1485. and
*note heappop(): 1486. functions for adding and removing items
while maintaining the heap property on top of some other mutable
Python sequence type. Here’s an example that uses a Python list:
>>> import heapq
>>> heap = []
>>> for item in [3, 7, 5, 11, 1]:
... heapq.heappush(heap, item)
...
>>> heap
[1, 3, 5, 11, 7]
>>> heapq.heappop(heap)
1
>>> heapq.heappop(heap)
3
>>> heap
[5, 7, 11]
(Contributed by Kevin O’Connor.)
* The IDLE integrated development environment has been updated using
the code from the IDLEfork project
(‘https://idlefork.sourceforge.net’). The most notable feature is
that the code being developed is now executed in a subprocess,
meaning that there’s no longer any need for manual ‘reload()’
operations. IDLE’s core code has been incorporated into the
standard library as the *note idlelib: 73. package.
* The *note imaplib: 74. module now supports IMAP over SSL.
(Contributed by Piers Lauder and Tino Lange.)
* The *note itertools: 81. contains a number of useful functions for
use with iterators, inspired by various functions provided by the
ML and Haskell languages. For example,
‘itertools.ifilter(predicate, iterator)’ returns all elements in
the iterator for which the function ‘predicate()’ returns *note
True: c0d, and ‘itertools.repeat(obj, N)’ returns ‘obj’ 'N' times.
There are a number of other functions in the module; see the
package’s reference documentation for details. (Contributed by
Raymond Hettinger.)
* Two new functions in the *note math: 8e. module, ‘degrees(rads)’
and ‘radians(degs)’, convert between radians and degrees. Other
functions in the *note math: 8e. module such as *note math.sin():
1487. and *note math.cos(): 1488. have always required input values
measured in radians. Also, an optional 'base' argument was added
to *note math.log(): 1489. to make it easier to compute logarithms
for bases other than ‘e’ and ‘10’. (Contributed by Raymond
Hettinger.)
* Several new POSIX functions (‘getpgid()’, ‘killpg()’, ‘lchown()’,
‘loadavg()’, ‘major()’, ‘makedev()’, ‘minor()’, and ‘mknod()’) were
added to the *note posix: ad. module that underlies the *note os:
a1. module. (Contributed by Gustavo Niemeyer, Geert Jansen, and
Denis S. Otkidach.)
* In the *note os: a1. module, the ‘*stat()’ family of functions can
now report fractions of a second in a timestamp. Such time stamps
are represented as floats, similar to the value returned by *note
time.time(): 256.
During testing, it was found that some applications will break if
time stamps are floats. For compatibility, when using the tuple
interface of the *note stat_result: 148a. time stamps will be
represented as integers. When using named fields (a feature first
introduced in Python 2.2), time stamps are still represented as
integers, unless ‘os.stat_float_times()’ is invoked to enable float
return values:
>>> os.stat("/tmp").st_mtime
1034791200
>>> os.stat_float_times(True)
>>> os.stat("/tmp").st_mtime
1034791200.6335014
In Python 2.4, the default will change to always returning floats.
Application developers should enable this feature only if all their
libraries work properly when confronted with floating-point time
stamps, or if they use the tuple API. If used, the feature should
be activated on an application level instead of trying to enable it
on a per-use basis.
* The *note optparse: a0. module contains a new parser for
command-line arguments that can convert option values to a
particular Python type and will automatically generate a usage
message. See the following section for more details.
* The old and never-documented ‘linuxaudiodev’ module has been
deprecated, and a new version named ‘ossaudiodev’ has been added.
The module was renamed because the OSS sound drivers can be used on
platforms other than Linux, and the interface has also been tidied
and brought up to date in various ways. (Contributed by Greg Ward
and Nicholas FitzRoy-Dale.)
* The new *note platform: aa. module contains a number of functions
that try to determine various properties of the platform you’re
running on. There are functions for getting the architecture, CPU
type, the Windows OS version, and even the Linux distribution
version. (Contributed by Marc-André Lemburg.)
* The parser objects provided by the *note pyexpat: 126. module can
now optionally buffer character data, resulting in fewer calls to
your character data handler and therefore faster performance.
Setting the parser object’s *note buffer_text: 148b. attribute to
*note True: c0d. will enable buffering.
* The ‘sample(population, k)’ function was added to the *note random:
b8. module. 'population' is a sequence or ‘xrange’ object
containing the elements of a population, and *note sample(): 741.
chooses 'k' elements from the population without replacing chosen
elements. 'k' can be any value up to ‘len(population)’. For
example:
>>> days = ['Mo', 'Tu', 'We', 'Th', 'Fr', 'St', 'Sn']
>>> random.sample(days, 3) # Choose 3 elements
['St', 'Sn', 'Th']
>>> random.sample(days, 7) # Choose 7 elements
['Tu', 'Th', 'Mo', 'We', 'St', 'Fr', 'Sn']
>>> random.sample(days, 7) # Choose 7 again
['We', 'Mo', 'Sn', 'Fr', 'Tu', 'St', 'Th']
>>> random.sample(days, 8) # Can't choose eight
Traceback (most recent call last):
File "", line 1, in ?
File "random.py", line 414, in sample
raise ValueError, "sample larger than population"
ValueError: sample larger than population
>>> random.sample(xrange(1,10000,2), 10) # Choose ten odd nos. under 10000
[3407, 3805, 1505, 7023, 2401, 2267, 9733, 3151, 8083, 9195]
The *note random: b8. module now uses a new algorithm, the Mersenne
Twister, implemented in C. It’s faster and more extensively studied
than the previous algorithm.
(All changes contributed by Raymond Hettinger.)
* The *note readline: ba. module also gained a number of new
functions: *note get_history_item(): 148c, *note
get_current_history_length(): 148d, and *note redisplay(): 148e.
* The ‘rexec’ and ‘Bastion’ modules have been declared dead, and
attempts to import them will fail with a *note RuntimeError: 195.
New-style classes provide new ways to break out of the restricted
execution environment provided by ‘rexec’, and no one has interest
in fixing them or time to do so. If you have applications using
‘rexec’, rewrite them to use something else.
(Sticking with Python 2.2 or 2.1 will not make your applications
any safer because there are known bugs in the ‘rexec’ module in
those versions. To repeat: if you’re using ‘rexec’, stop using it
immediately.)
* The ‘rotor’ module has been deprecated because the algorithm it
uses for encryption is not believed to be secure. If you need
encryption, use one of the several AES Python modules that are
available separately.
* The *note shutil: c5. module gained a ‘move(src, dest)’ function
that recursively moves a file or directory to a new location.
* Support for more advanced POSIX signal handling was added to the
*note signal: c6. but then removed again as it proved impossible to
make it work reliably across platforms.
* The *note socket: cc. module now supports timeouts. You can call
the ‘settimeout(t)’ method on a socket object to set a timeout of
't' seconds. Subsequent socket operations that take longer than
't' seconds to complete will abort and raise a *note
socket.timeout: 830. exception.
The original timeout implementation was by Tim O’Malley. Michael
Gilfix integrated it into the Python *note socket: cc. module and
shepherded it through a lengthy review. After the code was checked
in, Guido van Rossum rewrote parts of it. (This is a good example
of a collaborative development process in action.)
* On Windows, the *note socket: cc. module now ships with Secure
Sockets Layer (SSL) support.
* The value of the C ‘PYTHON_API_VERSION’ macro is now exposed at the
Python level as ‘sys.api_version’. The current exception can be
cleared by calling the new ‘sys.exc_clear()’ function.
* The new *note tarfile: de. module allows reading from and writing
to ‘tar’-format archive files. (Contributed by Lars Gustäbel.)
* The new *note textwrap: ec. module contains functions for wrapping
strings containing paragraphs of text. The ‘wrap(text, width)’
function takes a string and returns a list containing the text
split into lines of no more than the chosen width. The ‘fill(text,
width)’ function returns a single string, reformatted to fit into
lines no longer than the chosen width. (As you can guess, *note
fill(): 148f. is built on top of *note wrap(): 1490. For example:
>>> import textwrap
>>> paragraph = "Not a whit, we defy augury: ... more text ..."
>>> textwrap.wrap(paragraph, 60)
["Not a whit, we defy augury: there's a special providence in",
"the fall of a sparrow. If it be now, 'tis not to come; if it",
...]
>>> print textwrap.fill(paragraph, 35)
Not a whit, we defy augury: there's
a special providence in the fall of
a sparrow. If it be now, 'tis not
to come; if it be not to come, it
will be now; if it be not now, yet
it will come: the readiness is all.
>>>
The module also contains a *note TextWrapper: fc8. class that
actually implements the text wrapping strategy. Both the *note
TextWrapper: fc8. class and the *note wrap(): 1490. and *note
fill(): 148f. functions support a number of additional keyword
arguments for fine-tuning the formatting; consult the module’s
documentation for details. (Contributed by Greg Ward.)
* The ‘thread’ and *note threading: ed. modules now have companion
modules, ‘dummy_thread’ and ‘dummy_threading’, that provide a
do-nothing implementation of the ‘thread’ module’s interface for
platforms where threads are not supported. The intention is to
simplify thread-aware modules (ones that 'don’t' rely on threads to
run) by putting the following code at the top:
try:
import threading as _threading
except ImportError:
import dummy_threading as _threading
In this example, ‘_threading’ is used as the module name to make it
clear that the module being used is not necessarily the actual
*note threading: ed. module. Code can call functions and use
classes in ‘_threading’ whether or not threads are supported,
avoiding an *note if: 2ed. statement and making the code slightly
clearer. This module will not magically make multithreaded code
run without threads; code that waits for another thread to return
or to do something will simply hang forever.
* The *note time: ee. module’s *note strptime(): 13f8. function has
long been an annoyance because it uses the platform C library’s
*note strptime(): 13f8. implementation, and different platforms
sometimes have odd bugs. Brett Cannon contributed a portable
implementation that’s written in pure Python and should behave
identically on all platforms.
* The new *note timeit: ef. module helps measure how long snippets of
Python code take to execute. The ‘timeit.py’ file can be run
directly from the command line, or the module’s *note Timer: 1491.
class can be imported and used directly. Here’s a short example
that figures out whether it’s faster to convert an 8-bit string to
Unicode by appending an empty Unicode string to it or by using the
‘unicode()’ function:
import timeit
timer1 = timeit.Timer('unicode("abc")')
timer2 = timeit.Timer('"abc" + u""')
# Run three trials
print timer1.repeat(repeat=3, number=100000)
print timer2.repeat(repeat=3, number=100000)
# On my laptop this outputs:
# [0.36831796169281006, 0.37441694736480713, 0.35304892063140869]
# [0.17574405670166016, 0.18193507194519043, 0.17565798759460449]
* The ‘Tix’ module has received various bug fixes and updates for the
current version of the Tix package.
* The ‘Tkinter’ module now works with a thread-enabled version of
Tcl. Tcl’s threading model requires that widgets only be accessed
from the thread in which they’re created; accesses from another
thread can cause Tcl to panic. For certain Tcl interfaces,
‘Tkinter’ will now automatically avoid this when a widget is
accessed from a different thread by marshalling a command, passing
it to the correct thread, and waiting for the results. Other
interfaces can’t be handled automatically but ‘Tkinter’ will now
raise an exception on such an access so that you can at least find
out about the problem. See
‘https://mail.python.org/pipermail/python-dev/2002-December/031107.html’
for a more detailed explanation of this change. (Implemented by
Martin von Löwis.)
* Calling Tcl methods through ‘_tkinter’ no longer returns only
strings. Instead, if Tcl returns other objects those objects are
converted to their Python equivalent, if one exists, or wrapped
with a ‘_tkinter.Tcl_Obj’ object if no Python equivalent exists.
This behavior can be controlled through the ‘wantobjects()’ method
of ‘tkapp’ objects.
When using ‘_tkinter’ through the ‘Tkinter’ module (as most Tkinter
applications will), this feature is always activated. It should
not cause compatibility problems, since Tkinter would always
convert string results to Python types where possible.
If any incompatibilities are found, the old behavior can be
restored by setting the ‘wantobjects’ variable in the ‘Tkinter’
module to false before creating the first ‘tkapp’ object.
import Tkinter
Tkinter.wantobjects = 0
Any breakage caused by this change should be reported as a bug.
* The ‘UserDict’ module has a new ‘DictMixin’ class which defines all
dictionary methods for classes that already have a minimum mapping
interface. This greatly simplifies writing classes that need to be
substitutable for dictionaries, such as the classes in the *note
shelve: c3. module.
Adding the mix-in as a superclass provides the full dictionary
interface whenever the class defines *note __getitem__(): 285,
*note __setitem__(): 12be, *note __delitem__(): 12bf, and ‘keys()’.
For example:
>>> import UserDict
>>> class SeqDict(UserDict.DictMixin):
... """Dictionary lookalike implemented with lists."""
... def __init__(self):
... self.keylist = []
... self.valuelist = []
... def __getitem__(self, key):
... try:
... i = self.keylist.index(key)
... except ValueError:
... raise KeyError
... return self.valuelist[i]
... def __setitem__(self, key, value):
... try:
... i = self.keylist.index(key)
... self.valuelist[i] = value
... except ValueError:
... self.keylist.append(key)
... self.valuelist.append(value)
... def __delitem__(self, key):
... try:
... i = self.keylist.index(key)
... except ValueError:
... raise KeyError
... self.keylist.pop(i)
... self.valuelist.pop(i)
... def keys(self):
... return list(self.keylist)
...
>>> s = SeqDict()
>>> dir(s) # See that other dictionary methods are implemented
['__cmp__', '__contains__', '__delitem__', '__doc__', '__getitem__',
'__init__', '__iter__', '__len__', '__module__', '__repr__',
'__setitem__', 'clear', 'get', 'has_key', 'items', 'iteritems',
'iterkeys', 'itervalues', 'keylist', 'keys', 'pop', 'popitem',
'setdefault', 'update', 'valuelist', 'values']
(Contributed by Raymond Hettinger.)
* The DOM implementation in *note xml.dom.minidom: 122. can now
generate XML output in a particular encoding by providing an
optional encoding argument to the *note toxml(): 1492. and *note
toprettyxml(): 1493. methods of DOM nodes.
* The ‘xmlrpclib’ module now supports an XML-RPC extension for
handling nil data values such as Python’s ‘None’. Nil values are
always supported on unmarshalling an XML-RPC response. To generate
requests containing ‘None’, you must supply a true value for the
'allow_none' parameter when creating a ‘Marshaller’ instance.
* The new ‘DocXMLRPCServer’ module allows writing self-documenting
XML-RPC servers. Run it in demo mode (as a program) to see it in
action. Pointing the web browser to the RPC server produces
pydoc-style documentation; pointing xmlrpclib to the server allows
invoking the actual methods. (Contributed by Brian Quinlan.)
* Support for internationalized domain names (RFCs 3454, 3490, 3491,
and 3492) has been added. The “idna” encoding can be used to
convert between a Unicode domain name and the ASCII-compatible
encoding (ACE) of that name.
>{}>{}> u"www.Alliancefrançaise.nu".encode("idna")
'www.xn--alliancefranaise-npb.nu'
The *note socket: cc. module has also been extended to
transparently convert Unicode hostnames to the ACE version before
passing them to the C library. Modules that deal with hostnames
such as ‘httplib’ and *note ftplib: 5e.) also support Unicode host
names; ‘httplib’ also sends HTTP ‘Host’ headers using the ACE
version of the domain name. *note urllib: 108. supports Unicode
URLs with non-ASCII host names as long as the ‘path’ part of the
URL is ASCII only.
To implement this change, the *note stringprep: d4. module, the
‘mkstringprep’ tool and the ‘punycode’ encoding have been added.
* Menu:
* Date/Time Type::
* The optparse Module::
---------- Footnotes ----------
(1) https://pybsddb.sourceforge.net
File: python.info, Node: Date/Time Type, Next: The optparse Module, Up: New Improved and Deprecated Modules<4>
1.19.17.1 Date/Time Type
........................
Date and time types suitable for expressing timestamps were added as the
*note datetime: 30. module. The types don’t support different calendars
or many fancy features, and just stick to the basics of representing
time.
The three primary types are: *note date: 1cd, representing a day, month,
and year; *note time: 1ce, consisting of hour, minute, and second; and
*note datetime: 1cc, which contains all the attributes of both *note
date: 1cd. and *note time: 1ce. There’s also a *note timedelta: 9cf.
class representing differences between two points in time, and time zone
logic is implemented by classes inheriting from the abstract *note
tzinfo: 5da. class.
You can create instances of *note date: 1cd. and *note time: 1ce. by
either supplying keyword arguments to the appropriate constructor, e.g.
‘datetime.date(year=1972, month=10, day=15)’, or by using one of a
number of class methods. For example, the *note today(): 1495. class
method returns the current local date.
Once created, instances of the date/time classes are all immutable.
There are a number of methods for producing formatted strings from
objects:
>>> import datetime
>>> now = datetime.datetime.now()
>>> now.isoformat()
'2002-12-30T21:27:03.994956'
>>> now.ctime() # Only available on date, datetime
'Mon Dec 30 21:27:03 2002'
>>> now.strftime('%Y %d %b')
'2002 30 Dec'
The *note replace(): 1496. method allows modifying one or more fields of
a *note date: 1cd. or *note datetime: 1cc. instance, returning a new
instance:
>>> d = datetime.datetime.now()
>>> d
datetime.datetime(2002, 12, 30, 22, 15, 38, 827738)
>>> d.replace(year=2001, hour = 12)
datetime.datetime(2001, 12, 30, 12, 15, 38, 827738)
>>>
Instances can be compared, hashed, and converted to strings (the result
is the same as that of *note isoformat(): b30.). *note date: 1cd. and
*note datetime: 1cc. instances can be subtracted from each other, and
added to *note timedelta: 9cf. instances. The largest missing feature
is that there’s no standard library support for parsing strings and
getting back a *note date: 1cd. or *note datetime: 1cc.
For more information, refer to the module’s reference documentation.
(Contributed by Tim Peters.)
File: python.info, Node: The optparse Module, Prev: Date/Time Type, Up: New Improved and Deprecated Modules<4>
1.19.17.2 The optparse Module
.............................
The *note getopt: 61. module provides simple parsing of command-line
arguments. The new *note optparse: a0. module (originally named Optik)
provides more elaborate command-line parsing that follows the Unix
conventions, automatically creates the output for ‘--help’, and can
perform different actions for different options.
You start by creating an instance of *note OptionParser: 1498. and
telling it what your program’s options are.
import sys
from optparse import OptionParser
op = OptionParser()
op.add_option('-i', '--input',
action='store', type='string', dest='input',
help='set input filename')
op.add_option('-l', '--length',
action='store', type='int', dest='length',
help='set maximum length of output')
Parsing a command line is then done by calling the *note parse_args():
1499. method.
options, args = op.parse_args(sys.argv[1:])
print options
print args
This returns an object containing all of the option values, and a list
of strings containing the remaining arguments.
Invoking the script with the various arguments now works as you’d expect
it to. Note that the length argument is automatically converted to an
integer.
$ ./python opt.py -i data arg1
['arg1']
$ ./python opt.py --input=data --length=4
[]
$
The help message is automatically generated for you:
$ ./python opt.py --help
usage: opt.py [options]
options:
-h, --help show this help message and exit
-iINPUT, --input=INPUT
set input filename
-lLENGTH, --length=LENGTH
set maximum length of output
$
See the module’s documentation for more details.
Optik was written by Greg Ward, with suggestions from the readers of the
Getopt SIG.
File: python.info, Node: Pymalloc A Specialized Object Allocator, Next: Build and C API Changes<12>, Prev: New Improved and Deprecated Modules<4>, Up: What’s New in Python 2 3
1.19.18 Pymalloc: A Specialized Object Allocator
------------------------------------------------
Pymalloc, a specialized object allocator written by Vladimir Marangozov,
was a feature added to Python 2.1. Pymalloc is intended to be faster
than the system ‘malloc()’ and to have less memory overhead for
allocation patterns typical of Python programs. The allocator uses C’s
‘malloc()’ function to get large pools of memory and then fulfills
smaller memory requests from these pools.
In 2.1 and 2.2, pymalloc was an experimental feature and wasn’t enabled
by default; you had to explicitly enable it when compiling Python by
providing the ‘--with-pymalloc’ option to the ‘configure’ script. In
2.3, pymalloc has had further enhancements and is now enabled by
default; you’ll have to supply ‘--without-pymalloc’ to disable it.
This change is transparent to code written in Python; however, pymalloc
may expose bugs in C extensions. Authors of C extension modules should
test their code with pymalloc enabled, because some incorrect code may
cause core dumps at runtime.
There’s one particularly common error that causes problems. There are a
number of memory allocation functions in Python’s C API that have
previously just been aliases for the C library’s ‘malloc()’ and
‘free()’, meaning that if you accidentally called mismatched functions
the error wouldn’t be noticeable. When the object allocator is enabled,
these functions aren’t aliases of ‘malloc()’ and ‘free()’ any more, and
calling the wrong function to free memory may get you a core dump. For
example, if memory was allocated using *note PyObject_Malloc(): c68, it
has to be freed using *note PyObject_Free(): c65, not ‘free()’. A few
modules included with Python fell afoul of this and had to be fixed;
doubtless there are more third-party modules that will have the same
problem.
As part of this change, the confusing multiple interfaces for allocating
memory have been consolidated down into two API families. Memory
allocated with one family must not be manipulated with functions from
the other family. There is one family for allocating chunks of memory
and another family of functions specifically for allocating Python
objects.
* To allocate and free an undistinguished chunk of memory use the
“raw memory” family: *note PyMem_Malloc(): c66, *note
PyMem_Realloc(): 102b, and *note PyMem_Free(): 140e.
* The “object memory” family is the interface to the pymalloc
facility described above and is biased towards a large number of
“small” allocations: *note PyObject_Malloc(): c68, *note
PyObject_Realloc(): 140f, and *note PyObject_Free(): c65.
* To allocate and free Python objects, use the “object” family *note
PyObject_New: 985, *note PyObject_NewVar: 986, and *note
PyObject_Del(): 149c.
Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides
debugging features to catch memory overwrites and doubled frees in both
extension modules and in the interpreter itself. To enable this
support, compile a debugging version of the Python interpreter by
running ‘configure’ with ‘--with-pydebug’.
To aid extension writers, a header file ‘Misc/pymemcompat.h’ is
distributed with the source to Python 2.3 that allows Python extensions
to use the 2.3 interfaces to memory allocation while compiling against
any version of Python since 1.5.2. You would copy the file from
Python’s source distribution and bundle it with the source of your
extension.
See also
........
‘https://hg.python.org/cpython/file/default/Objects/obmalloc.c’
For the full details of the pymalloc implementation, see the
comments at the top of the file ‘Objects/obmalloc.c’ in the Python
source code. The above link points to the file within the
python.org SVN browser.
File: python.info, Node: Build and C API Changes<12>, Next: Other Changes and Fixes<2>, Prev: Pymalloc A Specialized Object Allocator, Up: What’s New in Python 2 3
1.19.19 Build and C API Changes
-------------------------------
Changes to Python’s build process and to the C API include:
* The cycle detection implementation used by the garbage collection
has proven to be stable, so it’s now been made mandatory. You can
no longer compile Python without it, and the ‘--with-cycle-gc’
switch to ‘configure’ has been removed.
* Python can now optionally be built as a shared library
(‘libpython2.3.so’) by supplying ‘--enable-shared’ when running
Python’s ‘configure’ script. (Contributed by Ondrej Palkovsky.)
* The ‘DL_EXPORT’ and ‘DL_IMPORT’ macros are now deprecated.
Initialization functions for Python extension modules should now be
declared using the new macro *note PyMODINIT_FUNC: 149e, while the
Python core will generally use the ‘PyAPI_FUNC’ and ‘PyAPI_DATA’
macros.
* The interpreter can be compiled without any docstrings for the
built-in functions and modules by supplying ‘--without-doc-strings’
to the ‘configure’ script. This makes the Python executable about
10% smaller, but will also mean that you can’t get help for
Python’s built-ins. (Contributed by Gustavo Niemeyer.)
* The ‘PyArg_NoArgs()’ macro is now deprecated, and code that uses it
should be changed. For Python 2.2 and later, the method definition
table can specify the *note METH_NOARGS: 149f. flag, signalling
that there are no arguments, and the argument checking can then be
removed. If compatibility with pre-2.2 versions of Python is
important, the code could use ‘PyArg_ParseTuple(args, "")’ instead,
but this will be slower than using *note METH_NOARGS: 149f.
* *note PyArg_ParseTuple(): 56e. accepts new format characters for
various sizes of unsigned integers: ‘B’ for unsigned char, ‘H’ for
unsigned short int, ‘I’ for unsigned int, and ‘K’ for unsigned long
long.
* A new function, ‘PyObject_DelItemString(mapping, char *key)’ was
added as shorthand for ‘PyObject_DelItem(mapping,
PyString_New(key))’.
* File objects now manage their internal string buffer differently,
increasing it exponentially when needed. This results in the
benchmark tests in ‘Lib/test/test_bufio.py’ speeding up
considerably (from 57 seconds to 1.7 seconds, according to one
measurement).
* It’s now possible to define class and static methods for a C
extension type by setting either the *note METH_CLASS: 14a0. or
*note METH_STATIC: 14a1. flags in a method’s *note PyMethodDef:
14a2. structure.
* Python now includes a copy of the Expat XML parser’s source code,
removing any dependence on a system version or local installation
of Expat.
* If you dynamically allocate type objects in your extension, you
should be aware of a change in the rules relating to the *note
__module__: 36f. and *note __name__: 1474. attributes. In summary,
you will want to ensure the type’s dictionary contains a
‘'__module__'’ key; making the module name the part of the type
name leading up to the final period will no longer have the desired
effect. For more detail, read the API reference documentation or
the source.
* Menu:
* Port-Specific Changes: Port-Specific Changes<3>.
File: python.info, Node: Port-Specific Changes<3>, Up: Build and C API Changes<12>
1.19.19.1 Port-Specific Changes
...............................
Support for a port to IBM’s OS/2 using the EMX runtime environment was
merged into the main Python source tree. EMX is a POSIX emulation layer
over the OS/2 system APIs. The Python port for EMX tries to support all
the POSIX-like capability exposed by the EMX runtime, and mostly
succeeds; ‘fork()’ and *note fcntl(): 59. are restricted by the
limitations of the underlying emulation layer. The standard OS/2 port,
which uses IBM’s Visual Age compiler, also gained support for
case-sensitive import semantics as part of the integration of the EMX
port into CVS. (Contributed by Andrew MacIntyre.)
On MacOS, most toolbox modules have been weaklinked to improve backward
compatibility. This means that modules will no longer fail to load if a
single routine is missing on the current OS version. Instead calling
the missing routine will raise an exception. (Contributed by Jack
Jansen.)
The RPM spec files, found in the ‘Misc/RPM/’ directory in the Python
source distribution, were updated for 2.3. (Contributed by Sean
Reifschneider.)
Other new platforms now supported by Python include AtheOS
(‘http://www.atheos.cx/’), GNU/Hurd, and OpenVMS.
File: python.info, Node: Other Changes and Fixes<2>, Next: Porting to Python 2 3, Prev: Build and C API Changes<12>, Up: What’s New in Python 2 3
1.19.20 Other Changes and Fixes
-------------------------------
As usual, there were a bunch of other improvements and bugfixes
scattered throughout the source tree. A search through the CVS change
logs finds there were 523 patches applied and 514 bugs fixed between
Python 2.2 and 2.3. Both figures are likely to be underestimates.
Some of the more notable changes are:
* If the *note PYTHONINSPECT: 14a5. environment variable is set, the
Python interpreter will enter the interactive prompt after running
a Python program, as if Python had been invoked with the *note -i:
14a6. option. The environment variable can be set before running
the Python interpreter, or it can be set by the Python program as
part of its execution.
* The ‘regrtest.py’ script now provides a way to allow “all resources
except 'foo'.” A resource name passed to the ‘-u’ option can now be
prefixed with a hyphen (‘'-'’) to mean “remove this resource.” For
example, the option ‘‘-uall,-bsddb’’ could be used to enable the
use of all resources except ‘bsddb’.
* The tools used to build the documentation now work under Cygwin as
well as Unix.
* The ‘SET_LINENO’ opcode has been removed. Back in the mists of
time, this opcode was needed to produce line numbers in tracebacks
and support trace functions (for, e.g., *note pdb: a5.). Since
Python 1.5, the line numbers in tracebacks have been computed using
a different mechanism that works with “python -O”. For Python 2.3
Michael Hudson implemented a similar scheme to determine when to
call the trace function, removing the need for ‘SET_LINENO’
entirely.
It would be difficult to detect any resulting difference from
Python code, apart from a slight speed up when Python is run
without *note -O: db4.
C extensions that access the *note f_lineno: 7a7. field of frame
objects should instead call ‘PyCode_Addr2Line(f->f_code,
f->f_lasti)’. This will have the added effect of making the code
work as desired under “python -O” in earlier versions of Python.
A nifty new feature is that trace functions can now assign to the
*note f_lineno: 7a7. attribute of frame objects, changing the line
that will be executed next. A ‘jump’ command has been added to the
*note pdb: a5. debugger taking advantage of this new feature.
(Implemented by Richie Hindle.)
File: python.info, Node: Porting to Python 2 3, Next: Acknowledgements<5>, Prev: Other Changes and Fixes<2>, Up: What’s New in Python 2 3
1.19.21 Porting to Python 2.3
-----------------------------
This section lists previously described changes that may require changes
to your code:
* *note yield: 9cd. is now always a keyword; if it’s used as a
variable name in your code, a different name must be chosen.
* For strings 'X' and 'Y', ‘X in Y’ now works if 'X' is more than one
character long.
* The *note int(): 259. type constructor will now return a long
integer instead of raising an *note OverflowError: 87f. when a
string or floating-point number is too large to fit into an
integer.
* If you have Unicode strings that contain 8-bit characters, you must
declare the file’s encoding (UTF-8, Latin-1, or whatever) by adding
a comment to the top of the file. See section *note PEP 263;
Source Code Encodings: 1451. for more information.
* Calling Tcl methods through ‘_tkinter’ no longer returns only
strings. Instead, if Tcl returns other objects those objects are
converted to their Python equivalent, if one exists, or wrapped
with a ‘_tkinter.Tcl_Obj’ object if no Python equivalent exists.
* Large octal and hex literals such as ‘0xffffffff’ now trigger a
*note FutureWarning: 411. Currently they’re stored as 32-bit
numbers and result in a negative value, but in Python 2.4 they’ll
become positive long integers.
There are a few ways to fix this warning. If you really need a
positive number, just add an ‘L’ to the end of the literal. If
you’re trying to get a 32-bit integer with low bits set and have
previously used an expression such as ‘~(1 << 31)’, it’s probably
clearest to start with all bits set and clear the desired upper
bits. For example, to clear just the top bit (bit 31), you could
write ‘0xffffffffL &~(1L<<31)’.
* You can no longer disable assertions by assigning to ‘__debug__’.
* The Distutils ‘setup()’ function has gained various new keyword
arguments such as 'depends'. Old versions of the Distutils will
abort if passed unknown keywords. A solution is to check for the
presence of the new ‘get_distutil_options()’ function in your
‘setup.py’ and only uses the new keywords with a version of the
Distutils that supports them:
from distutils import core
kw = {'sources': 'foo.c', ...}
if hasattr(core, 'get_distutil_options'):
kw['depends'] = ['foo.h']
ext = Extension(**kw)
* Using ‘None’ as a variable name will now result in a *note
SyntaxWarning: 461. warning.
* Names of extension types defined by the modules included with
Python now contain the module and a ‘'.'’ in front of the type
name.
File: python.info, Node: Acknowledgements<5>, Prev: Porting to Python 2 3, Up: What’s New in Python 2 3
1.19.22 Acknowledgements
------------------------
The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Jeff Bauer, Simon Brunning, Brett Cannon, Michael Chermside,
Andrew Dalke, Scott David Daniels, Fred L. Drake, Jr., David Fraser,
Kelly Gerber, Raymond Hettinger, Michael Hudson, Chris Lambert, Detlef
Lannert, Martin von Löwis, Andrew MacIntyre, Lalo Martins, Chad Netzer,
Gustavo Niemeyer, Neal Norwitz, Hans Nowak, Chris Reedy, Francesco
Ricciardi, Vinay Sajip, Neil Schemenauer, Roman Suzi, Jason Tishler,
Just van Rossum.
File: python.info, Node: What’s New in Python 2 2, Next: What’s New in Python 2 1, Prev: What’s New in Python 2 3, Up: What’s New in Python
1.20 What’s New in Python 2.2
=============================
Author: A.M. Kuchling
* Menu:
* Introduction::
* PEPs 252 and 253; Type and Class Changes: PEPs 252 and 253 Type and Class Changes.
* PEP 234; Iterators: PEP 234 Iterators.
* PEP 255; Simple Generators: PEP 255 Simple Generators<2>.
* PEP 237; Unifying Long Integers and Integers: PEP 237 Unifying Long Integers and Integers<2>.
* PEP 238; Changing the Division Operator: PEP 238 Changing the Division Operator.
* Unicode Changes::
* PEP 227; Nested Scopes: PEP 227 Nested Scopes.
* New and Improved Modules: New and Improved Modules<3>.
* Interpreter Changes and Fixes::
* Other Changes and Fixes: Other Changes and Fixes<3>.
* Acknowledgements: Acknowledgements<6>.
File: python.info, Node: Introduction, Next: PEPs 252 and 253 Type and Class Changes, Up: What’s New in Python 2 2
1.20.1 Introduction
-------------------
This article explains the new features in Python 2.2.2, released on
October 14, 2002. Python 2.2.2 is a bugfix release of Python 2.2,
originally released on December 21, 2001.
Python 2.2 can be thought of as the “cleanup release”. There are some
features such as generators and iterators that are completely new, but
most of the changes, significant and far-reaching though they may be,
are aimed at cleaning up irregularities and dark corners of the language
design.
This article doesn’t attempt to provide a complete specification of the
new features, but instead provides a convenient overview. For full
details, you should refer to the documentation for Python 2.2, such as
the Python Library Reference(1) and the Python Reference Manual(2). If
you want to understand the complete implementation and design rationale
for a change, refer to the PEP for a particular new feature.
---------- Footnotes ----------
(1) https://docs.python.org/2.2/lib/lib.html
(2) https://docs.python.org/2.2/ref/ref.html
File: python.info, Node: PEPs 252 and 253 Type and Class Changes, Next: PEP 234 Iterators, Prev: Introduction, Up: What’s New in Python 2 2
1.20.2 PEPs 252 and 253: Type and Class Changes
-----------------------------------------------
The largest and most far-reaching changes in Python 2.2 are to Python’s
model of objects and classes. The changes should be backward
compatible, so it’s likely that your code will continue to run
unchanged, but the changes provide some amazing new capabilities.
Before beginning this, the longest and most complicated section of this
article, I’ll provide an overview of the changes and offer some
comments.
A long time ago I wrote a web page listing flaws in Python’s design.
One of the most significant flaws was that it’s impossible to subclass
Python types implemented in C. In particular, it’s not possible to
subclass built-in types, so you can’t just subclass, say, lists in order
to add a single useful method to them. The ‘UserList’ module provides a
class that supports all of the methods of lists and that can be
subclassed further, but there’s lots of C code that expects a regular
Python list and won’t accept a *note UserList: 14ae. instance.
Python 2.2 fixes this, and in the process adds some exciting new
capabilities. A brief summary:
* You can subclass built-in types such as lists and even integers,
and your subclasses should work in every place that requires the
original type.
* It’s now possible to define static and class methods, in addition
to the instance methods available in previous versions of Python.
* It’s also possible to automatically call methods on accessing or
setting an instance attribute by using a new mechanism called
'properties'. Many uses of *note __getattr__(): 4cf. can be
rewritten to use properties instead, making the resulting code
simpler and faster. As a small side benefit, attributes can now
have docstrings, too.
* The list of legal attributes for an instance can be limited to a
particular set using 'slots', making it possible to safeguard
against typos and perhaps make more optimizations possible in
future versions of Python.
Some users have voiced concern about all these changes. Sure, they say,
the new features are neat and lend themselves to all sorts of tricks
that weren’t possible in previous versions of Python, but they also make
the language more complicated. Some people have said that they’ve
always recommended Python for its simplicity, and feel that its
simplicity is being lost.
Personally, I think there’s no need to worry. Many of the new features
are quite esoteric, and you can write a lot of Python code without ever
needed to be aware of them. Writing a simple class is no more difficult
than it ever was, so you don’t need to bother learning or teaching them
unless they’re actually needed. Some very complicated tasks that were
previously only possible from C will now be possible in pure Python, and
to my mind that’s all for the better.
I’m not going to attempt to cover every single corner case and small
change that were required to make the new features work. Instead this
section will paint only the broad strokes. See section *note Related
Links: 14af, “Related Links”, for further sources of information about
Python 2.2’s new object model.
* Menu:
* Old and New Classes::
* Descriptors::
* Multiple Inheritance; The Diamond Rule: Multiple Inheritance The Diamond Rule.
* Attribute Access::
* Related Links::
File: python.info, Node: Old and New Classes, Next: Descriptors, Up: PEPs 252 and 253 Type and Class Changes
1.20.2.1 Old and New Classes
............................
First, you should know that Python 2.2 really has two kinds of classes:
classic or old-style classes, and new-style classes. The old-style
class model is exactly the same as the class model in earlier versions
of Python. All the new features described in this section apply only to
new-style classes. This divergence isn’t intended to last forever;
eventually old-style classes will be dropped, possibly in Python 3.0.
So how do you define a new-style class? You do it by subclassing an
existing new-style class. Most of Python’s built-in types, such as
integers, lists, dictionaries, and even files, are new-style classes
now. A new-style class named *note object: a8c, the base class for all
built-in types, has also been added so if no built-in type is suitable,
you can just subclass *note object: a8c.:
class C(object):
def __init__ (self):
...
...
This means that *note class: 12ca. statements that don’t have any base
classes are always classic classes in Python 2.2. (Actually you can
also change this by setting a module-level variable named
‘__metaclass__’ — see PEP 253(1) for the details — but it’s easier to
just subclass *note object: a8c.)
The type objects for the built-in types are available as built-ins,
named using a clever trick. Python has always had built-in functions
named *note int(): 259, *note float(): 2f1, and *note str(): 447. In
2.2, they aren’t functions any more, but type objects that behave as
factories when called.
>>> int
>>> int('123')
123
To make the set of types complete, new type objects such as *note
dict(): 258. and ‘file()’ have been added. Here’s a more interesting
example, adding a ‘lock()’ method to file objects:
class LockableFile(file):
def lock (self, operation, length=0, start=0, whence=0):
import fcntl
return fcntl.lockf(self.fileno(), operation,
length, start, whence)
The now-obsolete ‘posixfile’ module contained a class that emulated all
of a file object’s methods and also added a ‘lock()’ method, but this
class couldn’t be passed to internal functions that expected a built-in
file, something which is possible with our new ‘LockableFile’.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0253/
File: python.info, Node: Descriptors, Next: Multiple Inheritance The Diamond Rule, Prev: Old and New Classes, Up: PEPs 252 and 253 Type and Class Changes
1.20.2.2 Descriptors
....................
In previous versions of Python, there was no consistent way to discover
what attributes and methods were supported by an object. There were
some informal conventions, such as defining ‘__members__’ and
‘__methods__’ attributes that were lists of names, but often the author
of an extension type or a class wouldn’t bother to define them. You
could fall back on inspecting the *note __dict__: 558. of an object, but
when class inheritance or an arbitrary ‘__getattr__()’ hook were in use
this could still be inaccurate.
The one big idea underlying the new class model is that an API for
describing the attributes of an object using 'descriptors' has been
formalized. Descriptors specify the value of an attribute, stating
whether it’s a method or a field. With the descriptor API, static
methods and class methods become possible, as well as more exotic
constructs.
Attribute descriptors are objects that live inside class objects, and
have a few attributes of their own:
* *note __name__: 105f. is the attribute’s name.
* *note __doc__: 927. is the attribute’s docstring.
* ‘__get__(object)’ is a method that retrieves the attribute value
from 'object'.
* ‘__set__(object, value)’ sets the attribute on 'object' to 'value'.
* ‘__delete__(object, value)’ deletes the 'value' attribute of
'object'.
For example, when you write ‘obj.x’, the steps that Python actually
performs are:
descriptor = obj.__class__.x
descriptor.__get__(obj)
For methods, *note descriptor.__get__: 14b2. returns a temporary object
that’s callable, and wraps up the instance and the method to be called
on it. This is also why static methods and class methods are now
possible; they have descriptors that wrap up just the method, or the
method and the class. As a brief explanation of these new kinds of
methods, static methods aren’t passed the instance, and therefore
resemble regular functions. Class methods are passed the class of the
object, but not the object itself. Static and class methods are defined
like this:
class C(object):
def f(arg1, arg2):
...
f = staticmethod(f)
def g(cls, arg1, arg2):
...
g = classmethod(g)
The *note staticmethod(): 412. function takes the function ‘f()’, and
returns it wrapped up in a descriptor so it can be stored in the class
object. You might expect there to be special syntax for creating such
methods (‘def static f’, ‘defstatic f()’, or something like that) but no
such syntax has been defined yet; that’s been left for future versions
of Python.
More new features, such as slots and properties, are also implemented as
new kinds of descriptors, and it’s not difficult to write a descriptor
class that does something novel. For example, it would be possible to
write a descriptor class that made it possible to write Eiffel-style
preconditions and postconditions for a method. A class that used this
feature might be defined like this:
from eiffel import eiffelmethod
class C(object):
def f(self, arg1, arg2):
# The actual function
...
def pre_f(self):
# Check preconditions
...
def post_f(self):
# Check postconditions
...
f = eiffelmethod(f, pre_f, post_f)
Note that a person using the new ‘eiffelmethod()’ doesn’t have to
understand anything about descriptors. This is why I think the new
features don’t increase the basic complexity of the language. There
will be a few wizards who need to know about it in order to write
‘eiffelmethod()’ or the ZODB or whatever, but most users will just write
code on top of the resulting libraries and ignore the implementation
details.
File: python.info, Node: Multiple Inheritance The Diamond Rule, Next: Attribute Access, Prev: Descriptors, Up: PEPs 252 and 253 Type and Class Changes
1.20.2.3 Multiple Inheritance: The Diamond Rule
...............................................
Multiple inheritance has also been made more useful through changing the
rules under which names are resolved. Consider this set of classes
(diagram taken from PEP 253(1) by Guido van Rossum):
class A:
^ ^ def save(self): ...
/ \
/ \
/ \
/ \
class B class C:
^ ^ def save(self): ...
\ /
\ /
\ /
\ /
class D
The lookup rule for classic classes is simple but not very smart; the
base classes are searched depth-first, going from left to right. A
reference to ‘D.save()’ will search the classes ‘D’, ‘B’, and then ‘A’,
where ‘save()’ would be found and returned. ‘C.save()’ would never be
found at all. This is bad, because if ‘C’’s ‘save()’ method is saving
some internal state specific to ‘C’, not calling it will result in that
state never getting saved.
New-style classes follow a different algorithm that’s a bit more
complicated to explain, but does the right thing in this situation.
(Note that Python 2.3 changes this algorithm to one that produces the
same results in most cases, but produces more useful results for really
complicated inheritance graphs.)
1. List all the base classes, following the classic lookup rule and
include a class multiple times if it’s visited repeatedly. In the
above example, the list of visited classes is [‘D’, ‘B’, ‘A’, ‘C’,
‘A’].
2. Scan the list for duplicated classes. If any are found, remove all
but one occurrence, leaving the 'last' one in the list. In the
above example, the list becomes [‘D’, ‘B’, ‘C’, ‘A’] after dropping
duplicates.
Following this rule, referring to ‘D.save()’ will return ‘C.save()’,
which is the behaviour we’re after. This lookup rule is the same as the
one followed by Common Lisp. A new built-in function, *note super():
4d7, provides a way to get at a class’s superclasses without having to
reimplement Python’s algorithm. The most commonly used form will be
‘super(class, obj)’, which returns a bound superclass object (not the
actual class object). This form will be used in methods to call a
method in the superclass; for example, ‘D’’s ‘save()’ method would look
like this:
class D (B,C):
def save (self):
# Call superclass .save()
super(D, self).save()
# Save D's private information here
...
*note super(): 4d7. can also return unbound superclass objects when
called as ‘super(class)’ or ‘super(class1, class2)’, but this probably
won’t often be useful.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0253/
File: python.info, Node: Attribute Access, Next: Related Links, Prev: Multiple Inheritance The Diamond Rule, Up: PEPs 252 and 253 Type and Class Changes
1.20.2.4 Attribute Access
.........................
A fair number of sophisticated Python classes define hooks for attribute
access using *note __getattr__(): 4cf.; most commonly this is done for
convenience, to make code more readable by automatically mapping an
attribute access such as ‘obj.parent’ into a method call such as
‘obj.get_parent’. Python 2.2 adds some new ways of controlling
attribute access.
First, ‘__getattr__(attr_name)’ is still supported by new-style classes,
and nothing about it has changed. As before, it will be called when an
attempt is made to access ‘obj.foo’ and no attribute named ‘foo’ is
found in the instance’s dictionary.
New-style classes also support a new method,
‘__getattribute__(attr_name)’. The difference between the two methods
is that *note __getattribute__(): bd2. is 'always' called whenever any
attribute is accessed, while the old *note __getattr__(): 4cf. is only
called if ‘foo’ isn’t found in the instance’s dictionary.
However, Python 2.2’s support for 'properties' will often be a simpler
way to trap attribute references. Writing a ‘__getattr__()’ method is
complicated because to avoid recursion you can’t use regular attribute
accesses inside them, and instead have to mess around with the contents
of *note __dict__: 558. *note __getattr__(): 4cf. methods also end up
being called by Python when it checks for other methods such as *note
__repr__(): 618. or ‘__coerce__()’, and so have to be written with this
in mind. Finally, calling a function on every attribute access results
in a sizable performance loss.
*note property: 194. is a new built-in type that packages up three
functions that get, set, or delete an attribute, and a docstring. For
example, if you want to define a ‘size’ attribute that’s computed, but
also settable, you could write:
class C(object):
def get_size (self):
result = ... computation ...
return result
def set_size (self, size):
... compute something based on the size
and set internal state appropriately ...
# Define a property. The 'delete this attribute'
# method is defined as None, so the attribute
# can't be deleted.
size = property(get_size, set_size,
None,
"Storage size of this instance")
That is certainly clearer and easier to write than a pair of
‘__getattr__()’/‘__setattr__()’ methods that check for the ‘size’
attribute and handle it specially while retrieving all other attributes
from the instance’s *note __dict__: 558. Accesses to ‘size’ are also
the only ones which have to perform the work of calling a function, so
references to other attributes run at their usual speed.
Finally, it’s possible to constrain the list of attributes that can be
referenced on an object using the new *note __slots__: 14b5. class
attribute. Python objects are usually very dynamic; at any time it’s
possible to define a new attribute on an instance by just doing
‘obj.new_attr=1’. A new-style class can define a class attribute named
*note __slots__: 14b5. to limit the legal attributes to a particular set
of names. An example will make this clear:
>>> class C(object):
... __slots__ = ('template', 'name')
...
>>> obj = C()
>>> print obj.template
None
>>> obj.template = 'Test'
>>> print obj.template
Test
>>> obj.newattr = None
Traceback (most recent call last):
File "", line 1, in ?
AttributeError: 'C' object has no attribute 'newattr'
Note how you get an *note AttributeError: 348. on the attempt to assign
to an attribute not listed in *note __slots__: 14b5.
File: python.info, Node: Related Links, Prev: Attribute Access, Up: PEPs 252 and 253 Type and Class Changes
1.20.2.5 Related Links
......................
This section has just been a quick overview of the new features, giving
enough of an explanation to start you programming, but many details have
been simplified or ignored. Where should you go to get a more complete
picture?
The *note Descriptor Guide: 14b7. is a lengthy tutorial introduction to
the descriptor features, written by Guido van Rossum. If my description
has whetted your appetite, go read this tutorial next, because it goes
into much more detail about the new features while still remaining quite
easy to read.
Next, there are two relevant PEPs, PEP 252(1) and PEP 253(2). PEP
252(3) is titled “Making Types Look More Like Classes”, and covers the
descriptor API. PEP 253(4) is titled “Subtyping Built-in Types”, and
describes the changes to type objects that make it possible to subtype
built-in objects. PEP 253(5) is the more complicated PEP of the two,
and at a few points the necessary explanations of types and meta-types
may cause your head to explode. Both PEPs were written and implemented
by Guido van Rossum, with substantial assistance from the rest of the
Zope Corp. team.
Finally, there’s the ultimate authority: the source code. Most of the
machinery for the type handling is in ‘Objects/typeobject.c’, but you
should only resort to it after all other avenues have been exhausted,
including posting a question to python-list or python-dev.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0252/
(2) https://peps.python.org/pep-0253/
(3) https://peps.python.org/pep-0252/
(4) https://peps.python.org/pep-0253/
(5) https://peps.python.org/pep-0253/
File: python.info, Node: PEP 234 Iterators, Next: PEP 255 Simple Generators<2>, Prev: PEPs 252 and 253 Type and Class Changes, Up: What’s New in Python 2 2
1.20.3 PEP 234: Iterators
-------------------------
Another significant addition to 2.2 is an iteration interface at both
the C and Python levels. Objects can define how they can be looped over
by callers.
In Python versions up to 2.1, the usual way to make ‘for item in obj’
work is to define a *note __getitem__(): 285. method that looks
something like this:
def __getitem__(self, index):
return
*note __getitem__(): 285. is more properly used to define an indexing
operation on an object so that you can write ‘obj[5]’ to retrieve the
sixth element. It’s a bit misleading when you’re using this only to
support *note for: 2ec. loops. Consider some file-like object that
wants to be looped over; the 'index' parameter is essentially
meaningless, as the class probably assumes that a series of *note
__getitem__(): 285. calls will be made with 'index' incrementing by one
each time. In other words, the presence of the *note __getitem__():
285. method doesn’t mean that using ‘file[5]’ to randomly access the
sixth element will work, though it really should.
In Python 2.2, iteration can be implemented separately, and *note
__getitem__(): 285. methods can be limited to classes that really do
support random access. The basic idea of iterators is simple. A new
built-in function, ‘iter(obj)’ or ‘iter(C, sentinel)’, is used to get an
iterator. ‘iter(obj)’ returns an iterator for the object 'obj', while
‘iter(C, sentinel)’ returns an iterator that will invoke the callable
object 'C' until it returns 'sentinel' to signal that the iterator is
done.
Python classes can define an ‘__iter__()’ method, which should create
and return a new iterator for the object; if the object is its own
iterator, this method can just return ‘self’. In particular, iterators
will usually be their own iterators. Extension types implemented in C
can implement a *note tp_iter: 14b9. function in order to return an
iterator, and extension types that want to behave as iterators can
define a *note tp_iternext: 14ba. function.
So, after all this, what do iterators actually do? They have one
required method, *note next(): 7d3, which takes no arguments and returns
the next value. When there are no more values to be returned, calling
*note next(): 7d3. should raise the *note StopIteration: bfa. exception.
>>> L = [1,2,3]
>>> i = iter(L)
>>> print i
>>> i.next()
1
>>> i.next()
2
>>> i.next()
3
>>> i.next()
Traceback (most recent call last):
File "", line 1, in ?
StopIteration
>>>
In 2.2, Python’s *note for: 2ec. statement no longer expects a sequence;
it expects something for which *note iter(): 7d2. will return an
iterator. For backward compatibility and convenience, an iterator is
automatically constructed for sequences that don’t implement
‘__iter__()’ or a *note tp_iter: 14b9. slot, so ‘for i in [1,2,3]’ will
still work. Wherever the Python interpreter loops over a sequence, it’s
been changed to use the iterator protocol. This means you can do things
like this:
>>> L = [1,2,3]
>>> i = iter(L)
>>> a,b,c = i
>>> a,b,c
(1, 2, 3)
Iterator support has been added to some of Python’s basic types.
Calling *note iter(): 7d2. on a dictionary will return an iterator which
loops over its keys:
>>> m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
... 'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
>>> for key in m: print key, m[key]
...
Mar 3
Feb 2
Aug 8
Sep 9
May 5
Jun 6
Jul 7
Jan 1
Apr 4
Nov 11
Dec 12
Oct 10
That’s just the default behaviour. If you want to iterate over keys,
values, or key/value pairs, you can explicitly call the ‘iterkeys()’,
‘itervalues()’, or ‘iteritems()’ methods to get an appropriate iterator.
In a minor related change, the *note in: 2ee. operator now works on
dictionaries, so ‘key in dict’ is now equivalent to ‘dict.has_key(key)’.
Files also provide an iterator, which calls the *note readline(): ba.
method until there are no more lines in the file. This means you can
now read each line of a file using code like this:
for line in file:
# do something for each line
...
Note that you can only go forward in an iterator; there’s no way to get
the previous element, reset the iterator, or make a copy of it. An
iterator object could provide such additional capabilities, but the
iterator protocol only requires a *note next(): 7d3. method.
See also
........
PEP 234(1) - Iterators
Written by Ka-Ping Yee and GvR; implemented by the Python Labs
crew, mostly by GvR and Tim Peters.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0234/
File: python.info, Node: PEP 255 Simple Generators<2>, Next: PEP 237 Unifying Long Integers and Integers<2>, Prev: PEP 234 Iterators, Up: What’s New in Python 2 2
1.20.4 PEP 255: Simple Generators
---------------------------------
Generators are another new feature, one that interacts with the
introduction of iterators.
You’re doubtless familiar with how function calls work in Python or C.
When you call a function, it gets a private namespace where its local
variables are created. When the function reaches a *note return: 9ce.
statement, the local variables are destroyed and the resulting value is
returned to the caller. A later call to the same function will get a
fresh new set of local variables. But, what if the local variables
weren’t thrown away on exiting a function? What if you could later
resume the function where it left off? This is what generators provide;
they can be thought of as resumable functions.
Here’s the simplest example of a generator function:
def generate_ints(N):
for i in range(N):
yield i
A new keyword, *note yield: 9cd, was introduced for generators. Any
function containing a ‘yield’ statement is a generator function; this is
detected by Python’s bytecode compiler which compiles the function
specially as a result. Because a new keyword was introduced, generators
must be explicitly enabled in a module by including a ‘from __future__
import generators’ statement near the top of the module’s source code.
In Python 2.3 this statement will become unnecessary.
When you call a generator function, it doesn’t return a single value;
instead it returns a generator object that supports the iterator
protocol. On executing the *note yield: 9cd. statement, the generator
outputs the value of ‘i’, similar to a *note return: 9ce. statement.
The big difference between ‘yield’ and a ‘return’ statement is that on
reaching a ‘yield’ the generator’s state of execution is suspended and
local variables are preserved. On the next call to the generator’s
‘next()’ method, the function will resume executing immediately after
the ‘yield’ statement. (For complicated reasons, the ‘yield’ statement
isn’t allowed inside the ‘try’ block of a *note try: 6e4.…*note finally:
9ca. statement; read PEP 255(1) for a full explanation of the
interaction between ‘yield’ and exceptions.)
Here’s a sample usage of the ‘generate_ints()’ generator:
>>> gen = generate_ints(3)
>>> gen
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
File "", line 1, in ?
File "", line 2, in generate_ints
StopIteration
You could equally write ‘for i in generate_ints(5)’, or ‘a,b,c =
generate_ints(3)’.
Inside a generator function, the *note return: 9ce. statement can only
be used without a value, and signals the end of the procession of
values; afterwards the generator cannot return any further values.
‘return’ with a value, such as ‘return 5’, is a syntax error inside a
generator function. The end of the generator’s results can also be
indicated by raising *note StopIteration: bfa. manually, or by just
letting the flow of execution fall off the bottom of the function.
You could achieve the effect of generators manually by writing your own
class and storing all the local variables of the generator as instance
variables. For example, returning a list of integers could be done by
setting ‘self.count’ to 0, and having the *note next(): 7d3. method
increment ‘self.count’ and return it. However, for a moderately
complicated generator, writing a corresponding class would be much
messier. ‘Lib/test/test_generators.py’ contains a number of more
interesting examples. The simplest one implements an in-order traversal
of a tree using generators recursively.
# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
if t:
for x in inorder(t.left):
yield x
yield t.label
for x in inorder(t.right):
yield x
Two other examples in ‘Lib/test/test_generators.py’ produce solutions
for the N-Queens problem (placing $N$ queens on an $NxN$ chess board so
that no queen threatens another) and the Knight’s Tour (a route that
takes a knight to every square of an $NxN$ chessboard without visiting
any square twice).
The idea of generators comes from other programming languages,
especially Icon (‘https://www2.cs.arizona.edu/icon/’), where the idea of
generators is central. In Icon, every expression and function call
behaves like a generator. One example from “An Overview of the Icon
Programming Language” at
‘https://www2.cs.arizona.edu/icon/docs/ipd266.htm’ gives an idea of what
this looks like:
sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)
In Icon the ‘find()’ function returns the indexes at which the substring
“or” is found: 3, 23, 33. In the *note if: 2ed. statement, ‘i’ is first
assigned a value of 3, but 3 is less than 5, so the comparison fails,
and Icon retries it with the second value of 23. 23 is greater than 5,
so the comparison now succeeds, and the code prints the value 23 to the
screen.
Python doesn’t go nearly as far as Icon in adopting generators as a
central concept. Generators are considered a new part of the core
Python language, but learning or using them isn’t compulsory; if they
don’t solve any problems that you have, feel free to ignore them. One
novel feature of Python’s interface as compared to Icon’s is that a
generator’s state is represented as a concrete object (the iterator)
that can be passed around to other functions or stored in a data
structure.
See also
........
PEP 255(2) - Simple Generators
Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland.
Implemented mostly by Neil Schemenauer and Tim Peters, with other
fixes from the Python Labs crew.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0255/
(2) https://peps.python.org/pep-0255/
File: python.info, Node: PEP 237 Unifying Long Integers and Integers<2>, Next: PEP 238 Changing the Division Operator, Prev: PEP 255 Simple Generators<2>, Up: What’s New in Python 2 2
1.20.5 PEP 237: Unifying Long Integers and Integers
---------------------------------------------------
In recent versions, the distinction between regular integers, which are
32-bit values on most machines, and long integers, which can be of
arbitrary size, was becoming an annoyance. For example, on platforms
that support files larger than ‘2**32’ bytes, the ‘tell()’ method of
file objects has to return a long integer. However, there were various
bits of Python that expected plain integers and would raise an error if
a long integer was provided instead. For example, in Python 1.5, only
regular integers could be used as a slice index, and ‘'abc'[1L:]’ would
raise a *note TypeError: 534. exception with the message ‘slice index
must be int’.
Python 2.2 will shift values from short to long integers as required.
The ‘L’ suffix is no longer needed to indicate a long integer literal,
as now the compiler will choose the appropriate type. (Using the ‘L’
suffix will be discouraged in future 2.x versions of Python, triggering
a warning in Python 2.4, and probably dropped in Python 3.0.) Many
operations that used to raise an *note OverflowError: 87f. will now
return a long integer as their result. For example:
>>> 1234567890123
1234567890123L
>>> 2 ** 64
18446744073709551616L
In most cases, integers and long integers will now be treated
identically. You can still distinguish them with the *note type(): d48.
built-in function, but that’s rarely needed.
See also
........
PEP 237(1) - Unifying Long Integers and Integers
Written by Moshe Zadka and Guido van Rossum. Implemented mostly by
Guido van Rossum.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0237/
File: python.info, Node: PEP 238 Changing the Division Operator, Next: Unicode Changes, Prev: PEP 237 Unifying Long Integers and Integers<2>, Up: What’s New in Python 2 2
1.20.6 PEP 238: Changing the Division Operator
----------------------------------------------
The most controversial change in Python 2.2 heralds the start of an
effort to fix an old design flaw that’s been in Python from the
beginning. Currently Python’s division operator, ‘/’, behaves like C’s
division operator when presented with two integer arguments: it returns
an integer result that’s truncated down when there would be a fractional
part. For example, ‘3/2’ is 1, not 1.5, and ‘(-1)/2’ is -1, not -0.5.
This means that the results of division can vary unexpectedly depending
on the type of the two operands and because Python is dynamically typed,
it can be difficult to determine the possible types of the operands.
(The controversy is over whether this is 'really' a design flaw, and
whether it’s worth breaking existing code to fix this. It’s caused
endless discussions on python-dev, and in July 2001 erupted into a storm
of acidly sarcastic postings on ‘comp.lang.python’. I won’t argue for
either side here and will stick to describing what’s implemented in 2.2.
Read PEP 238(1) for a summary of arguments and counter-arguments.)
Because this change might break code, it’s being introduced very
gradually. Python 2.2 begins the transition, but the switch won’t be
complete until Python 3.0.
First, I’ll borrow some terminology from PEP 238(2). “True division” is
the division that most non-programmers are familiar with: 3/2 is 1.5,
1/4 is 0.25, and so forth. “Floor division” is what Python’s ‘/’
operator currently does when given integer operands; the result is the
floor of the value returned by true division. “Classic division” is the
current mixed behaviour of ‘/’; it returns the result of floor division
when the operands are integers, and returns the result of true division
when one of the operands is a floating-point number.
Here are the changes 2.2 introduces:
* A new operator, ‘//’, is the floor division operator. (Yes, we
know it looks like C++’s comment symbol.) ‘//’ 'always' performs
floor division no matter what the types of its operands are, so ‘1
// 2’ is 0 and ‘1.0 // 2.0’ is also 0.0.
‘//’ is always available in Python 2.2; you don’t need to enable it
using a ‘__future__’ statement.
* By including a ‘from __future__ import division’ in a module, the
‘/’ operator will be changed to return the result of true division,
so ‘1/2’ is 0.5. Without the ‘__future__’ statement, ‘/’ still
means classic division. The default meaning of ‘/’ will not change
until Python 3.0.
* Classes can define methods called *note __truediv__(): 14be. and
*note __floordiv__(): 14bf. to overload the two division operators.
At the C level, there are also slots in the *note PyNumberMethods:
13e9. structure so extension types can define the two operators.
* Python 2.2 supports some command-line arguments for testing whether
code will work with the changed division semantics. Running python
with ‘-Q warn’ will cause a warning to be issued whenever division
is applied to two integers. You can use this to find code that’s
affected by the change and fix it. By default, Python 2.2 will
simply perform classic division without a warning; the warning will
be turned on by default in Python 2.3.
See also
........
PEP 238(3) - Changing the Division Operator
Written by Moshe Zadka and Guido van Rossum. Implemented by Guido
van Rossum..
---------- Footnotes ----------
(1) https://peps.python.org/pep-0238/
(2) https://peps.python.org/pep-0238/
(3) https://peps.python.org/pep-0238/
File: python.info, Node: Unicode Changes, Next: PEP 227 Nested Scopes, Prev: PEP 238 Changing the Division Operator, Up: What’s New in Python 2 2
1.20.7 Unicode Changes
----------------------
Python’s Unicode support has been enhanced a bit in 2.2. Unicode
strings are usually stored as UCS-2, as 16-bit unsigned integers.
Python 2.2 can also be compiled to use UCS-4, 32-bit unsigned integers,
as its internal encoding by supplying ‘--enable-unicode=ucs4’ to the
configure script. (It’s also possible to specify ‘--disable-unicode’ to
completely disable Unicode support.)
When built to use UCS-4 (a “wide Python”), the interpreter can natively
handle Unicode characters from U+000000 to U+110000, so the range of
legal values for the ‘unichr()’ function is expanded accordingly. Using
an interpreter compiled to use UCS-2 (a “narrow Python”), values greater
than 65535 will still cause ‘unichr()’ to raise a *note ValueError: 204.
exception. This is all described in PEP 261(1), “Support for ‘wide’
Unicode characters”; consult it for further details.
Another change is simpler to explain. Since their introduction, Unicode
strings have supported an ‘encode()’ method to convert the string to a
selected encoding such as UTF-8 or Latin-1. A symmetric
‘decode([*encoding*])’ method has been added to 8-bit strings (though
not to Unicode strings) in 2.2. ‘decode()’ assumes that the string is
in the specified encoding and decodes it, returning whatever is returned
by the codec.
Using this new feature, codecs have been added for tasks not directly
related to Unicode. For example, codecs have been added for
uu-encoding, MIME’s base64 encoding, and compression with the *note
zlib: 133. module:
>>> s = """Here is a lengthy piece of redundant, overly verbose,
... and repetitive text.
... """
>>> data = s.encode('zlib')
>>> data
'x\x9c\r\xc9\xc1\r\x80 \x10\x04\xc0?Ul...'
>>> data.decode('zlib')
'Here is a lengthy piece of redundant, overly verbose,\nand repetitive text.\n'
>>> print s.encode('uu')
begin 666
M2&5R92!I=F5R8F]S92P*86YD(')E<&5T:71I=F4@=&5X="X*
end
>>> "sheesh".encode('rot-13')
'furrfu'
To convert a class instance to Unicode, a ‘__unicode__()’ method can be
defined by a class, analogous to ‘__str__()’.
‘encode()’, ‘decode()’, and ‘__unicode__()’ were implemented by
Marc-André Lemburg. The changes to support using UCS-4 internally were
implemented by Fredrik Lundh and Martin von Löwis.
See also
........
PEP 261(2) - Support for ‘wide’ Unicode characters
Written by Paul Prescod.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0261/
(2) https://peps.python.org/pep-0261/
File: python.info, Node: PEP 227 Nested Scopes, Next: New and Improved Modules<3>, Prev: Unicode Changes, Up: What’s New in Python 2 2
1.20.8 PEP 227: Nested Scopes
-----------------------------
In Python 2.1, statically nested scopes were added as an optional
feature, to be enabled by a ‘from __future__ import nested_scopes’
directive. In 2.2 nested scopes no longer need to be specially enabled,
and are now always present. The rest of this section is a copy of the
description of nested scopes from my “What’s New in Python 2.1”
document; if you read it when 2.1 came out, you can skip the rest of
this section.
The largest change introduced in Python 2.1, and made complete in 2.2,
is to Python’s scoping rules. In Python 2.0, at any given time there
are at most three namespaces used to look up variable names: local,
module-level, and the built-in namespace. This often surprised people
because it didn’t match their intuitive expectations. For example, a
nested recursive function definition doesn’t work:
def f():
...
def g(value):
...
return g(value-1) + 1
...
The function ‘g()’ will always raise a *note NameError: 43a. exception,
because the binding of the name ‘g’ isn’t in either its local namespace
or in the module-level namespace. This isn’t much of a problem in
practice (how often do you recursively define interior functions like
this?), but this also made using the *note lambda: 128f. expression
clumsier, and this was a problem in practice. In code which uses
‘lambda’ you can often find local variables being copied by passing them
as the default values of arguments.
def find(self, name):
"Return list of any entries equal to 'name'"
L = filter(lambda x, name=name: x == name,
self.list_attribute)
return L
The readability of Python code written in a strongly functional style
suffers greatly as a result.
The most significant change to Python 2.2 is that static scoping has
been added to the language to fix this problem. As a first effect, the
‘name=name’ default argument is now unnecessary in the above example.
Put simply, when a given variable name is not assigned a value within a
function (by an assignment, or the *note def: 1423, *note class: 12ca,
or *note import: 5de. statements), references to the variable will be
looked up in the local namespace of the enclosing scope. A more
detailed explanation of the rules, and a dissection of the
implementation, can be found in the PEP.
This change may cause some compatibility problems for code where the
same variable name is used both at the module level and as a local
variable within a function that contains further function definitions.
This seems rather unlikely though, since such code would have been
pretty confusing to read in the first place.
One side effect of the change is that the ‘from module import *’ and
‘exec’ statements have been made illegal inside a function scope under
certain conditions. The Python reference manual has said all along that
‘from module import *’ is only legal at the top level of a module, but
the CPython interpreter has never enforced this before. As part of the
implementation of nested scopes, the compiler which turns Python source
into bytecodes has to generate different code to access variables in a
containing scope. ‘from module import *’ and ‘exec’ make it impossible
for the compiler to figure this out, because they add names to the local
namespace that are unknowable at compile time. Therefore, if a function
contains function definitions or *note lambda: 128f. expressions with
free variables, the compiler will flag this by raising a *note
SyntaxError: 18d. exception.
To make the preceding explanation a bit clearer, here’s an example:
x = 1
def f():
# The next line is a syntax error
exec 'x=2'
def g():
return x
Line 4 containing the ‘exec’ statement is a syntax error, since ‘exec’
would define a new local variable named ‘x’ whose value should be
accessed by ‘g()’.
This shouldn’t be much of a limitation, since ‘exec’ is rarely used in
most Python code (and when it is used, it’s often a sign of a poor
design anyway).
See also
........
PEP 227(1) - Statically Nested Scopes
Written and implemented by Jeremy Hylton.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0227/
File: python.info, Node: New and Improved Modules<3>, Next: Interpreter Changes and Fixes, Prev: PEP 227 Nested Scopes, Up: What’s New in Python 2 2
1.20.9 New and Improved Modules
-------------------------------
* The *note xmlrpclib: 12e. module was contributed to the standard
library by Fredrik Lundh, providing support for writing XML-RPC
clients. XML-RPC is a simple remote procedure call protocol built
on top of HTTP and XML. For example, the following snippet
retrieves a list of RSS channels from the O’Reilly Network, and
then lists the recent headlines for one channel:
import xmlrpclib
s = xmlrpclib.Server(
'http://www.oreillynet.com/meerkat/xml-rpc/server.php')
channels = s.meerkat.getChannels()
# channels is a list of dictionaries, like this:
# [{'id': 4, 'title': 'Freshmeat Daily News'}
# {'id': 190, 'title': '32Bits Online'},
# {'id': 4549, 'title': '3DGamers'}, ... ]
# Get the items for one channel
items = s.meerkat.getItems( {'channel': 4} )
# 'items' is another list of dictionaries, like this:
# [{'link': 'http://freshmeat.net/releases/52719/',
# 'description': 'A utility which converts HTML to XSL FO.',
# 'title': 'html2fo 0.3 (Default)'}, ... ]
The *note SimpleXMLRPCServer: 12f. module makes it easy to create
straightforward XML-RPC servers. See
‘http://xmlrpc.scripting.com/’ for more information about XML-RPC.
* The new *note hmac: 6a. module implements the HMAC algorithm
described by RFC 2104(1). (Contributed by Gerhard Häring.)
* Several functions that originally returned lengthy tuples now
return pseudo-sequences that still behave like tuples but also have
mnemonic attributes such as ‘memberst_mtime’ or *note tm_year:
14c3. The enhanced functions include *note stat(): 49c, *note
fstat(): d8b, *note statvfs(): 10f0, and *note fstatvfs(): d8c. in
the *note os: a1. module, and *note localtime(): 14c4, *note
gmtime(): 11b2, and *note strptime(): 13f8. in the *note time: ee.
module.
For example, to obtain a file’s size using the old tuples, you’d
end up writing something like ‘file_size =
os.stat(filename)[stat.ST_SIZE]’, but now this can be written more
clearly as ‘file_size = os.stat(filename).st_size’.
The original patch for this feature was contributed by Nick
Mathewson.
* The Python profiler has been extensively reworked and various
errors in its output have been corrected. (Contributed by Fred L.
Drake, Jr. and Tim Peters.)
* The *note socket: cc. module can be compiled to support IPv6;
specify the ‘--enable-ipv6’ option to Python’s configure script.
(Contributed by Jun-ichiro “itojun” Hagino.)
* Two new format characters were added to the *note struct: d5.
module for 64-bit integers on platforms that support the C long
long type. ‘q’ is for a signed 64-bit integer, and ‘Q’ is for an
unsigned one. The value is returned in Python’s long integer type.
(Contributed by Tim Peters.)
* In the interpreter’s interactive mode, there’s a new built-in
function *note help(): 8d6. that uses the *note pydoc: b5. module
introduced in Python 2.1 to provide interactive help.
‘help(object)’ displays any available help text about 'object'.
*note help(): 8d6. with no argument puts you in an online help
utility, where you can enter the names of functions, classes, or
modules to read their help text. (Contributed by Guido van Rossum,
using Ka-Ping Yee’s *note pydoc: b5. module.)
* Various bugfixes and performance improvements have been made to the
SRE engine underlying the *note re: b9. module. For example, the
*note re.sub(): 2a5. and *note re.split(): 2a4. functions have been
rewritten in C. Another contributed patch speeds up certain Unicode
character ranges by a factor of two, and a new *note finditer():
14c5. method that returns an iterator over all the non-overlapping
matches in a given string. (SRE is maintained by Fredrik Lundh.
The BIGCHARSET patch was contributed by Martin von Löwis.)
* The *note smtplib: ca. module now supports RFC 2487(2), “Secure
SMTP over TLS”, so it’s now possible to encrypt the SMTP traffic
between a Python program and the mail transport agent being handed
a message. *note smtplib: ca. also supports SMTP authentication.
(Contributed by Gerhard Häring.)
* The *note imaplib: 74. module, maintained by Piers Lauder, has
support for several new extensions: the NAMESPACE extension defined
in RFC 2342(3), SORT, GETACL and SETACL. (Contributed by Anthony
Baxter and Michel Pelletier.)
* The ‘rfc822’ module’s parsing of email addresses is now compliant
with RFC 2822(4), an update to RFC 822(5). (The module’s name is
'not' going to be changed to ‘rfc2822’.) A new package, *note
email: 3b, has also been added for parsing and generating e-mail
messages. (Contributed by Barry Warsaw, and arising out of his
work on Mailman.)
* The *note difflib: 37. module now contains a new ‘Differ’ class for
producing human-readable lists of changes (a “delta”) between two
sequences of lines of text. There are also two generator
functions, ‘ndiff()’ and ‘restore()’, which respectively return a
delta from two sequences, or one of the original sequences from a
delta. (Grunt work contributed by David Goodger, from ndiff.py
code by Tim Peters who then did the generatorization.)
* New constants ‘ascii_letters’, ‘ascii_lowercase’, and
‘ascii_uppercase’ were added to the *note string: d3. module.
There were several modules in the standard library that used
‘string.letters’ to mean the ranges A-Za-z, but that assumption is
incorrect when locales are in use, because ‘string.letters’ varies
depending on the set of legal characters defined by the current
locale. The buggy modules have all been fixed to use
‘ascii_letters’ instead. (Reported by an unknown person; fixed by
Fred L. Drake, Jr.)
* The *note mimetypes: 8f. module now makes it easier to use
alternative MIME-type databases by the addition of a *note
MimeTypes: 14c6. class, which takes a list of filenames to be
parsed. (Contributed by Fred L. Drake, Jr.)
* A *note Timer: 114d. class was added to the *note threading: ed.
module that allows scheduling an activity to happen at some future
time. (Contributed by Itamar Shtull-Trauring.)
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2104.html
(2) https://datatracker.ietf.org/doc/html/rfc2487.html
(3) https://datatracker.ietf.org/doc/html/rfc2342.html
(4) https://datatracker.ietf.org/doc/html/rfc2822.html
(5) https://datatracker.ietf.org/doc/html/rfc822.html
File: python.info, Node: Interpreter Changes and Fixes, Next: Other Changes and Fixes<3>, Prev: New and Improved Modules<3>, Up: What’s New in Python 2 2
1.20.10 Interpreter Changes and Fixes
-------------------------------------
Some of the changes only affect people who deal with the Python
interpreter at the C level because they’re writing Python extension
modules, embedding the interpreter, or just hacking on the interpreter
itself. If you only write Python code, none of the changes described
here will affect you very much.
* Profiling and tracing functions can now be implemented in C, which
can operate at much higher speeds than Python-based functions and
should reduce the overhead of profiling and tracing. This will be
of interest to authors of development environments for Python. Two
new C functions were added to Python’s API, *note
PyEval_SetProfile(): 14c8. and *note PyEval_SetTrace(): 41d. The
existing *note sys.setprofile(): 14c9. and *note sys.settrace():
14ca. functions still exist, and have simply been changed to use
the new C-level interface. (Contributed by Fred L. Drake, Jr.)
* Another low-level API, primarily of interest to implementers of
Python debuggers and development tools, was added. *note
PyInterpreterState_Head(): 14cb. and *note
PyInterpreterState_Next(): 14cc. let a caller walk through all the
existing interpreter objects; *note
PyInterpreterState_ThreadHead(): 14cd. and *note
PyThreadState_Next(): 14ce. allow looping over all the thread
states for a given interpreter. (Contributed by David Beazley.)
* The C-level interface to the garbage collector has been changed to
make it easier to write extension types that support garbage
collection and to debug misuses of the functions. Various
functions have slightly different semantics, so a bunch of
functions had to be renamed. Extensions that use the old API will
still compile but will 'not' participate in garbage collection, so
updating them for 2.2 should be considered fairly high priority.
To upgrade an extension module to the new API, perform the
following steps:
* Rename ‘Py_TPFLAGS_GC’ to *note Py_TPFLAGS_HAVE_GC: 778.
*
Use *note PyObject_GC_New(): aa2. or *note PyObject_GC_NewVar(): aa3. to allocate
objects, and *note PyObject_GC_Del(): 14cf. to deallocate
them.
* Rename ‘PyObject_GC_Init()’ to *note PyObject_GC_Track(): 14d0. and
‘PyObject_GC_Fini()’ to *note PyObject_GC_UnTrack(): 14d1.
* Remove ‘PyGC_HEAD_SIZE’ from object size calculations.
* Remove calls to ‘PyObject_AS_GC()’ and ‘PyObject_FROM_GC()’.
* A new ‘et’ format sequence was added to *note PyArg_ParseTuple():
56e.; ‘et’ takes both a parameter and an encoding name, and
converts the parameter to the given encoding if the parameter turns
out to be a Unicode string, or leaves it alone if it’s an 8-bit
string, assuming it to already be in the desired encoding. This
differs from the ‘es’ format character, which assumes that 8-bit
strings are in Python’s default ASCII encoding and converts them to
the specified new encoding. (Contributed by M.-A. Lemburg, and
used for the MBCS support on Windows described in the following
section.)
* A different argument parsing function, *note PyArg_UnpackTuple():
14d2, has been added that’s simpler and presumably faster. Instead
of specifying a format string, the caller simply gives the minimum
and maximum number of arguments expected, and a set of pointers to
*note PyObject: 334.* variables that will be filled in with
argument values.
* Two new flags *note METH_NOARGS: 149f. and *note METH_O: 14d3. are
available in method definition tables to simplify implementation of
methods with no arguments or a single untyped argument. Calling
such methods is more efficient than calling a corresponding method
that uses *note METH_VARARGS: 14d4. Also, the old ‘METH_OLDARGS’
style of writing C methods is now officially deprecated.
* Two new wrapper functions, *note PyOS_snprintf(): 14d5. and *note
PyOS_vsnprintf(): 14d6. were added to provide cross-platform
implementations for the relatively new ‘snprintf()’ and
‘vsnprintf()’ C lib APIs. In contrast to the standard ‘sprintf()’
and ‘vsprintf()’ functions, the Python versions check the bounds of
the buffer used to protect against buffer overruns. (Contributed
by M.-A. Lemburg.)
* The *note _PyTuple_Resize(): 14d7. function has lost an unused
parameter, so now it takes 2 parameters instead of 3. The third
argument was never used, and can simply be discarded when porting
code from earlier versions to Python 2.2.
File: python.info, Node: Other Changes and Fixes<3>, Next: Acknowledgements<6>, Prev: Interpreter Changes and Fixes, Up: What’s New in Python 2 2
1.20.11 Other Changes and Fixes
-------------------------------
As usual there were a bunch of other improvements and bugfixes scattered
throughout the source tree. A search through the CVS change logs finds
there were 527 patches applied and 683 bugs fixed between Python 2.1 and
2.2; 2.2.1 applied 139 patches and fixed 143 bugs; 2.2.2 applied 106
patches and fixed 82 bugs. These figures are likely to be
underestimates.
Some of the more notable changes are:
* The code for the MacOS port for Python, maintained by Jack Jansen,
is now kept in the main Python CVS tree, and many changes have been
made to support MacOS X.
The most significant change is the ability to build Python as a
framework, enabled by supplying the ‘--enable-framework’ option to
the configure script when compiling Python. According to Jack
Jansen, “This installs a self-contained Python installation plus
the OS X framework “glue” into
‘/Library/Frameworks/Python.framework’ (or another location of
choice). For now there is little immediate added benefit to this
(actually, there is the disadvantage that you have to change your
PATH to be able to find Python), but it is the basis for creating a
full-blown Python application, porting the MacPython IDE, possibly
using Python as a standard OSA scripting language and much more.”
Most of the MacPython toolbox modules, which interface to MacOS
APIs such as windowing, QuickTime, scripting, etc. have been
ported to OS X, but they’ve been left commented out in ‘setup.py’.
People who want to experiment with these modules can uncomment them
manually.
* Keyword arguments passed to built-in functions that don’t take them
now cause a *note TypeError: 534. exception to be raised, with the
message “'function' takes no keyword arguments”.
* Weak references, added in Python 2.1 as an extension module, are
now part of the core because they’re used in the implementation of
new-style classes. The *note ReferenceError: 14d9. exception has
therefore moved from the *note weakref: 114. module to become a
built-in exception.
* A new script, ‘Tools/scripts/cleanfuture.py’ by Tim Peters,
automatically removes obsolete ‘__future__’ statements from Python
source code.
* An additional 'flags' argument has been added to the built-in
function *note compile(): 192, so the behaviour of ‘__future__’
statements can now be correctly observed in simulated shells, such
as those presented by IDLE and other development environments.
This is described in PEP 264(1). (Contributed by Michael Hudson.)
* The new license introduced with Python 1.6 wasn’t GPL-compatible.
This is fixed by some minor textual changes to the 2.2 license, so
it’s now legal to embed Python inside a GPLed program again. Note
that Python itself is not GPLed, but instead is under a license
that’s essentially equivalent to the BSD license, same as it always
was. The license changes were also applied to the Python 2.0.1 and
2.1.1 releases.
* When presented with a Unicode filename on Windows, Python will now
convert it to an MBCS encoded string, as used by the Microsoft file
APIs. As MBCS is explicitly used by the file APIs, Python’s choice
of ASCII as the default encoding turns out to be an annoyance. On
Unix, the locale’s character set is used if
‘locale.nl_langinfo(CODESET)’ is available. (Windows support was
contributed by Mark Hammond with assistance from Marc-André
Lemburg. Unix support was added by Martin von Löwis.)
* Large file support is now enabled on Windows. (Contributed by Tim
Peters.)
* The ‘Tools/scripts/ftpmirror.py’ script now parses a ‘.netrc’ file,
if you have one. (Contributed by Mike Romberg.)
* Some features of the object returned by the ‘xrange()’ function are
now deprecated, and trigger warnings when they’re accessed; they’ll
disappear in Python 2.3. ‘xrange’ objects tried to pretend they
were full sequence types by supporting slicing, sequence
multiplication, and the *note in: 2ee. operator, but these features
were rarely used and therefore buggy. The ‘tolist()’ method and
the ‘start’, ‘stop’, and ‘step’ attributes are also being
deprecated. At the C level, the fourth argument to the
‘PyRange_New()’ function, ‘repeat’, has also been deprecated.
* There were a bunch of patches to the dictionary implementation,
mostly to fix potential core dumps if a dictionary contains objects
that sneakily changed their hash value, or mutated the dictionary
they were contained in. For a while python-dev fell into a gentle
rhythm of Michael Hudson finding a case that dumped core, Tim
Peters fixing the bug, Michael finding another case, and round and
round it went.
* On Windows, Python can now be compiled with Borland C thanks to a
number of patches contributed by Stephen Hansen, though the result
isn’t fully functional yet. (But this 'is' progress…)
* Another Windows enhancement: Wise Solutions generously offered
PythonLabs use of their InstallerMaster 8.1 system. Earlier
PythonLabs Windows installers used Wise 5.0a, which was beginning
to show its age. (Packaged up by Tim Peters.)
* Files ending in ‘.pyw’ can now be imported on Windows. ‘.pyw’ is a
Windows-only thing, used to indicate that a script needs to be run
using PYTHONW.EXE instead of PYTHON.EXE in order to prevent a DOS
console from popping up to display the output. This patch makes it
possible to import such scripts, in case they’re also usable as
modules. (Implemented by David Bolen.)
* On platforms where Python uses the C ‘dlopen()’ function to load
extension modules, it’s now possible to set the flags used by
‘dlopen()’ using the *note sys.getdlopenflags(): 14da. and *note
sys.setdlopenflags(): 1113. functions. (Contributed by Bram
Stolk.)
* The *note pow(): 9d2. built-in function no longer supports 3
arguments when floating-point numbers are supplied. ‘pow(x, y, z)’
returns ‘(x**y) % z’, but this is never useful for floating-point
numbers, and the final result varies unpredictably depending on the
platform. A call such as ‘pow(2.0, 8.0, 7.0)’ will now raise a
*note TypeError: 534. exception.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0264/
File: python.info, Node: Acknowledgements<6>, Prev: Other Changes and Fixes<3>, Up: What’s New in Python 2 2
1.20.12 Acknowledgements
------------------------
The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Fred Bremmer, Keith Briggs, Andrew Dalke, Fred L. Drake, Jr.,
Carel Fellinger, David Goodger, Mark Hammond, Stephen Hansen, Michael
Hudson, Jack Jansen, Marc-André Lemburg, Martin von Löwis, Fredrik
Lundh, Michael McLay, Nick Mathewson, Paul Moore, Gustavo Niemeyer, Don
O’Donnell, Joonas Paalasma, Tim Peters, Jens Quade, Tom Reinhardt, Neil
Schemenauer, Guido van Rossum, Greg Ward, Edward Welbourne.
File: python.info, Node: What’s New in Python 2 1, Next: What’s New in Python 2 0, Prev: What’s New in Python 2 2, Up: What’s New in Python
1.21 What’s New in Python 2.1
=============================
Author: A.M. Kuchling
* Menu:
* Introduction: Introduction<2>.
* PEP 227; Nested Scopes: PEP 227 Nested Scopes<2>.
* PEP 236; __future__ Directives: PEP 236 __future__ Directives.
* PEP 207; Rich Comparisons: PEP 207 Rich Comparisons.
* PEP 230; Warning Framework: PEP 230 Warning Framework.
* PEP 229; New Build System: PEP 229 New Build System.
* PEP 205; Weak References: PEP 205 Weak References.
* PEP 232; Function Attributes: PEP 232 Function Attributes.
* PEP 235; Importing Modules on Case-Insensitive Platforms: PEP 235 Importing Modules on Case-Insensitive Platforms.
* PEP 217; Interactive Display Hook: PEP 217 Interactive Display Hook.
* PEP 208; New Coercion Model: PEP 208 New Coercion Model.
* PEP 241; Metadata in Python Packages: PEP 241 Metadata in Python Packages.
* New and Improved Modules: New and Improved Modules<4>.
* Other Changes and Fixes: Other Changes and Fixes<4>.
* Acknowledgements: Acknowledgements<7>.
File: python.info, Node: Introduction<2>, Next: PEP 227 Nested Scopes<2>, Up: What’s New in Python 2 1
1.21.1 Introduction
-------------------
This article explains the new features in Python 2.1. While there
aren’t as many changes in 2.1 as there were in Python 2.0, there are
still some pleasant surprises in store. 2.1 is the first release to be
steered through the use of Python Enhancement Proposals, or PEPs, so
most of the sizable changes have accompanying PEPs that provide more
complete documentation and a design rationale for the change. This
article doesn’t attempt to document the new features completely, but
simply provides an overview of the new features for Python programmers.
Refer to the Python 2.1 documentation, or to the specific PEP, for more
details about any new feature that particularly interests you.
One recent goal of the Python development team has been to accelerate
the pace of new releases, with a new release coming every 6 to 9 months.
2.1 is the first release to come out at this faster pace, with the first
alpha appearing in January, 3 months after the final version of 2.0 was
released.
The final release of Python 2.1 was made on April 17, 2001.
File: python.info, Node: PEP 227 Nested Scopes<2>, Next: PEP 236 __future__ Directives, Prev: Introduction<2>, Up: What’s New in Python 2 1
1.21.2 PEP 227: Nested Scopes
-----------------------------
The largest change in Python 2.1 is to Python’s scoping rules. In
Python 2.0, at any given time there are at most three namespaces used to
look up variable names: local, module-level, and the built-in namespace.
This often surprised people because it didn’t match their intuitive
expectations. For example, a nested recursive function definition
doesn’t work:
def f():
...
def g(value):
...
return g(value-1) + 1
...
The function ‘g()’ will always raise a *note NameError: 43a. exception,
because the binding of the name ‘g’ isn’t in either its local namespace
or in the module-level namespace. This isn’t much of a problem in
practice (how often do you recursively define interior functions like
this?), but this also made using the *note lambda: 128f. expression
clumsier, and this was a problem in practice. In code which uses *note
lambda: 128f. you can often find local variables being copied by passing
them as the default values of arguments.
def find(self, name):
"Return list of any entries equal to 'name'"
L = filter(lambda x, name=name: x == name,
self.list_attribute)
return L
The readability of Python code written in a strongly functional style
suffers greatly as a result.
The most significant change to Python 2.1 is that static scoping has
been added to the language to fix this problem. As a first effect, the
‘name=name’ default argument is now unnecessary in the above example.
Put simply, when a given variable name is not assigned a value within a
function (by an assignment, or the *note def: 1423, *note class: 12ca,
or *note import: 5de. statements), references to the variable will be
looked up in the local namespace of the enclosing scope. A more
detailed explanation of the rules, and a dissection of the
implementation, can be found in the PEP.
This change may cause some compatibility problems for code where the
same variable name is used both at the module level and as a local
variable within a function that contains further function definitions.
This seems rather unlikely though, since such code would have been
pretty confusing to read in the first place.
One side effect of the change is that the ‘from module import *’ and
‘exec’ statements have been made illegal inside a function scope under
certain conditions. The Python reference manual has said all along that
‘from module import *’ is only legal at the top level of a module, but
the CPython interpreter has never enforced this before. As part of the
implementation of nested scopes, the compiler which turns Python source
into bytecodes has to generate different code to access variables in a
containing scope. ‘from module import *’ and ‘exec’ make it impossible
for the compiler to figure this out, because they add names to the local
namespace that are unknowable at compile time. Therefore, if a function
contains function definitions or *note lambda: 128f. expressions with
free variables, the compiler will flag this by raising a *note
SyntaxError: 18d. exception.
To make the preceding explanation a bit clearer, here’s an example:
x = 1
def f():
# The next line is a syntax error
exec 'x=2'
def g():
return x
Line 4 containing the ‘exec’ statement is a syntax error, since ‘exec’
would define a new local variable named ‘x’ whose value should be
accessed by ‘g()’.
This shouldn’t be much of a limitation, since ‘exec’ is rarely used in
most Python code (and when it is used, it’s often a sign of a poor
design anyway).
Compatibility concerns have led to nested scopes being introduced
gradually; in Python 2.1, they aren’t enabled by default, but can be
turned on within a module by using a future statement as described in
PEP 236(1). (See the following section for further discussion of PEP
236(2).) In Python 2.2, nested scopes will become the default and there
will be no way to turn them off, but users will have had all of 2.1’s
lifetime to fix any breakage resulting from their introduction.
See also
........
PEP 227(3) - Statically Nested Scopes
Written and implemented by Jeremy Hylton.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0236/
(2) https://peps.python.org/pep-0236/
(3) https://peps.python.org/pep-0227/
File: python.info, Node: PEP 236 __future__ Directives, Next: PEP 207 Rich Comparisons, Prev: PEP 227 Nested Scopes<2>, Up: What’s New in Python 2 1
1.21.3 PEP 236: __future__ Directives
-------------------------------------
The reaction to nested scopes was widespread concern about the dangers
of breaking code with the 2.1 release, and it was strong enough to make
the Pythoneers take a more conservative approach. This approach
consists of introducing a convention for enabling optional functionality
in release N that will become compulsory in release N+1.
The syntax uses a ‘from...import’ statement using the reserved module
name *note __future__: 0. Nested scopes can be enabled by the following
statement:
from __future__ import nested_scopes
While it looks like a normal *note import: 5de. statement, it’s not;
there are strict rules on where such a future statement can be put.
They can only be at the top of a module, and must precede any Python
code or regular ‘import’ statements. This is because such statements
can affect how the Python bytecode compiler parses code and generates
bytecode, so they must precede any statement that will result in
bytecodes being produced.
See also
........
PEP 236(1) - Back to the *note __future__: 0.
Written by Tim Peters, and primarily implemented by Jeremy Hylton.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0236/
File: python.info, Node: PEP 207 Rich Comparisons, Next: PEP 230 Warning Framework, Prev: PEP 236 __future__ Directives, Up: What’s New in Python 2 1
1.21.4 PEP 207: Rich Comparisons
--------------------------------
In earlier versions, Python’s support for implementing comparisons on
user-defined classes and extension types was quite simple. Classes
could implement a ‘__cmp__()’ method that was given two instances of a
class, and could only return 0 if they were equal or +1 or -1 if they
weren’t; the method couldn’t raise an exception or return anything other
than a Boolean value. Users of Numeric Python often found this model
too weak and restrictive, because in the number-crunching programs that
numeric Python is used for, it would be more useful to be able to
perform elementwise comparisons of two matrices, returning a matrix
containing the results of a given comparison for each element. If the
two matrices are of different sizes, then the compare has to be able to
raise an exception to signal the error.
In Python 2.1, rich comparisons were added in order to support this
need. Python classes can now individually overload each of the ‘<’,
‘<=’, ‘>’, ‘>=’, ‘==’, and ‘!=’ operations. The new magic method names
are:
Operation Method name
---------------------------------------------
‘<’ *note __lt__(): 1292.
‘<=’ *note __le__(): 12fe.
‘>’ *note __gt__(): 12ff.
‘>=’ *note __ge__(): 1300.
‘==’ *note __eq__(): afa.
‘!=’ *note __ne__(): 14e2.
(The magic methods are named after the corresponding Fortran operators
‘.LT.’. ‘.LE.’, &c. Numeric programmers are almost certainly quite
familiar with these names and will find them easy to remember.)
Each of these magic methods is of the form ‘method(self, other)’, where
‘self’ will be the object on the left-hand side of the operator, while
‘other’ will be the object on the right-hand side. For example, the
expression ‘A < B’ will cause ‘A.__lt__(B)’ to be called.
Each of these magic methods can return anything at all: a Boolean, a
matrix, a list, or any other Python object. Alternatively they can
raise an exception if the comparison is impossible, inconsistent, or
otherwise meaningless.
The built-in ‘cmp(A,B)’ function can use the rich comparison machinery,
and now accepts an optional argument specifying which comparison
operation to use; this is given as one of the strings ‘"<"’, ‘"<="’,
‘">"’, ‘">="’, ‘"=="’, or ‘"!="’. If called without the optional third
argument, ‘cmp()’ will only return -1, 0, or +1 as in previous versions
of Python; otherwise it will call the appropriate method and can return
any Python object.
There are also corresponding changes of interest to C programmers;
there’s a new slot ‘tp_richcmp’ in type objects and an API for
performing a given rich comparison. I won’t cover the C API here, but
will refer you to PEP 207(1), or to 2.1’s C API documentation, for the
full list of related functions.
See also
........
PEP 207(2) - Rich Comparisons
Written by Guido van Rossum, heavily based on earlier work by David
Ascher, and implemented by Guido van Rossum.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0207/
(2) https://peps.python.org/pep-0207/
File: python.info, Node: PEP 230 Warning Framework, Next: PEP 229 New Build System, Prev: PEP 207 Rich Comparisons, Up: What’s New in Python 2 1
1.21.5 PEP 230: Warning Framework
---------------------------------
Over its 10 years of existence, Python has accumulated a certain number
of obsolete modules and features along the way. It’s difficult to know
when a feature is safe to remove, since there’s no way of knowing how
much code uses it — perhaps no programs depend on the feature, or
perhaps many do. To enable removing old features in a more structured
way, a warning framework was added. When the Python developers want to
get rid of a feature, it will first trigger a warning in the next
version of Python. The following Python version can then drop the
feature, and users will have had a full release cycle to remove uses of
the old feature.
Python 2.1 adds the warning framework to be used in this scheme. It
adds a *note warnings: 112. module that provide functions to issue
warnings, and to filter out warnings that you don’t want to be
displayed. Third-party modules can also use this framework to deprecate
old features that they no longer wish to support.
For example, in Python 2.1 the ‘regex’ module is deprecated, so
importing it causes a warning to be printed:
>>> import regex
__main__:1: DeprecationWarning: the regex module
is deprecated; please use the re module
>>>
Warnings can be issued by calling the *note warnings.warn(): 14e4.
function:
warnings.warn("feature X no longer supported")
The first parameter is the warning message; an additional optional
parameters can be used to specify a particular warning category.
Filters can be added to disable certain warnings; a regular expression
pattern can be applied to the message or to the module name in order to
suppress a warning. For example, you may have a program that uses the
‘regex’ module and not want to spare the time to convert it to use the
*note re: b9. module right now. The warning can be suppressed by
calling
import warnings
warnings.filterwarnings(action = 'ignore',
message='.*regex module is deprecated',
category=DeprecationWarning,
module = '__main__')
This adds a filter that will apply only to warnings of the class *note
DeprecationWarning: 1a5. triggered in the *note __main__: 1. module, and
applies a regular expression to only match the message about the ‘regex’
module being deprecated, and will cause such warnings to be ignored.
Warnings can also be printed only once, printed every time the offending
code is executed, or turned into exceptions that will cause the program
to stop (unless the exceptions are caught in the usual way, of course).
Functions were also added to Python’s C API for issuing warnings; refer
to PEP 230 or to Python’s API documentation for the details.
See also
........
PEP 5(1) - Guidelines for Language Evolution
Written by Paul Prescod, to specify procedures to be followed when
removing old features from Python. The policy described in this
PEP hasn’t been officially adopted, but the eventual policy
probably won’t be too different from Prescod’s proposal.
PEP 230(2) - Warning Framework
Written and implemented by Guido van Rossum.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0005/
(2) https://peps.python.org/pep-0230/
File: python.info, Node: PEP 229 New Build System, Next: PEP 205 Weak References, Prev: PEP 230 Warning Framework, Up: What’s New in Python 2 1
1.21.6 PEP 229: New Build System
--------------------------------
When compiling Python, the user had to go in and edit the
‘Modules/Setup’ file in order to enable various additional modules; the
default set is relatively small and limited to modules that compile on
most Unix platforms. This means that on Unix platforms with many more
features, most notably Linux, Python installations often don’t contain
all useful modules they could.
Python 2.0 added the Distutils, a set of modules for distributing and
installing extensions. In Python 2.1, the Distutils are used to compile
much of the standard library of extension modules, autodetecting which
ones are supported on the current machine. It’s hoped that this will
make Python installations easier and more featureful.
Instead of having to edit the ‘Modules/Setup’ file in order to enable
modules, a ‘setup.py’ script in the top directory of the Python source
distribution is run at build time, and attempts to discover which
modules can be enabled by examining the modules and header files on the
system. If a module is configured in ‘Modules/Setup’, the ‘setup.py’
script won’t attempt to compile that module and will defer to the
‘Modules/Setup’ file’s contents. This provides a way to specific any
strange command-line flags or libraries that are required for a specific
platform.
In another far-reaching change to the build mechanism, Neil Schemenauer
restructured things so Python now uses a single makefile that isn’t
recursive, instead of makefiles in the top directory and in each of the
‘Python/’, ‘Parser/’, ‘Objects/’, and ‘Modules/’ subdirectories. This
makes building Python faster and also makes hacking the Makefiles
clearer and simpler.
See also
........
PEP 229(1) - Using Distutils to Build Python
Written and implemented by A.M. Kuchling.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0229/
File: python.info, Node: PEP 205 Weak References, Next: PEP 232 Function Attributes, Prev: PEP 229 New Build System, Up: What’s New in Python 2 1
1.21.7 PEP 205: Weak References
-------------------------------
Weak references, available through the *note weakref: 114. module, are a
minor but useful new data type in the Python programmer’s toolbox.
Storing a reference to an object (say, in a dictionary or a list) has
the side effect of keeping that object alive forever. There are a few
specific cases where this behaviour is undesirable, object caches being
the most common one, and another being circular references in data
structures such as trees.
For example, consider a memoizing function that caches the results of
another function ‘f(x)’ by storing the function’s argument and its
result in a dictionary:
_cache = {}
def memoize(x):
if _cache.has_key(x):
return _cache[x]
retval = f(x)
# Cache the returned object
_cache[x] = retval
return retval
This version works for simple things such as integers, but it has a side
effect; the ‘_cache’ dictionary holds a reference to the return values,
so they’ll never be deallocated until the Python process exits and
cleans up. This isn’t very noticeable for integers, but if ‘f()’
returns an object, or a data structure that takes up a lot of memory,
this can be a problem.
Weak references provide a way to implement a cache that won’t keep
objects alive beyond their time. If an object is only accessible
through weak references, the object will be deallocated and the weak
references will now indicate that the object it referred to no longer
exists. A weak reference to an object 'obj' is created by calling ‘wr =
weakref.ref(obj)’. The object being referred to is returned by calling
the weak reference as if it were a function: ‘wr()’. It will return the
referenced object, or ‘None’ if the object no longer exists.
This makes it possible to write a ‘memoize()’ function whose cache
doesn’t keep objects alive, by storing weak references in the cache.
_cache = {}
def memoize(x):
if _cache.has_key(x):
obj = _cache[x]()
# If weak reference object still exists,
# return it
if obj is not None: return obj
retval = f(x)
# Cache a weak reference
_cache[x] = weakref.ref(retval)
return retval
The *note weakref: 114. module also allows creating proxy objects which
behave like weak references — an object referenced only by proxy objects
is deallocated – but instead of requiring an explicit call to retrieve
the object, the proxy transparently forwards all operations to the
object as long as the object still exists. If the object is
deallocated, attempting to use a proxy will cause a
‘weakref.ReferenceError’ exception to be raised.
proxy = weakref.proxy(obj)
proxy.attr # Equivalent to obj.attr
proxy.meth() # Equivalent to obj.meth()
del obj
proxy.attr # raises weakref.ReferenceError
See also
........
PEP 205(1) - Weak References
Written and implemented by Fred L. Drake, Jr.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0205/
File: python.info, Node: PEP 232 Function Attributes, Next: PEP 235 Importing Modules on Case-Insensitive Platforms, Prev: PEP 205 Weak References, Up: What’s New in Python 2 1
1.21.8 PEP 232: Function Attributes
-----------------------------------
In Python 2.1, functions can now have arbitrary information attached to
them. People were often using docstrings to hold information about
functions and methods, because the *note __doc__: 11cb. attribute was
the only way of attaching any information to a function. For example,
in the Zope web application server, functions are marked as safe for
public access by having a docstring, and in John Aycock’s SPARK parsing
framework, docstrings hold parts of the BNF grammar to be parsed. This
overloading is unfortunate, since docstrings are really intended to hold
a function’s documentation; for example, it means you can’t properly
document functions intended for private use in Zope.
Arbitrary attributes can now be set and retrieved on functions using the
regular Python syntax:
def f(): pass
f.publish = 1
f.secure = 1
f.grammar = "A ::= B (C D)*"
The dictionary containing attributes can be accessed as the function’s
*note __dict__: 12c5. Unlike the *note __dict__: c5d. attribute of
class instances, in functions you can actually assign a new dictionary
to *note __dict__: 12c5, though the new value is restricted to a regular
Python dictionary; you 'can’t' be tricky and set it to a ‘UserDict’
instance, or any other random object that behaves like a mapping.
See also
........
PEP 232(1) - Function Attributes
Written and implemented by Barry Warsaw.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0232/
File: python.info, Node: PEP 235 Importing Modules on Case-Insensitive Platforms, Next: PEP 217 Interactive Display Hook, Prev: PEP 232 Function Attributes, Up: What’s New in Python 2 1
1.21.9 PEP 235: Importing Modules on Case-Insensitive Platforms
---------------------------------------------------------------
Some operating systems have filesystems that are case-insensitive, MacOS
and Windows being the primary examples; on these systems, it’s
impossible to distinguish the filenames ‘FILE.PY’ and ‘file.py’, even
though they do store the file’s name in its original case (they’re
case-preserving, too).
In Python 2.1, the *note import: 5de. statement will work to simulate
case-sensitivity on case-insensitive platforms. Python will now search
for the first case-sensitive match by default, raising an *note
ImportError: 415. if no such file is found, so ‘import file’ will not
import a module named ‘FILE.PY’. Case-insensitive matching can be
requested by setting the *note PYTHONCASEOK: 95c. environment variable
before starting the Python interpreter.
File: python.info, Node: PEP 217 Interactive Display Hook, Next: PEP 208 New Coercion Model, Prev: PEP 235 Importing Modules on Case-Insensitive Platforms, Up: What’s New in Python 2 1
1.21.10 PEP 217: Interactive Display Hook
-----------------------------------------
When using the Python interpreter interactively, the output of commands
is displayed using the built-in *note repr(): 7f9. function. In Python
2.1, the variable *note sys.displayhook(): 14ea. can be set to a
callable object which will be called instead of *note repr(): 7f9. For
example, you can set it to a special pretty-printing function:
>>> # Create a recursive data structure
... L = [1,2,3]
>>> L.append(L)
>>> L # Show Python's default output
[1, 2, 3, [...]]
>>> # Use pprint.pprint() as the display function
... import sys, pprint
>>> sys.displayhook = pprint.pprint
>>> L
[1, 2, 3, ]
>>>
See also
........
PEP 217(1) - Display Hook for Interactive Use
Written and implemented by Moshe Zadka.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0217/
File: python.info, Node: PEP 208 New Coercion Model, Next: PEP 241 Metadata in Python Packages, Prev: PEP 217 Interactive Display Hook, Up: What’s New in Python 2 1
1.21.11 PEP 208: New Coercion Model
-----------------------------------
How numeric coercion is done at the C level was significantly modified.
This will only affect the authors of C extensions to Python, allowing
them more flexibility in writing extension types that support numeric
operations.
Extension types can now set the type flag ‘Py_TPFLAGS_CHECKTYPES’ in
their ‘PyTypeObject’ structure to indicate that they support the new
coercion model. In such extension types, the numeric slot functions can
no longer assume that they’ll be passed two arguments of the same type;
instead they may be passed two arguments of differing types, and can
then perform their own internal coercion. If the slot function is
passed a type it can’t handle, it can indicate the failure by returning
a reference to the ‘Py_NotImplemented’ singleton value. The numeric
functions of the other type will then be tried, and perhaps they can
handle the operation; if the other type also returns
‘Py_NotImplemented’, then a *note TypeError: 534. will be raised.
Numeric methods written in Python can also return ‘Py_NotImplemented’,
causing the interpreter to act as if the method did not exist (perhaps
raising a *note TypeError: 534, perhaps trying another object’s numeric
methods).
See also
........
PEP 208(1) - Reworking the Coercion Model
Written and implemented by Neil Schemenauer, heavily based upon
earlier work by Marc-André Lemburg. Read this to understand the
fine points of how numeric operations will now be processed at the
C level.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0208/
File: python.info, Node: PEP 241 Metadata in Python Packages, Next: New and Improved Modules<4>, Prev: PEP 208 New Coercion Model, Up: What’s New in Python 2 1
1.21.12 PEP 241: Metadata in Python Packages
--------------------------------------------
A common complaint from Python users is that there’s no single catalog
of all the Python modules in existence. T. Middleton’s Vaults of
Parnassus at ‘www.vex.net/parnassus/’ (retired in February 2009,
available in the Internet Archive Wayback Machine(1)) was the largest
catalog of Python modules, but registering software at the Vaults is
optional, and many people did not bother.
As a first small step toward fixing the problem, Python software
packaged using the Distutils ‘sdist’ command will include a file named
‘PKG-INFO’ containing information about the package such as its name,
version, and author (metadata, in cataloguing terminology). PEP 241(2)
contains the full list of fields that can be present in the ‘PKG-INFO’
file. As people began to package their software using Python 2.1, more
and more packages will include metadata, making it possible to build
automated cataloguing systems and experiment with them. With the result
experience, perhaps it’ll be possible to design a really good catalog
and then build support for it into Python 2.2. For example, the
Distutils ‘sdist’ and ‘bdist_*’ commands could support an ‘upload’
option that would automatically upload your package to a catalog server.
You can start creating packages containing ‘PKG-INFO’ even if you’re not
using Python 2.1, since a new release of the Distutils will be made for
users of earlier Python versions. Version 1.0.2 of the Distutils
includes the changes described in PEP 241(3), as well as various
bugfixes and enhancements. It will be available from the Distutils SIG
at ‘https://www.python.org/community/sigs/current/distutils-sig/’.
See also
........
PEP 241(4) - Metadata for Python Software Packages
Written and implemented by A.M. Kuchling.
PEP 243(5) - Module Repository Upload Mechanism
Written by Sean Reifschneider, this draft PEP describes a proposed
mechanism for uploading Python packages to a central server.
---------- Footnotes ----------
(1)
https://web.archive.org/web/20090130140102/http://www.vex.net/parnassus/
(2) https://peps.python.org/pep-0241/
(3) https://peps.python.org/pep-0241/
(4) https://peps.python.org/pep-0241/
(5) https://peps.python.org/pep-0243/
File: python.info, Node: New and Improved Modules<4>, Next: Other Changes and Fixes<4>, Prev: PEP 241 Metadata in Python Packages, Up: What’s New in Python 2 1
1.21.13 New and Improved Modules
--------------------------------
* Ka-Ping Yee contributed two new modules: ‘inspect.py’, a module for
getting information about live Python code, and ‘pydoc.py’, a
module for interactively converting docstrings to HTML or text. As
a bonus, ‘Tools/scripts/pydoc’, which is now automatically
installed, uses ‘pydoc.py’ to display documentation given a Python
module, package, or class name. For example, ‘pydoc xml.dom’
displays the following:
Python Library Documentation: package xml.dom in xml
NAME
xml.dom - W3C Document Object Model implementation for Python.
FILE
/usr/local/lib/python2.1/xml/dom/__init__.pyc
DESCRIPTION
The Python mapping of the Document Object Model is documented in the
Python Library Reference in the section on the xml.dom package.
This package contains the following modules:
...
‘pydoc’ also includes a Tk-based interactive help browser. ‘pydoc’
quickly becomes addictive; try it out!
* Two different modules for unit testing were added to the standard
library. The *note doctest: 3a. module, contributed by Tim Peters,
provides a testing framework based on running embedded examples in
docstrings and comparing the results against the expected output.
PyUnit, contributed by Steve Purcell, is a unit testing framework
inspired by JUnit, which was in turn an adaptation of Kent Beck’s
Smalltalk testing framework. See ‘https://pyunit.sourceforge.net/’
for more information about PyUnit.
* The *note difflib: 37. module contains a class, *note
SequenceMatcher: 1012, which compares two sequences and computes
the changes required to transform one sequence into the other. For
example, this module can be used to write a tool similar to the
Unix ‘diff’ program, and in fact the sample program
‘Tools/scripts/ndiff.py’ demonstrates how to write such a script.
* *note curses.panel: 2d, a wrapper for the panel library, part of
ncurses and of SYSV curses, was contributed by Thomas Gellekum.
The panel library provides windows with the additional feature of
depth. Windows can be moved higher or lower in the depth ordering,
and the panel library figures out where panels overlap and which
sections are visible.
* The PyXML package has gone through a few releases since Python 2.0,
and Python 2.1 includes an updated version of the *note xml: 120.
package. Some of the noteworthy changes include support for Expat
1.2 and later versions, the ability for Expat parsers to handle
files in any encoding supported by Python, and various bugfixes for
SAX, DOM, and the ‘minidom’ module.
* Ping also contributed another hook for handling uncaught
exceptions. *note sys.excepthook(): 807. can be set to a callable
object. When an exception isn’t caught by any *note try:
6e4.…*note except: 18b. blocks, the exception will be passed to
*note sys.excepthook(): 807, which can then do whatever it likes.
At the Ninth Python Conference, Ping demonstrated an application
for this hook: printing an extended traceback that not only lists
the stack frames, but also lists the function arguments and the
local variables for each frame.
* Various functions in the *note time: ee. module, such as *note
asctime(): 11d9. and *note localtime(): 14c4, require a
floating-point argument containing the time in seconds since the
epoch. The most common use of these functions is to work with the
current time, so the floating-point argument has been made
optional; when a value isn’t provided, the current time will be
used. For example, log file entries usually need a string
containing the current time; in Python 2.1, ‘time.asctime()’ can be
used, instead of the lengthier
‘time.asctime(time.localtime(time.time()))’ that was previously
required.
This change was proposed and implemented by Thomas Wouters.
* The *note ftplib: 5e. module now defaults to retrieving files in
passive mode, because passive mode is more likely to work from
behind a firewall. This request came from the Debian bug tracking
system, since other Debian packages use *note ftplib: 5e. to
retrieve files and then don’t work from behind a firewall. It’s
deemed unlikely that this will cause problems for anyone, because
Netscape defaults to passive mode and few people complain, but if
passive mode is unsuitable for your application or network setup,
call ‘set_pasv(0)’ on FTP objects to disable passive mode.
* Support for raw socket access has been added to the *note socket:
cc. module, contributed by Grant Edwards.
* The *note pstats: b0. module now contains a simple interactive
statistics browser for displaying timing profiles for Python
programs, invoked when the module is run as a script. Contributed
by Eric S. Raymond.
* A new implementation-dependent function, ‘sys._getframe([depth])’,
has been added to return a given frame object from the current call
stack. *note sys._getframe(): 6dc. returns the frame at the top of
the call stack; if the optional integer argument 'depth' is
supplied, the function returns the frame that is 'depth' calls
below the top of the stack. For example, ‘sys._getframe(1)’
returns the caller’s frame object.
This function is only present in CPython, not in Jython or the .NET
implementation. Use it for debugging, and resist the temptation to
put it into production code.
File: python.info, Node: Other Changes and Fixes<4>, Next: Acknowledgements<7>, Prev: New and Improved Modules<4>, Up: What’s New in Python 2 1
1.21.14 Other Changes and Fixes
-------------------------------
There were relatively few smaller changes made in Python 2.1 due to the
shorter release cycle. A search through the CVS change logs turns up
117 patches applied, and 136 bugs fixed; both figures are likely to be
underestimates. Some of the more notable changes are:
* A specialized object allocator is now optionally available, that
should be faster than the system ‘malloc()’ and have less memory
overhead. The allocator uses C’s ‘malloc()’ function to get large
pools of memory, and then fulfills smaller memory requests from
these pools. It can be enabled by providing the ‘--with-pymalloc’
option to the ‘configure’ script; see ‘Objects/obmalloc.c’ for the
implementation details.
Authors of C extension modules should test their code with the
object allocator enabled, because some incorrect code may break,
causing core dumps at runtime. There are a bunch of memory
allocation functions in Python’s C API that have previously been
just aliases for the C library’s ‘malloc()’ and ‘free()’, meaning
that if you accidentally called mismatched functions, the error
wouldn’t be noticeable. When the object allocator is enabled,
these functions aren’t aliases of ‘malloc()’ and ‘free()’ any more,
and calling the wrong function to free memory will get you a core
dump. For example, if memory was allocated using *note PyMem_New:
14ef, it has to be freed using *note PyMem_Del(): 14f0, not
‘free()’. A few modules included with Python fell afoul of this
and had to be fixed; doubtless there are more third-party modules
that will have the same problem.
The object allocator was contributed by Vladimir Marangozov.
* The speed of line-oriented file I/O has been improved because
people often complain about its lack of speed, and because it’s
often been used as a naïve benchmark. The *note readline(): ba.
method of file objects has therefore been rewritten to be much
faster. The exact amount of the speedup will vary from platform to
platform depending on how slow the C library’s ‘getc()’ was, but is
around 66%, and potentially much faster on some particular
operating systems. Tim Peters did much of the benchmarking and
coding for this change, motivated by a discussion in
comp.lang.python.
A new module and method for file objects was also added,
contributed by Jeff Epler. The new method, ‘xreadlines()’, is
similar to the existing ‘xrange()’ built-in. ‘xreadlines()’
returns an opaque sequence object that only supports being iterated
over, reading a line on every iteration but not reading the entire
file into memory as the existing ‘readlines()’ method does. You’d
use it like this:
for line in sys.stdin.xreadlines():
# ... do something for each line ...
...
For a fuller discussion of the line I/O changes, see the python-dev
summary for January 1–15, 2001 at
‘https://mail.python.org/pipermail/python-dev/2001-January/’.
* A new method, *note popitem(): 14f1, was added to dictionaries to
enable destructively iterating through the contents of a
dictionary; this can be faster for large dictionaries because
there’s no need to construct a list containing all the keys or
values. ‘D.popitem()’ removes a random ‘(key, value)’ pair from
the dictionary ‘D’ and returns it as a 2-tuple. This was
implemented mostly by Tim Peters and Guido van Rossum, after a
suggestion and preliminary patch by Moshe Zadka.
* Modules can now control which names are imported when ‘from module
import *’ is used, by defining an ‘__all__’ attribute containing a
list of names that will be imported. One common complaint is that
if the module imports other modules such as *note sys: d9. or *note
string: d3, ‘from module import *’ will add them to the importing
module’s namespace. To fix this, simply list the public names in
‘__all__’:
# List public names
__all__ = ['Database', 'open']
A stricter version of this patch was first suggested and
implemented by Ben Wolfson, but after some python-dev discussion, a
weaker final version was checked in.
* Applying *note repr(): 7f9. to strings previously used octal
escapes for non-printable characters; for example, a newline was
‘'\012'’. This was a vestigial trace of Python’s C ancestry, but
today octal is of very little practical use. Ka-Ping Yee suggested
using hex escapes instead of octal ones, and using the ‘\n’, ‘\t’,
‘\r’ escapes for the appropriate characters, and implemented this
new formatting.
* Syntax errors detected at compile-time can now raise exceptions
containing the filename and line number of the error, a pleasant
side effect of the compiler reorganization done by Jeremy Hylton.
* C extensions which import other modules have been changed to use
*note PyImport_ImportModule(): 3ba, which means that they will use
any import hooks that have been installed. This is also encouraged
for third-party extensions that need to import some other module
from C code.
* The size of the Unicode character database was shrunk by another
340K thanks to Fredrik Lundh.
* Some new ports were contributed: MacOS X (by Steven Majewski),
Cygwin (by Jason Tishler); RISCOS (by Dietmar Schwertberger);
Unixware 7 (by Billy G. Allie).
And there’s the usual list of minor bugfixes, minor memory leaks,
docstring edits, and other tweaks, too lengthy to be worth itemizing;
see the CVS logs for the full details if you want them.
File: python.info, Node: Acknowledgements<7>, Prev: Other Changes and Fixes<4>, Up: What’s New in Python 2 1
1.21.15 Acknowledgements
------------------------
The author would like to thank the following people for offering
suggestions on various drafts of this article: Graeme Cross, David
Goodger, Jay Graves, Michael Hudson, Marc-André Lemburg, Fredrik Lundh,
Neil Schemenauer, Thomas Wouters.
File: python.info, Node: What’s New in Python 2 0, Next: Changelog, Prev: What’s New in Python 2 1, Up: What’s New in Python
1.22 What’s New in Python 2.0
=============================
Author: A.M. Kuchling and Moshe Zadka
* Menu:
* Introduction: Introduction<3>.
* What About Python 1.6?: What About Python 1 6?.
* New Development Process::
* Unicode: Unicode<2>.
* List Comprehensions::
* Augmented Assignment::
* String Methods::
* Garbage Collection of Cycles::
* Other Core Changes::
* Porting to 2.0: Porting to 2 0.
* Extending/Embedding Changes::
* Distutils; Making Modules Easy to Install: Distutils Making Modules Easy to Install.
* XML Modules::
* Module changes::
* New modules::
* IDLE Improvements::
* Deleted and Deprecated Modules::
* Acknowledgements: Acknowledgements<8>.
File: python.info, Node: Introduction<3>, Next: What About Python 1 6?, Up: What’s New in Python 2 0
1.22.1 Introduction
-------------------
A new release of Python, version 2.0, was released on October 16, 2000.
This article covers the exciting new features in 2.0, highlights some
other useful changes, and points out a few incompatible changes that may
require rewriting code.
Python’s development never completely stops between releases, and a
steady flow of bug fixes and improvements are always being submitted. A
host of minor fixes, a few optimizations, additional docstrings, and
better error messages went into 2.0; to list them all would be
impossible, but they’re certainly significant. Consult the publicly
available CVS logs if you want to see the full list. This progress is
due to the five developers working for PythonLabs are now getting paid
to spend their days fixing bugs, and also due to the improved
communication resulting from moving to SourceForge.
File: python.info, Node: What About Python 1 6?, Next: New Development Process, Prev: Introduction<3>, Up: What’s New in Python 2 0
1.22.2 What About Python 1.6?
-----------------------------
Python 1.6 can be thought of as the Contractual Obligations Python
release. After the core development team left CNRI in May 2000, CNRI
requested that a 1.6 release be created, containing all the work on
Python that had been performed at CNRI. Python 1.6 therefore represents
the state of the CVS tree as of May 2000, with the most significant new
feature being Unicode support. Development continued after May, of
course, so the 1.6 tree received a few fixes to ensure that it’s
forward-compatible with Python 2.0. 1.6 is therefore part of Python’s
evolution, and not a side branch.
So, should you take much interest in Python 1.6? Probably not. The
1.6final and 2.0beta1 releases were made on the same day (September 5,
2000), the plan being to finalize Python 2.0 within a month or so. If
you have applications to maintain, there seems little point in breaking
things by moving to 1.6, fixing them, and then having another round of
breakage within a month by moving to 2.0; you’re better off just going
straight to 2.0. Most of the really interesting features described in
this document are only in 2.0, because a lot of work was done between
May and September.
File: python.info, Node: New Development Process, Next: Unicode<2>, Prev: What About Python 1 6?, Up: What’s New in Python 2 0
1.22.3 New Development Process
------------------------------
The most important change in Python 2.0 may not be to the code at all,
but to how Python is developed: in May 2000 the Python developers began
using the tools made available by SourceForge for storing source code,
tracking bug reports, and managing the queue of patch submissions. To
report bugs or submit patches for Python 2.0, use the bug tracking and
patch manager tools available from Python’s project page, located at
‘https://sourceforge.net/projects/python/’.
The most important of the services now hosted at SourceForge is the
Python CVS tree, the version-controlled repository containing the source
code for Python. Previously, there were roughly 7 or so people who had
write access to the CVS tree, and all patches had to be inspected and
checked in by one of the people on this short list. Obviously, this
wasn’t very scalable. By moving the CVS tree to SourceForge, it became
possible to grant write access to more people; as of September 2000
there were 27 people able to check in changes, a fourfold increase.
This makes possible large-scale changes that wouldn’t be attempted if
they’d have to be filtered through the small group of core developers.
For example, one day Peter Schneider-Kamp took it into his head to drop
K&R C compatibility and convert the C source for Python to ANSI C. After
getting approval on the python-dev mailing list, he launched into a
flurry of checkins that lasted about a week, other developers joined in
to help, and the job was done. If there were only 5 people with write
access, probably that task would have been viewed as “nice, but not
worth the time and effort needed” and it would never have gotten done.
The shift to using SourceForge’s services has resulted in a remarkable
increase in the speed of development. Patches now get submitted,
commented on, revised by people other than the original submitter, and
bounced back and forth between people until the patch is deemed worth
checking in. Bugs are tracked in one central location and can be
assigned to a specific person for fixing, and we can count the number of
open bugs to measure progress. This didn’t come without a cost:
developers now have more e-mail to deal with, more mailing lists to
follow, and special tools had to be written for the new environment.
For example, SourceForge sends default patch and bug notification e-mail
messages that are completely unhelpful, so Ka-Ping Yee wrote an HTML
screen-scraper that sends more useful messages.
The ease of adding code caused a few initial growing pains, such as code
was checked in before it was ready or without getting clear agreement
from the developer group. The approval process that has emerged is
somewhat similar to that used by the Apache group. Developers can vote
+1, +0, -0, or -1 on a patch; +1 and -1 denote acceptance or rejection,
while +0 and -0 mean the developer is mostly indifferent to the change,
though with a slight positive or negative slant. The most significant
change from the Apache model is that the voting is essentially advisory,
letting Guido van Rossum, who has Benevolent Dictator For Life status,
know what the general opinion is. He can still ignore the result of a
vote, and approve or reject a change even if the community disagrees
with him.
Producing an actual patch is the last step in adding a new feature, and
is usually easy compared to the earlier task of coming up with a good
design. Discussions of new features can often explode into lengthy
mailing list threads, making the discussion hard to follow, and no one
can read every posting to python-dev. Therefore, a relatively formal
process has been set up to write Python Enhancement Proposals (PEPs),
modelled on the internet RFC process. PEPs are draft documents that
describe a proposed new feature, and are continually revised until the
community reaches a consensus, either accepting or rejecting the
proposal. Quoting from the introduction to PEP 1(1), “PEP Purpose and
Guidelines”:
PEP stands for Python Enhancement Proposal. A PEP is a design
document providing information to the Python community, or
describing a new feature for Python. The PEP should provide a
concise technical specification of the feature and a rationale for
the feature.
We intend PEPs to be the primary mechanisms for proposing new
features, for collecting community input on an issue, and for
documenting the design decisions that have gone into Python. The
PEP author is responsible for building consensus within the
community and documenting dissenting opinions.
Read the rest of PEP 1(2) for the details of the PEP editorial process,
style, and format. PEPs are kept in the Python CVS tree on SourceForge,
though they’re not part of the Python 2.0 distribution, and are also
available in HTML form from ‘https://peps.python.org/’. As of September
2000, there are 25 PEPs, ranging from PEP 201(3), “Lockstep Iteration”,
to PEP 225, “Elementwise/Objectwise Operators”.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0001/
(2) https://peps.python.org/pep-0001/
(3) https://peps.python.org/pep-0201/
File: python.info, Node: Unicode<2>, Next: List Comprehensions, Prev: New Development Process, Up: What’s New in Python 2 0
1.22.4 Unicode
--------------
The largest new feature in Python 2.0 is a new fundamental data type:
Unicode strings. Unicode uses 16-bit numbers to represent characters
instead of the 8-bit number used by ASCII, meaning that 65,536 distinct
characters can be supported.
The final interface for Unicode support was arrived at through countless
often-stormy discussions on the python-dev mailing list, and mostly
implemented by Marc-André Lemburg, based on a Unicode string type
implementation by Fredrik Lundh. A detailed explanation of the
interface was written up as PEP 100(1), “Python Unicode Integration”.
This article will simply cover the most significant points about the
Unicode interfaces.
In Python source code, Unicode strings are written as ‘u"string"’.
Arbitrary Unicode characters can be written using a new escape sequence,
‘\uHHHH’, where 'HHHH' is a 4-digit hexadecimal number from 0000 to
FFFF. The existing ‘\xHH’ escape sequence can also be used, and octal
escapes can be used for characters up to U+01FF, which is represented by
‘\777’.
Unicode strings, just like regular strings, are an immutable sequence
type. They can be indexed and sliced, but not modified in place.
Unicode strings have an ‘encode( [encoding] )’ method that returns an
8-bit string in the desired encoding. Encodings are named by strings,
such as ‘'ascii'’, ‘'utf-8'’, ‘'iso-8859-1'’, or whatever. A codec API
is defined for implementing and registering new encodings that are then
available throughout a Python program. If an encoding isn’t specified,
the default encoding is usually 7-bit ASCII, though it can be changed
for your Python installation by calling the
‘sys.setdefaultencoding(encoding)’ function in a customized version of
‘site.py’.
Combining 8-bit and Unicode strings always coerces to Unicode, using the
default ASCII encoding; the result of ‘'a' + u'bc'’ is ‘u'abc'’.
New built-in functions have been added, and existing built-ins modified
to support Unicode:
* ‘unichr(ch)’ returns a Unicode string 1 character long, containing
the character 'ch'.
* ‘ord(u)’, where 'u' is a 1-character regular or Unicode string,
returns the number of the character as an integer.
* ‘unicode(string [, encoding] [, errors] )’ creates a Unicode string
from an 8-bit string. ‘encoding’ is a string naming the encoding
to use. The ‘errors’ parameter specifies the treatment of
characters that are invalid for the current encoding; passing
‘'strict'’ as the value causes an exception to be raised on any
encoding error, while ‘'ignore'’ causes errors to be silently
ignored and ‘'replace'’ uses U+FFFD, the official replacement
character, in case of any problems.
* The ‘exec’ statement, and various built-ins such as ‘eval()’,
‘getattr()’, and ‘setattr()’ will also accept Unicode strings as
well as regular strings. (It’s possible that the process of fixing
this missed some built-ins; if you find a built-in function that
accepts strings but doesn’t accept Unicode strings at all, please
report it as a bug.)
A new module, *note unicodedata: 105, provides an interface to Unicode
character properties. For example, ‘unicodedata.category(u'A')’ returns
the 2-character string ‘Lu’, the ‘L’ denoting it’s a letter, and ‘u’
meaning that it’s uppercase. ‘unicodedata.bidirectional(u'\u0660')’
returns ‘AN’, meaning that U+0660 is an Arabic number.
The *note codecs: 1b. module contains functions to look up existing
encodings and register new ones. Unless you want to implement a new
encoding, you’ll most often use the ‘codecs.lookup(encoding)’ function,
which returns a 4-element tuple: ‘(encode_func, decode_func,
stream_reader, stream_writer)’.
* 'encode_func' is a function that takes a Unicode string, and
returns a 2-tuple ‘(string, length)’. 'string' is an 8-bit string
containing a portion (perhaps all) of the Unicode string converted
into the given encoding, and 'length' tells you how much of the
Unicode string was converted.
* 'decode_func' is the opposite of 'encode_func', taking an 8-bit
string and returning a 2-tuple ‘(ustring, length)’, consisting of
the resulting Unicode string 'ustring' and the integer 'length'
telling how much of the 8-bit string was consumed.
* 'stream_reader' is a class that supports decoding input from a
stream. 'stream_reader(file_obj)' returns an object that supports
the ‘read()’, ‘readline()’, and ‘readlines()’ methods. These
methods will all translate from the given encoding and return
Unicode strings.
* 'stream_writer', similarly, is a class that supports encoding
output to a stream. 'stream_writer(file_obj)' returns an object
that supports the ‘write()’ and ‘writelines()’ methods. These
methods expect Unicode strings, translating them to the given
encoding on output.
For example, the following code writes a Unicode string into a file,
encoding it as UTF-8:
import codecs
unistr = u'\u0660\u2000ab ...'
(UTF8_encode, UTF8_decode,
UTF8_streamreader, UTF8_streamwriter) = codecs.lookup('UTF-8')
output = UTF8_streamwriter( open( '/tmp/output', 'wb') )
output.write( unistr )
output.close()
The following code would then read UTF-8 input from the file:
input = UTF8_streamreader( open( '/tmp/output', 'rb') )
print repr(input.read())
input.close()
Unicode-aware regular expressions are available through the *note re:
b9. module, which has a new underlying implementation called SRE written
by Fredrik Lundh of Secret Labs AB.
A ‘-U’ command line option was added which causes the Python compiler to
interpret all string literals as Unicode string literals. This is
intended to be used in testing and future-proofing your Python code,
since some future version of Python may drop support for 8-bit strings
and provide only Unicode strings.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0100/
File: python.info, Node: List Comprehensions, Next: Augmented Assignment, Prev: Unicode<2>, Up: What’s New in Python 2 0
1.22.5 List Comprehensions
--------------------------
Lists are a workhorse data type in Python, and many programs manipulate
a list at some point. Two common operations on lists are to loop over
them, and either pick out the elements that meet a certain criterion, or
apply some function to each element. For example, given a list of
strings, you might want to pull out all the strings containing a given
substring, or strip off trailing whitespace from each line.
The existing *note map(): 860. and *note filter(): 861. functions can be
used for this purpose, but they require a function as one of their
arguments. This is fine if there’s an existing built-in function that
can be passed directly, but if there isn’t, you have to create a little
function to do the required work, and Python’s scoping rules make the
result ugly if the little function needs additional information. Take
the first example in the previous paragraph, finding all the strings in
the list containing a given substring. You could write the following to
do it:
# Given the list L, make a list of all strings
# containing the substring S.
sublist = filter( lambda s, substring=S:
string.find(s, substring) != -1,
L)
Because of Python’s scoping rules, a default argument is used so that
the anonymous function created by the *note lambda: 128f. expression
knows what substring is being searched for. List comprehensions make
this cleaner:
sublist = [ s for s in L if string.find(s, S) != -1 ]
List comprehensions have the form:
[ expression for expr in sequence1
for expr2 in sequence2 ...
for exprN in sequenceN
if condition ]
The ‘for’…‘in’ clauses contain the sequences to be iterated over. The
sequences do not have to be the same length, because they are 'not'
iterated over in parallel, but from left to right; this is explained
more clearly in the following paragraphs. The elements of the generated
list will be the successive values of 'expression'. The final ‘if’
clause is optional; if present, 'expression' is only evaluated and added
to the result if 'condition' is true.
To make the semantics very clear, a list comprehension is equivalent to
the following Python code:
for expr1 in sequence1:
for expr2 in sequence2:
...
for exprN in sequenceN:
if (condition):
# Append the value of
# the expression to the
# resulting list.
This means that when there are multiple ‘for’…‘in’ clauses, the
resulting list will be equal to the product of the lengths of all the
sequences. If you have two lists of length 3, the output list is 9
elements long:
seq1 = 'abc'
seq2 = (1,2,3)
>>> [ (x,y) for x in seq1 for y in seq2]
[('a', 1), ('a', 2), ('a', 3), ('b', 1), ('b', 2), ('b', 3), ('c', 1),
('c', 2), ('c', 3)]
To avoid introducing an ambiguity into Python’s grammar, if 'expression'
is creating a tuple, it must be surrounded with parentheses. The first
list comprehension below is a syntax error, while the second one is
correct:
# Syntax error
[ x,y for x in seq1 for y in seq2]
# Correct
[ (x,y) for x in seq1 for y in seq2]
The idea of list comprehensions originally comes from the functional
programming language Haskell (‘https://www.haskell.org’). Greg Ewing
argued most effectively for adding them to Python and wrote the initial
list comprehension patch, which was then discussed for a seemingly
endless time on the python-dev mailing list and kept up-to-date by Skip
Montanaro.
File: python.info, Node: Augmented Assignment, Next: String Methods, Prev: List Comprehensions, Up: What’s New in Python 2 0
1.22.6 Augmented Assignment
---------------------------
Augmented assignment operators, another long-requested feature, have
been added to Python 2.0. Augmented assignment operators include ‘+=’,
‘-=’, ‘*=’, and so forth. For example, the statement ‘a += 2’
increments the value of the variable ‘a’ by 2, equivalent to the
slightly lengthier ‘a = a + 2’.
The full list of supported assignment operators is ‘+=’, ‘-=’, ‘*=’,
‘/=’, ‘%=’, ‘**=’, ‘&=’, ‘|=’, ‘^=’, ‘>>=’, and ‘<<=’. Python classes
can override the augmented assignment operators by defining methods
named ‘__iadd__()’, ‘__isub__()’, etc. For example, the following
‘Number’ class stores a number and supports using += to create a new
instance with an incremented value.
class Number:
def __init__(self, value):
self.value = value
def __iadd__(self, increment):
return Number( self.value + increment)
n = Number(5)
n += 3
print n.value
The ‘__iadd__()’ special method is called with the value of the
increment, and should return a new instance with an appropriately
modified value; this return value is bound as the new value of the
variable on the left-hand side.
Augmented assignment operators were first introduced in the C
programming language, and most C-derived languages, such as ‘awk’, C++,
Java, Perl, and PHP also support them. The augmented assignment patch
was implemented by Thomas Wouters.
File: python.info, Node: String Methods, Next: Garbage Collection of Cycles, Prev: Augmented Assignment, Up: What’s New in Python 2 0
1.22.7 String Methods
---------------------
Until now string-manipulation functionality was in the *note string: d3.
module, which was usually a front-end for the ‘strop’ module written in
C. The addition of Unicode posed a difficulty for the ‘strop’ module,
because the functions would all need to be rewritten in order to accept
either 8-bit or Unicode strings. For functions such as
‘string.replace()’, which takes 3 string arguments, that means eight
possible permutations, and correspondingly complicated code.
Instead, Python 2.0 pushes the problem onto the string type, making
string manipulation functionality available through methods on both
8-bit strings and Unicode strings.
>>> 'andrew'.capitalize()
'Andrew'
>>> 'hostname'.replace('os', 'linux')
'hlinuxtname'
>>> 'moshe'.find('sh')
2
One thing that hasn’t changed, a noteworthy April Fools’ joke
notwithstanding, is that Python strings are immutable. Thus, the string
methods return new strings, and do not modify the string on which they
operate.
The old *note string: d3. module is still around for backwards
compatibility, but it mostly acts as a front-end to the new string
methods.
Two methods which have no parallel in pre-2.0 versions, although they
did exist in JPython for quite some time, are ‘startswith()’ and
‘endswith()’. ‘s.startswith(t)’ is equivalent to ‘s[:len(t)] == t’,
while ‘s.endswith(t)’ is equivalent to ‘s[-len(t):] == t’.
One other method which deserves special mention is ‘join()’. The
‘join()’ method of a string receives one parameter, a sequence of
strings, and is equivalent to the ‘string.join()’ function from the old
*note string: d3. module, with the arguments reversed. In other words,
‘s.join(seq)’ is equivalent to the old ‘string.join(seq, s)’.
File: python.info, Node: Garbage Collection of Cycles, Next: Other Core Changes, Prev: String Methods, Up: What’s New in Python 2 0
1.22.8 Garbage Collection of Cycles
-----------------------------------
The C implementation of Python uses reference counting to implement
garbage collection. Every Python object maintains a count of the number
of references pointing to itself, and adjusts the count as references
are created or destroyed. Once the reference count reaches zero, the
object is no longer accessible, since you need to have a reference to an
object to access it, and if the count is zero, no references exist any
longer.
Reference counting has some pleasant properties: it’s easy to understand
and implement, and the resulting implementation is portable, fairly
fast, and reacts well with other libraries that implement their own
memory handling schemes. The major problem with reference counting is
that it sometimes doesn’t realise that objects are no longer accessible,
resulting in a memory leak. This happens when there are cycles of
references.
Consider the simplest possible cycle, a class instance which has a
reference to itself:
instance = SomeClass()
instance.myself = instance
After the above two lines of code have been executed, the reference
count of ‘instance’ is 2; one reference is from the variable named
‘'instance'’, and the other is from the ‘myself’ attribute of the
instance.
If the next line of code is ‘del instance’, what happens? The reference
count of ‘instance’ is decreased by 1, so it has a reference count of 1;
the reference in the ‘myself’ attribute still exists. Yet the instance
is no longer accessible through Python code, and it could be deleted.
Several objects can participate in a cycle if they have references to
each other, causing all of the objects to be leaked.
Python 2.0 fixes this problem by periodically executing a cycle
detection algorithm which looks for inaccessible cycles and deletes the
objects involved. A new *note gc: 60. module provides functions to
perform a garbage collection, obtain debugging statistics, and tuning
the collector’s parameters.
Running the cycle detection algorithm takes some time, and therefore
will result in some additional overhead. It is hoped that after we’ve
gotten experience with the cycle collection from using 2.0, Python 2.1
will be able to minimize the overhead with careful tuning. It’s not yet
obvious how much performance is lost, because benchmarking this is
tricky and depends crucially on how often the program creates and
destroys objects. The detection of cycles can be disabled when Python
is compiled, if you can’t afford even a tiny speed penalty or suspect
that the cycle collection is buggy, by specifying the
‘--without-cycle-gc’ switch when running the ‘configure’ script.
Several people tackled this problem and contributed to a solution. An
early implementation of the cycle detection approach was written by Toby
Kelsey. The current algorithm was suggested by Eric Tiedemann during a
visit to CNRI, and Guido van Rossum and Neil Schemenauer wrote two
different implementations, which were later integrated by Neil. Lots of
other people offered suggestions along the way; the March 2000 archives
of the python-dev mailing list contain most of the relevant discussion,
especially in the threads titled “Reference cycle collection for Python”
and “Finalization again”.
File: python.info, Node: Other Core Changes, Next: Porting to 2 0, Prev: Garbage Collection of Cycles, Up: What’s New in Python 2 0
1.22.9 Other Core Changes
-------------------------
Various minor changes have been made to Python’s syntax and built-in
functions. None of the changes are very far-reaching, but they’re handy
conveniences.
* Menu:
* Minor Language Changes::
* Changes to Built-in Functions::
File: python.info, Node: Minor Language Changes, Next: Changes to Built-in Functions, Up: Other Core Changes
1.22.9.1 Minor Language Changes
...............................
A new syntax makes it more convenient to call a given function with a
tuple of arguments and/or a dictionary of keyword arguments. In Python
1.5 and earlier, you’d use the ‘apply()’ built-in function: ‘apply(f,
args, kw)’ calls the function ‘f()’ with the argument tuple 'args' and
the keyword arguments in the dictionary 'kw'. ‘apply()’ is the same in
2.0, but thanks to a patch from Greg Ewing, ‘f(*args, **kw)’ is a
shorter and clearer way to achieve the same effect. This syntax is
symmetrical with the syntax for defining functions:
def f(*args, **kw):
# args is a tuple of positional args,
# kw is a dictionary of keyword args
...
The ‘print’ statement can now have its output directed to a file-like
object by following the ‘print’ with ‘>> file’, similar to the
redirection operator in Unix shells. Previously you’d either have to
use the ‘write()’ method of the file-like object, which lacks the
convenience and simplicity of ‘print’, or you could assign a new value
to ‘sys.stdout’ and then restore the old value. For sending output to
standard error, it’s much easier to write this:
print >> sys.stderr, "Warning: action field not supplied"
Modules can now be renamed on importing them, using the syntax ‘import
module as name’ or ‘from module import name as othername’. The patch
was submitted by Thomas Wouters.
A new format style is available when using the ‘%’ operator; ‘%r’ will
insert the *note repr(): 7f9. of its argument. This was also added from
symmetry considerations, this time for symmetry with the existing ‘%s’
format style, which inserts the *note str(): 447. of its argument. For
example, ‘'%r %s' % ('abc', 'abc')’ returns a string containing ‘'abc'
abc’.
Previously there was no way to implement a class that overrode Python’s
built-in *note in: 2ee. operator and implemented a custom version. ‘obj
in seq’ returns true if 'obj' is present in the sequence 'seq'; Python
computes this by simply trying every index of the sequence until either
'obj' is found or an *note IndexError: 14ff. is encountered. Moshe
Zadka contributed a patch which adds a ‘__contains__()’ magic method for
providing a custom implementation for ‘in’. Additionally, new built-in
objects written in C can define what ‘in’ means for them via a new slot
in the sequence protocol.
Earlier versions of Python used a recursive algorithm for deleting
objects. Deeply nested data structures could cause the interpreter to
fill up the C stack and crash; Christian Tismer rewrote the deletion
logic to fix this problem. On a related note, comparing recursive
objects recursed infinitely and crashed; Jeremy Hylton rewrote the code
to no longer crash, producing a useful result instead. For example,
after this code:
a = []
b = []
a.append(a)
b.append(b)
The comparison ‘a==b’ returns true, because the two recursive data
structures are isomorphic. See the thread “trashcan and PR#7” in the
April 2000 archives of the python-dev mailing list for the discussion
leading up to this implementation, and some useful relevant links. Note
that comparisons can now also raise exceptions. In earlier versions of
Python, a comparison operation such as ‘cmp(a,b)’ would always produce
an answer, even if a user-defined ‘__cmp__()’ method encountered an
error, since the resulting exception would simply be silently swallowed.
Work has been done on porting Python to 64-bit Windows on the Itanium
processor, mostly by Trent Mick of ActiveState. (Confusingly,
‘sys.platform’ is still ‘'win32'’ on Win64 because it seems that for
ease of porting, MS Visual C++ treats code as 32 bit on Itanium.)
PythonWin also supports Windows CE; see the Python CE page at
‘https://pythonce.sourceforge.net/’ for more information.
Another new platform is Darwin/MacOS X; initial support for it is in
Python 2.0. Dynamic loading works, if you specify “configure –with-dyld
–with-suffix=.x”. Consult the README in the Python source distribution
for more instructions.
An attempt has been made to alleviate one of Python’s warts, the
often-confusing *note NameError: 43a. exception when code refers to a
local variable before the variable has been assigned a value. For
example, the following code raises an exception on the ‘print’ statement
in both 1.5.2 and 2.0; in 1.5.2 a *note NameError: 43a. exception is
raised, while 2.0 raises a new *note UnboundLocalError: 1500. exception.
*note UnboundLocalError: 1500. is a subclass of *note NameError: 43a, so
any existing code that expects *note NameError: 43a. to be raised should
still work.
def f():
print "i=",i
i = i + 1
f()
Two new exceptions, *note TabError: 540. and *note IndentationError:
7a3, have been introduced. They’re both subclasses of *note
SyntaxError: 18d, and are raised when Python code is found to be
improperly indented.
File: python.info, Node: Changes to Built-in Functions, Prev: Minor Language Changes, Up: Other Core Changes
1.22.9.2 Changes to Built-in Functions
......................................
A new built-in, ‘zip(seq1, seq2, ...)’, has been added. *note zip():
7cb. returns a list of tuples where each tuple contains the i-th element
from each of the argument sequences. The difference between *note
zip(): 7cb. and ‘map(None, seq1, seq2)’ is that *note map(): 860. pads
the sequences with ‘None’ if the sequences aren’t all of the same
length, while *note zip(): 7cb. truncates the returned list to the
length of the shortest argument sequence.
The *note int(): 259. and ‘long()’ functions now accept an optional
“base” parameter when the first argument is a string. ‘int('123', 10)’
returns 123, while ‘int('123', 16)’ returns 291. ‘int(123, 16)’ raises
a *note TypeError: 534. exception with the message “can’t convert
non-string with explicit base”.
A new variable holding more detailed version information has been added
to the *note sys: d9. module. ‘sys.version_info’ is a tuple ‘(major,
minor, micro, level, serial)’ For example, in a hypothetical 2.0.1beta1,
‘sys.version_info’ would be ‘(2, 0, 1, 'beta', 1)’. 'level' is a string
such as ‘"alpha"’, ‘"beta"’, or ‘"final"’ for a final release.
Dictionaries have an odd new method, ‘setdefault(key, default)’, which
behaves similarly to the existing ‘get()’ method. However, if the key
is missing, ‘setdefault()’ both returns the value of 'default' as
‘get()’ would do, and also inserts it into the dictionary as the value
for 'key'. Thus, the following lines of code:
if dict.has_key( key ): return dict[key]
else:
dict[key] = []
return dict[key]
can be reduced to a single ‘return dict.setdefault(key, [])’ statement.
The interpreter sets a maximum recursion depth in order to catch runaway
recursion before filling the C stack and causing a core dump or GPF..
Previously this limit was fixed when you compiled Python, but in 2.0 the
maximum recursion depth can be read and modified using *note
sys.getrecursionlimit(): 4c0. and *note sys.setrecursionlimit(): 4bf.
The default value is 1000, and a rough maximum value for a given
platform can be found by running a new script,
‘Misc/find_recursionlimit.py’.
File: python.info, Node: Porting to 2 0, Next: Extending/Embedding Changes, Prev: Other Core Changes, Up: What’s New in Python 2 0
1.22.10 Porting to 2.0
----------------------
New Python releases try hard to be compatible with previous releases,
and the record has been pretty good. However, some changes are
considered useful enough, usually because they fix initial design
decisions that turned out to be actively mistaken, that breaking
backward compatibility can’t always be avoided. This section lists the
changes in Python 2.0 that may cause old Python code to break.
The change which will probably break the most code is tightening up the
arguments accepted by some methods. Some methods would take multiple
arguments and treat them as a tuple, particularly various list methods
such as ‘append()’ and ‘insert()’. In earlier versions of Python, if
‘L’ is a list, ‘L.append( 1,2 )’ appends the tuple ‘(1,2)’ to the list.
In Python 2.0 this causes a *note TypeError: 534. exception to be
raised, with the message: ‘append requires exactly 1 argument; 2 given’.
The fix is to simply add an extra set of parentheses to pass both values
as a tuple: ‘L.append( (1,2) )’.
The earlier versions of these methods were more forgiving because they
used an old function in Python’s C interface to parse their arguments;
2.0 modernizes them to use *note PyArg_ParseTuple(): 56e, the current
argument parsing function, which provides more helpful error messages
and treats multi-argument calls as errors. If you absolutely must use
2.0 but can’t fix your code, you can edit ‘Objects/listobject.c’ and
define the preprocessor symbol ‘NO_STRICT_LIST_APPEND’ to preserve the
old behaviour; this isn’t recommended.
Some of the functions in the *note socket: cc. module are still
forgiving in this way. For example, ‘socket.connect( ('hostname', 25)
)’ is the correct form, passing a tuple representing an IP address, but
‘socket.connect('hostname', 25)’ also works. *note socket.connect_ex:
1503. and *note socket.bind: 1504. are similarly easy-going. 2.0alpha1
tightened these functions up, but because the documentation actually
used the erroneous multiple argument form, many people wrote code which
would break with the stricter checking. GvR backed out the changes in
the face of public reaction, so for the *note socket: cc. module, the
documentation was fixed and the multiple argument form is simply marked
as deprecated; it 'will' be tightened up again in a future Python
version.
The ‘\x’ escape in string literals now takes exactly 2 hex digits.
Previously it would consume all the hex digits following the ‘x’ and
take the lowest 8 bits of the result, so ‘\x123456’ was equivalent to
‘\x56’.
The *note AttributeError: 348. and *note NameError: 43a. exceptions have
a more friendly error message, whose text will be something like ‘'Spam'
instance has no attribute 'eggs'’ or ‘name 'eggs' is not defined’.
Previously the error message was just the missing attribute name ‘eggs’,
and code written to take advantage of this fact will break in 2.0.
Some work has been done to make integers and long integers a bit more
interchangeable. In 1.5.2, large-file support was added for Solaris, to
allow reading files larger than 2 GiB; this made the ‘tell()’ method of
file objects return a long integer instead of a regular integer. Some
code would subtract two file offsets and attempt to use the result to
multiply a sequence or slice a string, but this raised a *note
TypeError: 534. In 2.0, long integers can be used to multiply or slice
a sequence, and it’ll behave as you’d intuitively expect it to; ‘3L *
'abc'’ produces ‘abcabcabc’, and ‘(0,1,2,3)[2L:4L]’ produces (2,3).
Long integers can also be used in various contexts where previously only
integers were accepted, such as in the ‘seek()’ method of file objects,
and in the formats supported by the ‘%’ operator (‘%d’, ‘%i’, ‘%x’,
etc.). For example, ‘"%d" % 2L**64’ will produce the string
‘18446744073709551616’.
The subtlest long integer change of all is that the *note str(): 447. of
a long integer no longer has a trailing ‘L’ character, though *note
repr(): 7f9. still includes it. The ‘L’ annoyed many people who wanted
to print long integers that looked just like regular integers, since
they had to go out of their way to chop off the character. This is no
longer a problem in 2.0, but code which does ‘str(longval)[:-1]’ and
assumes the ‘L’ is there, will now lose the final digit.
Taking the *note repr(): 7f9. of a float now uses a different formatting
precision than *note str(): 447. *note repr(): 7f9. uses ‘%.17g’ format
string for C’s ‘sprintf()’, while *note str(): 447. uses ‘%.12g’ as
before. The effect is that *note repr(): 7f9. may occasionally show
more decimal places than *note str(): 447, for certain numbers. For
example, the number 8.1 can’t be represented exactly in binary, so
‘repr(8.1)’ is ‘'8.0999999999999996'’, while str(8.1) is ‘'8.1'’.
The ‘-X’ command-line option, which turned all standard exceptions into
strings instead of classes, has been removed; the standard exceptions
will now always be classes. The ‘exceptions’ module containing the
standard exceptions was translated from Python to a built-in C module,
written by Barry Warsaw and Fredrik Lundh.
File: python.info, Node: Extending/Embedding Changes, Next: Distutils Making Modules Easy to Install, Prev: Porting to 2 0, Up: What’s New in Python 2 0
1.22.11 Extending/Embedding Changes
-----------------------------------
Some of the changes are under the covers, and will only be apparent to
people writing C extension modules or embedding a Python interpreter in
a larger application. If you aren’t dealing with Python’s C API, you
can safely skip this section.
The version number of the Python C API was incremented, so C extensions
compiled for 1.5.2 must be recompiled in order to work with 2.0. On
Windows, it’s not possible for Python 2.0 to import a third party
extension built for Python 1.5.x due to how Windows DLLs work, so Python
will raise an exception and the import will fail.
Users of Jim Fulton’s ExtensionClass module will be pleased to find out
that hooks have been added so that ExtensionClasses are now supported by
*note isinstance(): 43d. and *note issubclass(): 7bc. This means you no
longer have to remember to write code such as ‘if type(obj) ==
myExtensionClass’, but can use the more natural ‘if isinstance(obj,
myExtensionClass)’.
The ‘Python/importdl.c’ file, which was a mass of #ifdefs to support
dynamic loading on many different platforms, was cleaned up and
reorganised by Greg Stein. ‘importdl.c’ is now quite small, and
platform-specific code has been moved into a bunch of
‘Python/dynload_*.c’ files. Another cleanup: there were also a number
of ‘my*.h’ files in the Include/ directory that held various portability
hacks; they’ve been merged into a single file, ‘Include/pyport.h’.
Vladimir Marangozov’s long-awaited malloc restructuring was completed,
to make it easy to have the Python interpreter use a custom allocator
instead of C’s standard ‘malloc()’. For documentation, read the
comments in ‘Include/pymem.h’ and ‘Include/objimpl.h’. For the lengthy
discussions during which the interface was hammered out, see the web
archives of the ‘patches’ and ‘python-dev’ lists at python.org.
Recent versions of the GUSI development environment for MacOS support
POSIX threads. Therefore, Python’s POSIX threading support now works on
the Macintosh. Threading support using the user-space GNU ‘pth’ library
was also contributed.
Threading support on Windows was enhanced, too. Windows supports thread
locks that use kernel objects only in case of contention; in the common
case when there’s no contention, they use simpler functions which are an
order of magnitude faster. A threaded version of Python 1.5.2 on NT is
twice as slow as an unthreaded version; with the 2.0 changes, the
difference is only 10%. These improvements were contributed by Yakov
Markovitch.
Python 2.0’s source now uses only ANSI C prototypes, so compiling Python
now requires an ANSI C compiler, and can no longer be done using a
compiler that only supports K&R C.
Previously the Python virtual machine used 16-bit numbers in its
bytecode, limiting the size of source files. In particular, this
affected the maximum size of literal lists and dictionaries in Python
source; occasionally people who are generating Python code would run
into this limit. A patch by Charles G. Waldman raises the limit from
‘2**16’ to ‘2**32’.
Three new convenience functions intended for adding constants to a
module’s dictionary at module initialization time were added: *note
PyModule_AddObject(): 361, *note PyModule_AddIntConstant(): 1506, and
*note PyModule_AddStringConstant(): 1507. Each of these functions takes
a module object, a null-terminated C string containing the name to be
added, and a third argument for the value to be assigned to the name.
This third argument is, respectively, a Python object, a C long, or a C
string.
A wrapper API was added for Unix-style signal handlers. *note
PyOS_getsig(): 1508. gets a signal handler and *note PyOS_setsig():
1509. will set a new handler.
File: python.info, Node: Distutils Making Modules Easy to Install, Next: XML Modules, Prev: Extending/Embedding Changes, Up: What’s New in Python 2 0
1.22.12 Distutils: Making Modules Easy to Install
-------------------------------------------------
Before Python 2.0, installing modules was a tedious affair – there was
no way to figure out automatically where Python is installed, or what
compiler options to use for extension modules. Software authors had to
go through an arduous ritual of editing Makefiles and configuration
files, which only really work on Unix and leave Windows and MacOS
unsupported. Python users faced wildly differing installation
instructions which varied between different extension packages, which
made administering a Python installation something of a chore.
The SIG for distribution utilities, shepherded by Greg Ward, has created
the Distutils, a system to make package installation much easier. They
form the ‘distutils’ package, a new part of Python’s standard library.
In the best case, installing a Python module from source will require
the same steps: first you simply mean unpack the tarball or zip archive,
and the run “‘python setup.py install’”. The platform will be
automatically detected, the compiler will be recognized, C extension
modules will be compiled, and the distribution installed into the proper
directory. Optional command-line arguments provide more control over
the installation process, the distutils package offers many places to
override defaults – separating the build from the install, building or
installing in non-default directories, and more.
In order to use the Distutils, you need to write a ‘setup.py’ script.
For the simple case, when the software contains only .py files, a
minimal ‘setup.py’ can be just a few lines long:
from distutils.core import setup
setup (name = "foo", version = "1.0",
py_modules = ["module1", "module2"])
The ‘setup.py’ file isn’t much more complicated if the software consists
of a few packages:
from distutils.core import setup
setup (name = "foo", version = "1.0",
packages = ["package", "package.subpackage"])
A C extension can be the most complicated case; here’s an example taken
from the PyXML package:
from distutils.core import setup, Extension
expat_extension = Extension('xml.parsers.pyexpat',
define_macros = [('XML_NS', None)],
include_dirs = [ 'extensions/expat/xmltok',
'extensions/expat/xmlparse' ],
sources = [ 'extensions/pyexpat.c',
'extensions/expat/xmltok/xmltok.c',
'extensions/expat/xmltok/xmlrole.c', ]
)
setup (name = "PyXML", version = "0.5.4",
ext_modules =[ expat_extension ] )
The Distutils can also take care of creating source and binary
distributions. The “sdist” command, run by “‘python setup.py sdist’’,
builds a source distribution such as ‘foo-1.0.tar.gz’. Adding new
commands isn’t difficult, “bdist_rpm” and “bdist_wininst” commands have
already been contributed to create an RPM distribution and a Windows
installer for the software, respectively. Commands to create other
distribution formats such as Debian packages and Solaris ‘.pkg’ files
are in various stages of development.
All this is documented in a new manual, 'Distributing Python Modules',
that joins the basic set of Python documentation.
File: python.info, Node: XML Modules, Next: Module changes, Prev: Distutils Making Modules Easy to Install, Up: What’s New in Python 2 0
1.22.13 XML Modules
-------------------
Python 1.5.2 included a simple XML parser in the form of the ‘xmllib’
module, contributed by Sjoerd Mullender. Since 1.5.2’s release, two
different interfaces for processing XML have become common: SAX2
(version 2 of the Simple API for XML) provides an event-driven interface
with some similarities to ‘xmllib’, and the DOM (Document Object Model)
provides a tree-based interface, transforming an XML document into a
tree of nodes that can be traversed and modified. Python 2.0 includes a
SAX2 interface and a stripped-down DOM interface as part of the *note
xml: 120. package. Here we will give a brief overview of these new
interfaces; consult the Python documentation or the source code for
complete details. The Python XML SIG is also working on improved
documentation.
* Menu:
* SAX2 Support::
* DOM Support::
* Relationship to PyXML::
File: python.info, Node: SAX2 Support, Next: DOM Support, Up: XML Modules
1.22.13.1 SAX2 Support
......................
SAX defines an event-driven interface for parsing XML. To use SAX, you
must write a SAX handler class. Handler classes inherit from various
classes provided by SAX, and override various methods that will then be
called by the XML parser. For example, the *note startElement(): 150d.
and *note endElement(): 150e. methods are called for every starting and
end tag encountered by the parser, the *note characters(): 150f. method
is called for every chunk of character data, and so forth.
The advantage of the event-driven approach is that the whole document
doesn’t have to be resident in memory at any one time, which matters if
you are processing really huge documents. However, writing the SAX
handler class can get very complicated if you’re trying to modify the
document structure in some elaborate way.
For example, this little example program defines a handler that prints a
message for every starting and ending tag, and then parses the file
‘hamlet.xml’ using it:
from xml import sax
class SimpleHandler(sax.ContentHandler):
def startElement(self, name, attrs):
print 'Start of element:', name, attrs.keys()
def endElement(self, name):
print 'End of element:', name
# Create a parser object
parser = sax.make_parser()
# Tell it what handler to use
handler = SimpleHandler()
parser.setContentHandler( handler )
# Parse a file!
parser.parse( 'hamlet.xml' )
For more information, consult the Python documentation, or the XML HOWTO
at ‘https://pyxml.sourceforge.net/topics/howto/xml-howto.html’.
File: python.info, Node: DOM Support, Next: Relationship to PyXML, Prev: SAX2 Support, Up: XML Modules
1.22.13.2 DOM Support
.....................
The Document Object Model is a tree-based representation for an XML
document. A top-level ‘Document’ instance is the root of the tree, and
has a single child which is the top-level ‘Element’ instance. This
‘Element’ has children nodes representing character data and any
sub-elements, which may have further children of their own, and so
forth. Using the DOM you can traverse the resulting tree any way you
like, access element and attribute values, insert and delete nodes, and
convert the tree back into XML.
The DOM is useful for modifying XML documents, because you can create a
DOM tree, modify it by adding new nodes or rearranging subtrees, and
then produce a new XML document as output. You can also construct a DOM
tree manually and convert it to XML, which can be a more flexible way of
producing XML output than simply writing ‘’…‘ ’ to a file.
The DOM implementation included with Python lives in the *note
xml.dom.minidom: 122. module. It’s a lightweight implementation of the
Level 1 DOM with support for XML namespaces. The ‘parse()’ and
‘parseString()’ convenience functions are provided for generating a DOM
tree:
from xml.dom import minidom
doc = minidom.parse('hamlet.xml')
‘doc’ is a ‘Document’ instance. ‘Document’, like all the other DOM
classes such as ‘Element’ and ‘Text’, is a subclass of the ‘Node’ base
class. All the nodes in a DOM tree therefore support certain common
methods, such as ‘toxml()’ which returns a string containing the XML
representation of the node and its children. Each class also has
special methods of its own; for example, ‘Element’ and ‘Document’
instances have a method to find all child elements with a given tag
name. Continuing from the previous 2-line example:
perslist = doc.getElementsByTagName( 'PERSONA' )
print perslist[0].toxml()
print perslist[1].toxml()
For the 'Hamlet' XML file, the above few lines output:
CLAUDIUS, king of Denmark.
HAMLET, son to the late, and nephew to the present king.
The root element of the document is available as ‘doc.documentElement’,
and its children can be easily modified by deleting, adding, or removing
nodes:
root = doc.documentElement
# Remove the first child
root.removeChild( root.childNodes[0] )
# Move the new first child to the end
root.appendChild( root.childNodes[0] )
# Insert the new first child (originally,
# the third child) before the 20th child.
root.insertBefore( root.childNodes[0], root.childNodes[20] )
Again, I will refer you to the Python documentation for a complete
listing of the different ‘Node’ classes and their various methods.
File: python.info, Node: Relationship to PyXML, Prev: DOM Support, Up: XML Modules
1.22.13.3 Relationship to PyXML
...............................
The XML Special Interest Group has been working on XML-related Python
code for a while. Its code distribution, called PyXML, is available
from the SIG’s web pages at
‘https://www.python.org/community/sigs/current/xml-sig’. The PyXML
distribution also used the package name ‘xml’. If you’ve written
programs that used PyXML, you’re probably wondering about its
compatibility with the 2.0 *note xml: 120. package.
The answer is that Python 2.0’s *note xml: 120. package isn’t compatible
with PyXML, but can be made compatible by installing a recent version
PyXML. Many applications can get by with the XML support that is
included with Python 2.0, but more complicated applications will require
that the full PyXML package will be installed. When installed, PyXML
versions 0.6.0 or greater will replace the *note xml: 120. package
shipped with Python, and will be a strict superset of the standard
package, adding a bunch of additional features. Some of the additional
features in PyXML include:
* 4DOM, a full DOM implementation from FourThought, Inc.
* The xmlproc validating parser, written by Lars Marius Garshol.
* The ‘sgmlop’ parser accelerator module, written by Fredrik Lundh.
File: python.info, Node: Module changes, Next: New modules, Prev: XML Modules, Up: What’s New in Python 2 0
1.22.14 Module changes
----------------------
Lots of improvements and bugfixes were made to Python’s extensive
standard library; some of the affected modules include *note readline:
ba, *note ConfigParser: 22, ‘cgi’, *note calendar: 14, *note posix: ad,
*note readline: ba, ‘xmllib’, ‘aifc’, ‘chunk’, *note wave: 113, *note
random: b8, *note shelve: c3, and ‘nntplib’. Consult the CVS logs for
the exact patch-by-patch details.
Brian Gallew contributed OpenSSL support for the *note socket: cc.
module. OpenSSL is an implementation of the Secure Socket Layer, which
encrypts the data being sent over a socket. When compiling Python, you
can edit ‘Modules/Setup’ to include SSL support, which adds an
additional function to the *note socket: cc. module: ‘socket.ssl(socket,
keyfile, certfile)’, which takes a socket object and returns an SSL
socket. The *note httplib: 6e. and *note urllib: 108. modules were also
changed to support ‘https://’ URLs, though no one has implemented FTP or
SMTP over SSL.
The *note httplib: 6e. module has been rewritten by Greg Stein to
support HTTP/1.1.
Backward compatibility with the 1.5 version of ‘httplib’ is provided,
though using HTTP/1.1 features such as pipelining will require rewriting
code to use a different set of interfaces.
The ‘Tkinter’ module now supports Tcl/Tk version 8.1, 8.2, or 8.3, and
support for the older 7.x versions has been dropped. The Tkinter module
now supports displaying Unicode strings in Tk widgets. Also, Fredrik
Lundh contributed an optimization which makes operations like
‘create_line’ and ‘create_polygon’ much faster, especially when using
lots of coordinates.
The *note curses: 2b. module has been greatly extended, starting from
Oliver Andrich’s enhanced version, to provide many additional functions
from ncurses and SYSV curses, such as colour, alternative character set
support, pads, and mouse support. This means the module is no longer
compatible with operating systems that only have BSD curses, but there
don’t seem to be any currently maintained OSes that fall into this
category.
As mentioned in the earlier discussion of 2.0’s Unicode support, the
underlying implementation of the regular expressions provided by the
*note re: b9. module has been changed. SRE, a new regular expression
engine written by Fredrik Lundh and partially funded by Hewlett Packard,
supports matching against both 8-bit strings and Unicode strings.
File: python.info, Node: New modules, Next: IDLE Improvements, Prev: Module changes, Up: What’s New in Python 2 0
1.22.15 New modules
-------------------
A number of new modules were added. We’ll simply list them with brief
descriptions; consult the 2.0 documentation for the details of a
particular module.
* *note atexit: c.: For registering functions to be called before the
Python interpreter exits. Code that currently sets ‘sys.exitfunc’
directly should be changed to use the *note atexit: c. module
instead, importing *note atexit: c. and calling *note
atexit.register(): 87b. with the function to be called on exit.
(Contributed by Skip Montanaro.)
* *note codecs: 1b, ‘encodings’, *note unicodedata: 105.: Added as
part of the new Unicode support.
* *note filecmp: 5a.: Supersedes the old ‘cmp’, ‘cmpcache’ and
‘dircmp’ modules, which have now become deprecated. (Contributed
by Gordon MacMillan and Moshe Zadka.)
* *note gettext: 63.: This module provides internationalization
(I18N) and localization (L10N) support for Python programs by
providing an interface to the GNU gettext message catalog library.
(Integrated by Barry Warsaw, from separate contributions by Martin
von Löwis, Peter Funk, and James Henstridge.)
* ‘linuxaudiodev’: Support for the ‘/dev/audio’ device on Linux, a
twin to the existing ‘sunaudiodev’ module. (Contributed by Peter
Bosch, with fixes by Jeremy Hylton.)
* *note mmap: 90.: An interface to memory-mapped files on both
Windows and Unix. A file’s contents can be mapped directly into
memory, at which point it behaves like a mutable string, so its
contents can be read and modified. They can even be passed to
functions that expect ordinary strings, such as the *note re: b9.
module. (Contributed by Sam Rushing, with some extensions by A.M.
Kuchling.)
* ‘pyexpat’: An interface to the Expat XML parser. (Contributed by
Paul Prescod.)
* *note robotparser: 10d.: Parse a ‘robots.txt’ file, which is used
for writing web spiders that politely avoid certain areas of a web
site. The parser accepts the contents of a ‘robots.txt’ file,
builds a set of rules from it, and can then answer questions about
the fetchability of a given URL. (Contributed by Skip Montanaro.)
* *note tabnanny: dd.: A module/script to check Python source code
for ambiguous indentation. (Contributed by Tim Peters.)
* ‘UserString’: A base class useful for deriving objects that behave
like strings.
* *note webbrowser: 115.: A module that provides a platform
independent way to launch a web browser on a specific URL. For each
platform, various browsers are tried in a specific order. The user
can alter which browser is launched by setting the 'BROWSER'
environment variable. (Originally inspired by Eric S. Raymond’s
patch to *note urllib: 108. which added similar functionality, but
the final module comes from code originally implemented by Fred
Drake as ‘Tools/idle/BrowserControl.py’, and adapted for the
standard library by Fred.)
* *note _winreg: 116.: An interface to the Windows registry.
‘_winreg’ is an adaptation of functions that have been part of
PythonWin since 1995, but has now been added to the core
distribution, and enhanced to support Unicode. ‘_winreg’ was
written by Bill Tutt and Mark Hammond.
* *note zipfile: 131.: A module for reading and writing ZIP-format
archives. These are archives produced by ‘PKZIP’ on DOS/Windows or
‘zip’ on Unix, not to be confused with ‘gzip’-format files (which
are supported by the *note gzip: 67. module) (Contributed by James
C. Ahlstrom.)
* ‘imputil’: A module that provides a simpler way for writing
customized import hooks, in comparison to the existing ‘ihooks’
module. (Implemented by Greg Stein, with much discussion on
python-dev along the way.)
File: python.info, Node: IDLE Improvements, Next: Deleted and Deprecated Modules, Prev: New modules, Up: What’s New in Python 2 0
1.22.16 IDLE Improvements
-------------------------
IDLE is the official Python cross-platform IDE, written using Tkinter.
Python 2.0 includes IDLE 0.6, which adds a number of new features and
improvements. A partial list:
* UI improvements and optimizations, especially in the area of syntax
highlighting and auto-indentation.
* The class browser now shows more information, such as the top level
functions in a module.
* Tab width is now a user settable option. When opening an existing
Python file, IDLE automatically detects the indentation
conventions, and adapts.
* There is now support for calling browsers on various platforms,
used to open the Python documentation in a browser.
* IDLE now has a command line, which is largely similar to the
vanilla Python interpreter.
* Call tips were added in many places.
* IDLE can now be installed as a package.
* In the editor window, there is now a line/column bar at the bottom.
* Three new keystroke commands: Check module (‘Alt’-‘F5’), Import
module (‘F5’) and Run script (‘Ctrl’-‘F5’).
File: python.info, Node: Deleted and Deprecated Modules, Next: Acknowledgements<8>, Prev: IDLE Improvements, Up: What’s New in Python 2 0
1.22.17 Deleted and Deprecated Modules
--------------------------------------
A few modules have been dropped because they’re obsolete, or because
there are now better ways to do the same thing. The ‘stdwin’ module is
gone; it was for a platform-independent windowing toolkit that’s no
longer developed.
A number of modules have been moved to the ‘lib-old’ subdirectory:
‘cmp’, ‘cmpcache’, ‘dircmp’, ‘dump’, ‘find’, ‘grep’, ‘packmail’, ‘poly’,
‘util’, ‘whatsound’, ‘zmod’. If you have code which relies on a module
that’s been moved to ‘lib-old’, you can simply add that directory to
‘sys.path’ to get them back, but you’re encouraged to update any code
that uses these modules.
File: python.info, Node: Acknowledgements<8>, Prev: Deleted and Deprecated Modules, Up: What’s New in Python 2 0
1.22.18 Acknowledgements
------------------------
The authors would like to thank the following people for offering
suggestions on various drafts of this article: David Bolen, Mark
Hammond, Gregg Hauser, Jeremy Hylton, Fredrik Lundh, Detlef Lannert,
Aahz Maruch, Skip Montanaro, Vladimir Marangozov, Tobias Polzin, Guido
van Rossum, Neil Schemenauer, and Russ Schmidt.
The “Changelog” is an HTML version of the file built(1) from the
contents of the Misc/NEWS.d(2) directory tree, which contains 'all'
nontrivial changes to Python for the current version.
---------- Footnotes ----------
(1) https://pypi.org/project/blurb/
(2) https://github.com/python/cpython/tree/3.13/Misc/NEWS.d
File: python.info, Node: Changelog, Prev: What’s New in Python 2 0, Up: What’s New in Python
1.23 Changelog
==============
* Menu:
* Python 3.13.7 final: Python 3 13 7 final.
* Python 3.13.6 final: Python 3 13 6 final.
* Python 3.13.5 final: Python 3 13 5 final.
* Python 3.13.4 final: Python 3 13 4 final.
* Python 3.13.3 final: Python 3 13 3 final.
* Python 3.13.2 final: Python 3 13 2 final.
* Python 3.13.1 final: Python 3 13 1 final.
* Python 3.13.0 final: Python 3 13 0 final.
* Python 3.13.0 release candidate 3: Python 3 13 0 release candidate 3.
* Python 3.13.0 release candidate 2: Python 3 13 0 release candidate 2.
* Python 3.13.0 release candidate 1: Python 3 13 0 release candidate 1.
* Python 3.13.0 beta 4: Python 3 13 0 beta 4.
* Python 3.13.0 beta 3: Python 3 13 0 beta 3.
* Python 3.13.0 beta 2: Python 3 13 0 beta 2.
* Python 3.13.0 beta 1: Python 3 13 0 beta 1.
* Python 3.13.0 alpha 6: Python 3 13 0 alpha 6.
* Python 3.13.0 alpha 5: Python 3 13 0 alpha 5.
* Python 3.13.0 alpha 4: Python 3 13 0 alpha 4.
* Python 3.13.0 alpha 3: Python 3 13 0 alpha 3.
* Python 3.13.0 alpha 2: Python 3 13 0 alpha 2.
* Python 3.13.0 alpha 1: Python 3 13 0 alpha 1.
* Python 3.12.0 beta 1: Python 3 12 0 beta 1.
* Python 3.12.0 alpha 7: Python 3 12 0 alpha 7.
* Python 3.12.0 alpha 6: Python 3 12 0 alpha 6.
* Python 3.12.0 alpha 5: Python 3 12 0 alpha 5.
* Python 3.12.0 alpha 4: Python 3 12 0 alpha 4.
* Python 3.12.0 alpha 3: Python 3 12 0 alpha 3.
* Python 3.12.0 alpha 2: Python 3 12 0 alpha 2.
* Python 3.12.0 alpha 1: Python 3 12 0 alpha 1.
* Python 3.11.0 beta 1: Python 3 11 0 beta 1.
* Python 3.11.0 alpha 7: Python 3 11 0 alpha 7.
* Python 3.11.0 alpha 6: Python 3 11 0 alpha 6.
* Python 3.11.0 alpha 5: Python 3 11 0 alpha 5.
* Python 3.11.0 alpha 4: Python 3 11 0 alpha 4.
* Python 3.11.0 alpha 3: Python 3 11 0 alpha 3.
* Python 3.11.0 alpha 2: Python 3 11 0 alpha 2.
* Python 3.11.0 alpha 1: Python 3 11 0 alpha 1.
* Python 3.10.0 beta 1: Python 3 10 0 beta 1.
* Python 3.10.0 alpha 7: Python 3 10 0 alpha 7.
* Python 3.10.0 alpha 6: Python 3 10 0 alpha 6.
* Python 3.10.0 alpha 5: Python 3 10 0 alpha 5.
* Python 3.10.0 alpha 4: Python 3 10 0 alpha 4.
* Python 3.10.0 alpha 3: Python 3 10 0 alpha 3.
* Python 3.10.0 alpha 2: Python 3 10 0 alpha 2.
* Python 3.10.0 alpha 1: Python 3 10 0 alpha 1.
* Python 3.9.0 beta 1: Python 3 9 0 beta 1.
* Python 3.9.0 alpha 6: Python 3 9 0 alpha 6.
* Python 3.9.0 alpha 5: Python 3 9 0 alpha 5.
* Python 3.9.0 alpha 4: Python 3 9 0 alpha 4.
* Python 3.9.0 alpha 3: Python 3 9 0 alpha 3.
* Python 3.9.0 alpha 2: Python 3 9 0 alpha 2.
* Python 3.9.0 alpha 1: Python 3 9 0 alpha 1.
* Python 3.8.0 beta 1: Python 3 8 0 beta 1.
* Python 3.8.0 alpha 4: Python 3 8 0 alpha 4.
* Python 3.8.0 alpha 3: Python 3 8 0 alpha 3.
* Python 3.8.0 alpha 2: Python 3 8 0 alpha 2.
* Python 3.8.0 alpha 1: Python 3 8 0 alpha 1.
* Python 3.7.0 final: Python 3 7 0 final.
* Python 3.7.0 release candidate 1: Python 3 7 0 release candidate 1.
* Python 3.7.0 beta 5: Python 3 7 0 beta 5.
* Python 3.7.0 beta 4: Python 3 7 0 beta 4.
* Python 3.7.0 beta 3: Python 3 7 0 beta 3.
* Python 3.7.0 beta 2: Python 3 7 0 beta 2.
* Python 3.7.0 beta 1: Python 3 7 0 beta 1.
* Python 3.7.0 alpha 4: Python 3 7 0 alpha 4.
* Python 3.7.0 alpha 3: Python 3 7 0 alpha 3.
* Python 3.7.0 alpha 2: Python 3 7 0 alpha 2.
* Python 3.7.0 alpha 1: Python 3 7 0 alpha 1.
* Python 3.6.6 final: Python 3 6 6 final.
* Python 3.6.6 release candidate 1: Python 3 6 6 release candidate 1.
* Python 3.6.5 final: Python 3 6 5 final.
* Python 3.6.5 release candidate 1: Python 3 6 5 release candidate 1.
* Python 3.6.4 final: Python 3 6 4 final.
* Python 3.6.4 release candidate 1: Python 3 6 4 release candidate 1.
* Python 3.6.3 final: Python 3 6 3 final.
* Python 3.6.3 release candidate 1: Python 3 6 3 release candidate 1.
* Python 3.6.2 final: Python 3 6 2 final.
* Python 3.6.2 release candidate 2: Python 3 6 2 release candidate 2.
* Python 3.6.2 release candidate 1: Python 3 6 2 release candidate 1.
* Python 3.6.1 final: Python 3 6 1 final.
* Python 3.6.1 release candidate 1: Python 3 6 1 release candidate 1.
* Python 3.6.0 final: Python 3 6 0 final.
* Python 3.6.0 release candidate 2: Python 3 6 0 release candidate 2.
* Python 3.6.0 release candidate 1: Python 3 6 0 release candidate 1.
* Python 3.6.0 beta 4: Python 3 6 0 beta 4.
* Python 3.6.0 beta 3: Python 3 6 0 beta 3.
* Python 3.6.0 beta 2: Python 3 6 0 beta 2.
* Python 3.6.0 beta 1: Python 3 6 0 beta 1.
* Python 3.6.0 alpha 4: Python 3 6 0 alpha 4.
* Python 3.6.0 alpha 3: Python 3 6 0 alpha 3.
* Python 3.6.0 alpha 2: Python 3 6 0 alpha 2.
* Python 3.6.0 alpha 1: Python 3 6 0 alpha 1.
* Python 3.5.5 final: Python 3 5 5 final.
* Python 3.5.5 release candidate 1: Python 3 5 5 release candidate 1.
* Python 3.5.4 final: Python 3 5 4 final.
* Python 3.5.4 release candidate 1: Python 3 5 4 release candidate 1.
* Python 3.5.3 final: Python 3 5 3 final.
* Python 3.5.3 release candidate 1: Python 3 5 3 release candidate 1.
* Python 3.5.2 final: Python 3 5 2 final.
* Python 3.5.2 release candidate 1: Python 3 5 2 release candidate 1.
* Python 3.5.1 final: Python 3 5 1 final.
* Python 3.5.1 release candidate 1: Python 3 5 1 release candidate 1.
* Python 3.5.0 final: Python 3 5 0 final.
* Python 3.5.0 release candidate 4: Python 3 5 0 release candidate 4.
* Python 3.5.0 release candidate 3: Python 3 5 0 release candidate 3.
* Python 3.5.0 release candidate 2: Python 3 5 0 release candidate 2.
* Python 3.5.0 release candidate 1: Python 3 5 0 release candidate 1.
* Python 3.5.0 beta 4: Python 3 5 0 beta 4.
* Python 3.5.0 beta 3: Python 3 5 0 beta 3.
* Python 3.5.0 beta 2: Python 3 5 0 beta 2.
* Python 3.5.0 beta 1: Python 3 5 0 beta 1.
* Python 3.5.0 alpha 4: Python 3 5 0 alpha 4.
* Python 3.5.0 alpha 3: Python 3 5 0 alpha 3.
* Python 3.5.0 alpha 2: Python 3 5 0 alpha 2.
* Python 3.5.0 alpha 1: Python 3 5 0 alpha 1.
File: python.info, Node: Python 3 13 7 final, Next: Python 3 13 6 final, Up: Changelog
1.23.1 Python 3.13.7 final
--------------------------
'Release date: 2025-08-14'
* Menu:
* Library::
* Documentation: Documentation<2>.
* Core and Builtins::
File: python.info, Node: Library, Next: Documentation<2>, Up: Python 3 13 7 final
1.23.1.1 Library
................
- gh-137583(1): Fix a deadlock introduced in 3.13.6 when a call to
*note ssl.SSLSocket.recv: da3. was blocked in one thread, and then
another method on the object (such as *note ssl.SSLSocket.send:
da6.) was subsequently called in another thread.
- gh-137044(2): Return large limit values as positive integers
instead of negative integers in *note resource.getrlimit(): 151b.
Accept large values and reject negative values (except *note
RLIM_INFINITY: 151c.) for limits in *note resource.setrlimit():
151d.
- gh-136914(3): Fix retrieval of *note doctest.DocTest.lineno: 151e.
for objects decorated with *note functools.cache(): 151f. or *note
functools.cached_property: 53e.
- gh-131788(4): Make ‘ResourceTracker.send’ from *note
multiprocessing: 94. re-entrant safe
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/137583
(2) https://github.com/python/cpython/issues/137044
(3) https://github.com/python/cpython/issues/136914
(4) https://github.com/python/cpython/issues/131788
File: python.info, Node: Documentation<2>, Next: Core and Builtins, Prev: Library, Up: Python 3 13 7 final
1.23.1.2 Documentation
......................
- gh-136155(1): We are now checking for fatal errors in EPUB builds
in CI.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/136155
File: python.info, Node: Core and Builtins, Prev: Documentation<2>, Up: Python 3 13 7 final
1.23.1.3 Core and Builtins
..........................
- gh-137400(1): Fix a crash in the *note free threading: 1522. build
when disabling profiling or tracing across all threads with *note
PyEval_SetProfileAllThreads(): 55c. or *note
PyEval_SetTraceAllThreads(): 41e. or their Python equivalents *note
threading.settrace_all_threads(): 4c4. and *note
threading.setprofile_all_threads(): 4c5.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/137400
File: python.info, Node: Python 3 13 6 final, Next: Python 3 13 5 final, Prev: Python 3 13 7 final, Up: Changelog
1.23.2 Python 3.13.6 final
--------------------------
'Release date: 2025-08-06'
* Menu:
* macOS::
* Windows::
* Tools/Demos::
* Tests::
* Security::
* Library: Library<2>.
* Documentation: Documentation<3>.
* Core and Builtins: Core and Builtins<2>.
* Build::
File: python.info, Node: macOS, Next: Windows, Up: Python 3 13 6 final
1.23.2.1 macOS
..............
- gh-137450(1): macOS installer shell path management improvements:
separate the installer ‘Shell profile updater’ postinstall script
from the ‘Update Shell Profile.command’ to enable more robust error
handling.
- gh-137134(2): Update macOS installer to ship with SQLite version
3.50.4.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/137450
(2) https://github.com/python/cpython/issues/137134
File: python.info, Node: Windows, Next: Tools/Demos, Prev: macOS, Up: Python 3 13 6 final
1.23.2.2 Windows
................
- gh-137134(1): Update Windows installer to ship with SQLite 3.50.4.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/137134
File: python.info, Node: Tools/Demos, Next: Tests, Prev: Windows, Up: Python 3 13 6 final
1.23.2.3 Tools/Demos
....................
- gh-135968(1): Stubs for ‘strip’ are now provided as part of an iOS
install.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135968
File: python.info, Node: Tests, Next: Security, Prev: Tools/Demos, Up: Python 3 13 6 final
1.23.2.4 Tests
..............
- gh-135966(1): The iOS testbed now handles the ‘app_packages’ folder
as a site directory.
- gh-135494(2): Fix regrtest to support excluding tests from ‘--pgo’
tests. Patch by Victor Stinner.
- gh-135489(3): Show verbose output for failing tests during PGO
profiling step with –enable-optimizations.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135966
(2) https://github.com/python/cpython/issues/135494
(3) https://github.com/python/cpython/issues/135489
File: python.info, Node: Security, Next: Library<2>, Prev: Tests, Up: Python 3 13 6 final
1.23.2.5 Security
.................
- gh-135661(1): Fix parsing start and end tags in *note
html.parser.HTMLParser: 874. according to the HTML5 standard.
* Whitespaces no longer accepted between ‘’ and the tag name.
E.g. ‘ script>’ does not end the script section.
* Vertical tabulation (‘\v’) and non-ASCII whitespaces no longer
recognized as whitespaces. The only whitespaces are
‘\t\n\r\f’ and space.
* Null character (U+0000) no longer ends the tag name.
* Attributes and slashes after the tag name in end tags are now
ignored, instead of terminating after the first ‘>’ in quoted
attribute value. E.g. ‘"/>’.
* Multiple slashes and whitespaces between the last attribute
and closing ‘>’ are now ignored in both start and end tags.
E.g. ‘ ’.
* Multiple ‘=’ between attribute name and value are no longer
collapsed. E.g. ‘’ produces attribute “foo” with
value “=bar”.
- gh-102555(2): Fix comment parsing in *note html.parser.HTMLParser:
874. according to the HTML5 standard. ‘--!>’ now ends the comment.
‘-- >’ no longer ends the comment. Support abnormally ended empty
comments ‘<-->’ and ‘<--->’.
- gh-135462(3): Fix quadratic complexity in processing specially
crafted input in *note html.parser.HTMLParser: 874. End-of-file
errors are now handled according to the HTML5 specs – comments and
declarations are automatically closed, tags are ignored.
- gh-118350(4): Fix support of escapable raw text mode (elements
“textarea” and “title”) in *note html.parser.HTMLParser: 874.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135661
(2) https://github.com/python/cpython/issues/102555
(3) https://github.com/python/cpython/issues/135462
(4) https://github.com/python/cpython/issues/118350
File: python.info, Node: Library<2>, Next: Documentation<3>, Prev: Security, Up: Python 3 13 6 final
1.23.2.6 Library
................
- gh-132710(1): If possible, ensure that *note uuid.getnode(): ba7.
returns the same result even across different processes.
Previously, the result was constant only within the same process.
Patch by Bénédikt Tran.
- gh-137273(2): Fix debug assertion failure in *note
locale.setlocale(): 2df. on Windows.
- gh-137257(3): Bump the version of pip bundled in ensurepip to
version 25.2
- gh-81325(4): *note tarfile.TarFile: 1202. now accepts a *note
path-like: 2a2. when working on a tar archive. (Contributed by
Alexander Enrique Urieles Nieto in gh-81325(5).)
- gh-130522(6): Fix unraisable *note TypeError: 534. raised during
*note interpreter shutdown: 14e. in the *note threading: ed.
module.
- gh-130577(7): *note tarfile: de. now validates archives to ensure
member offsets are non-negative. (Contributed by Alexander Enrique
Urieles Nieto in gh-130577(8).)
- gh-136549(9): Fix signature of *note threading.excepthook(): 847.
- gh-136523(10): Fix *note wave.Wave_write: 2b9. emitting an
unraisable when open raises.
- gh-52876(11): Add missing ‘keepends’ (default ‘True’) parameter to
‘codecs.StreamReaderWriter.readline()’ and
‘codecs.StreamReaderWriter.readlines()’.
- gh-85702(12): If ‘zoneinfo._common.load_tzdata’ is given a package
without a resource a *note zoneinfo.ZoneInfoNotFoundError: 152a. is
raised rather than a *note PermissionError: d42. Patch by Victor
Stinner.
- gh-134759(13): Fix *note UnboundLocalError: 1500. in *note
email.message.Message.get_payload(): 11bf. when the payload to
decode is a *note bytes: 1c2. object. Patch by Kliment Lamonov.
- gh-136028(14): Fix parsing month names containing “İ” (U+0130,
LATIN CAPITAL LETTER I WITH DOT ABOVE) in *note time.strptime():
13f8. This affects locales az_AZ, ber_DZ, ber_MA and crh_UA.
- gh-135995(15): In the palmos encoding, make byte ‘0x9b’ decode to
‘›’ (U+203A - SINGLE RIGHT-POINTING ANGLE QUOTATION MARK).
- gh-53203(16): Fix *note time.strptime(): 13f8. for ‘%c’ and ‘%x’
formats on locales byn_ER, wal_ET and lzh_TW, and for ‘%X’ format
on locales ar_SA, bg_BG and lzh_TW.
- gh-91555(17): An earlier change, which was introduced in 3.13.4,
has been reverted. It disabled logging for a logger during
handling of log messages for that logger. Since the reversion, the
behaviour should be as it was before 3.13.4.
- gh-135878(18): Fixes a crash of *note types.SimpleNamespace: 1d1.
on *note free threading: 1522. builds, when several threads were
calling its *note __repr__(): 618. method at the same time.
- gh-135836(19): Fix *note IndexError: 14ff. in *note
asyncio.loop.create_connection(): 5ff. that could occur when
non-*note OSError: 413. exception is raised during connection and
socket’s ‘close()’ raises ‘OSError’.
- gh-135836(20): Fix *note IndexError: 14ff. in *note
asyncio.loop.create_connection(): 5ff. that could occur when the
Happy Eyeballs algorithm resulted in an empty exceptions list
during connection attempts.
- gh-135855(21): Raise *note TypeError: 534. instead of *note
SystemError: 572. when ‘_interpreters.set___main___attrs()’ is
passed a non-dict object. Patch by Brian Schubert.
- gh-135815(22): *note netrc: 9b.: skip security checks if *note
os.getuid(): 152b. is missing. Patch by Bénédikt Tran.
- gh-135640(23): Address bug where it was possible to call *note
xml.etree.ElementTree.ElementTree.write(): ff0. on an ElementTree
object with an invalid root element. This behavior blanked the
file passed to ‘write’ if it already existed.
- gh-135444(24): Fix *note asyncio.DatagramTransport.sendto(): 1af.
to account for datagram header size when data cannot be sent.
- gh-135497(25): Fix *note os.getlogin(): 152c. failing for longer
usernames on BSD-based platforms.
- gh-135487(26): Fix ‘reprlib.Repr.repr_int()’ when given integers
with more than *note sys.get_int_max_str_digits(): 152d. digits.
Patch by Bénédikt Tran.
- gh-135335(27): *note multiprocessing: 94.: Flush ‘stdout’ and
‘stderr’ after preloading modules in the ‘forkserver’.
- gh-135244(28): *note uuid: 110.: when the MAC address cannot be
determined, the 48-bit node ID is now generated with a
cryptographically-secure pseudo-random number generator (CSPRNG) as
per RFC 9562, §6.10.3(29). This affects *note uuid1(): ba8.
- gh-135069(30): Fix the “Invalid error handling” exception in
‘encodings.idna.IncrementalDecoder’ to correctly replace the
‘errors’ parameter.
- gh-134698(31): Fix a crash when calling methods of *note
ssl.SSLContext: 296. or *note ssl.SSLSocket: 8e9. across multiple
threads.
- gh-132124(32): On POSIX-compliant systems,
‘multiprocessing.util.get_temp_dir()’ now ignores ‘TMPDIR’ (and
similar environment variables) if the path length of ‘AF_UNIX’
socket files exceeds the platform-specific maximum length when
using the 'forkserver' start method. Patch by Bénédikt Tran.
- gh-133439(33): Fix dot commands with trailing spaces are mistaken
for multi-line SQL statements in the sqlite3 command-line
interface.
- gh-132969(34): Prevent the *note ProcessPoolExecutor: 8ed. executor
thread, which remains running when *note shutdown(wait=False): 8ec,
from attempting to adjust the pool’s worker processes after the
object state has already been reset during shutdown. A combination
of conditions, including a worker process having terminated
abormally, resulted in an exception and a potential hang when the
still-running executor thread attempted to replace dead workers
within the pool.
- gh-130664(35): Support the ‘'_'’ digit separator in formatting of
the integral part of *note Decimal: 29f.’s. Patch by Sergey B
Kirpichev.
- gh-85702(36): If ‘zoneinfo._common.load_tzdata’ is given a package
without a resource a ‘ZoneInfoNotFoundError’ is raised rather than
a *note IsADirectoryError: 1052.
- gh-130664(37): Handle corner-case for *note Fraction: 1e9.’s
formatting: treat zero-padding (preceding the width field by a zero
(‘'0'’) character) as an equivalent to a fill character of ‘'0'’
with an alignment type of ‘'='’, just as in case of *note float:
2f1.’s.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/132710
(2) https://github.com/python/cpython/issues/137273
(3) https://github.com/python/cpython/issues/137257
(4) https://github.com/python/cpython/issues/81325
(5) https://github.com/python/cpython/issues/81325
(6) https://github.com/python/cpython/issues/130522
(7) https://github.com/python/cpython/issues/130577
(8) https://github.com/python/cpython/issues/130577
(9) https://github.com/python/cpython/issues/136549
(10) https://github.com/python/cpython/issues/136523
(11) https://github.com/python/cpython/issues/52876
(12) https://github.com/python/cpython/issues/85702
(13) https://github.com/python/cpython/issues/134759
(14) https://github.com/python/cpython/issues/136028
(15) https://github.com/python/cpython/issues/135995
(16) https://github.com/python/cpython/issues/53203
(17) https://github.com/python/cpython/issues/91555
(18) https://github.com/python/cpython/issues/135878
(19) https://github.com/python/cpython/issues/135836
(20) https://github.com/python/cpython/issues/135836
(21) https://github.com/python/cpython/issues/135855
(22) https://github.com/python/cpython/issues/135815
(23) https://github.com/python/cpython/issues/135640
(24) https://github.com/python/cpython/issues/135444
(25) https://github.com/python/cpython/issues/135497
(26) https://github.com/python/cpython/issues/135487
(27) https://github.com/python/cpython/issues/135335
(28) https://github.com/python/cpython/issues/135244
(29)
https://datatracker.ietf.org/doc/html/rfc9562.html#section-6.10-3
(30) https://github.com/python/cpython/issues/135069
(31) https://github.com/python/cpython/issues/134698
(32) https://github.com/python/cpython/issues/132124
(33) https://github.com/python/cpython/issues/133439
(34) https://github.com/python/cpython/issues/132969
(35) https://github.com/python/cpython/issues/130664
(36) https://github.com/python/cpython/issues/85702
(37) https://github.com/python/cpython/issues/130664
File: python.info, Node: Documentation<3>, Next: Core and Builtins<2>, Prev: Library<2>, Up: Python 3 13 6 final
1.23.2.7 Documentation
......................
- gh-135171(1): Document that the *note iterator: 1ac. for the
leftmost ‘for’ clause in the generator expression is created
immediately.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135171
File: python.info, Node: Core and Builtins<2>, Next: Build, Prev: Documentation<3>, Up: Python 3 13 6 final
1.23.2.8 Core and Builtins
..........................
- gh-58124(1): Fix name of the Python encoding in Unicode errors of
the code page codec: use “cp65000” and “cp65001” instead of
“CP_UTF7” and “CP_UTF8” which are not valid Python code names.
Patch by Victor Stinner.
- gh-137314(2): Fixed a regression where raw f-strings incorrectly
interpreted escape sequences in format specifications. Raw
f-strings now properly preserve literal backslashes in format
specs, matching the behavior from Python 3.11. For example,
‘rf"{obj:\xFF}"’ now correctly produces ‘'\\xFF'’ instead of ‘'ÿ'’.
Patch by Pablo Galindo.
- gh-136541(3): Fix some issues with the perf trampolines on x86-64
and aarch64. The trampolines were not being generated correctly
for some cases, which could lead to the perf integration not
working correctly. Patch by Pablo Galindo.
- gh-109700(4): Fix memory error handling in *note
PyDict_SetDefault(): 33b.
- gh-78465(5): Fix error message for ‘cls.__new__(cls, ...)’ where
‘cls’ is not instantiable builtin or extension type (with ‘tp_new’
set to ‘NULL’).
- gh-135871(6): Non-blocking mutex lock attempts now return
immediately when the lock is busy instead of briefly spinning in
the *note free threading: 1522. build.
- gh-135607(7): Fix potential *note weakref: 114. races in an
object’s destructor on the *note free threaded: 1522. build.
- gh-135496(8): Fix typo in the f-string conversion type error
(“exclamanation” -> “exclamation”).
- gh-130077(9): Properly raise custom syntax errors when incorrect
syntax containing names that are prefixes of soft keywords is
encountered. Patch by Pablo Galindo.
- gh-135148(10): Fixed a bug where f-string debug expressions (using
=) would incorrectly strip out parts of strings containing escaped
quotes and # characters. Patch by Pablo Galindo.
- gh-133136(11): Limit excess memory usage in the *note free
threading: 1522. build when a large dictionary or list is resized
and accessed by multiple threads.
- gh-132617(12): Fix *note dict.update(): 142e. modification check
that could incorrectly raise a “dict mutated during update” error
when a different dictionary was modified that happens to share the
same underlying keys object.
- gh-91153(13): Fix a crash when a *note bytearray: 53a. is
concurrently mutated during item assignment.
- gh-127971(14): Fix off-by-one read beyond the end of a string in
string search.
- gh-125723(15): Fix crash with ‘gi_frame.f_locals’ when generator
frames outlive their generator. Patch by Mikhail Efimov.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/58124
(2) https://github.com/python/cpython/issues/137314
(3) https://github.com/python/cpython/issues/136541
(4) https://github.com/python/cpython/issues/109700
(5) https://github.com/python/cpython/issues/78465
(6) https://github.com/python/cpython/issues/135871
(7) https://github.com/python/cpython/issues/135607
(8) https://github.com/python/cpython/issues/135496
(9) https://github.com/python/cpython/issues/130077
(10) https://github.com/python/cpython/issues/135148
(11) https://github.com/python/cpython/issues/133136
(12) https://github.com/python/cpython/issues/132617
(13) https://github.com/python/cpython/issues/91153
(14) https://github.com/python/cpython/issues/127971
(15) https://github.com/python/cpython/issues/125723
File: python.info, Node: Build, Prev: Core and Builtins<2>, Up: Python 3 13 6 final
1.23.2.9 Build
..............
- gh-135497(1): Fix the detection of ‘MAXLOGNAME’ in the
‘configure.ac’ script.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135497
File: python.info, Node: Python 3 13 5 final, Next: Python 3 13 4 final, Prev: Python 3 13 6 final, Up: Changelog
1.23.3 Python 3.13.5 final
--------------------------
'Release date: 2025-06-11'
* Menu:
* Windows: Windows<2>.
* Tests: Tests<2>.
* Library: Library<3>.
* Core and Builtins: Core and Builtins<3>.
* C API::
File: python.info, Node: Windows<2>, Next: Tests<2>, Up: Python 3 13 5 final
1.23.3.1 Windows
................
- gh-135151(1): Avoid distributing modified ‘pyconfig.h’ in the
traditional installer. Extension module builds must always specify
‘Py_GIL_DISABLED’ when targeting the free-threaded runtime.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135151
File: python.info, Node: Tests<2>, Next: Library<3>, Prev: Windows<2>, Up: Python 3 13 5 final
1.23.3.2 Tests
..............
- gh-135120(1): Add ‘test.support.subTests()’.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135120
File: python.info, Node: Library<3>, Next: Core and Builtins<3>, Prev: Tests<2>, Up: Python 3 13 5 final
1.23.3.3 Library
................
- gh-133967(1): Do not normalize *note locale: 86. name ‘C.UTF-8’ to
‘en_US.UTF-8’.
- gh-135326(2): Restore support of integer-like objects with
‘__index__()’ in *note random.getrandbits(): 1006.
- gh-135321(3): Raise a correct exception for values greater than
0x7fffffff for the ‘BINSTRING’ opcode in the C implementation of
*note pickle: a6.
- gh-135276(4): Backported bugfixes in zipfile.Path from zipp 3.23.
Fixed ‘.name’, ‘.stem’ and other basename-based properties on
Windows when working with a zipfile on disk.
- gh-134151(5): *note email: 3b.: Fix *note TypeError: 534. in *note
email.utils.decode_params(): 1535. when sorting RFC 2231(6)
continuations that contain an unnumbered section.
- gh-134152(7): *note email: 3b.: Fix parsing of email message ID
with invalid domain.
- gh-127081(8): Fix libc thread safety issues with *note os: a1. by
replacing ‘getlogin’ with ‘getlogin_r’ re-entrant version.
- gh-131884(9): Fix formatting issues in *note json.dump(): d44. when
both 'indent' and 'skipkeys' are used.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/133967
(2) https://github.com/python/cpython/issues/135326
(3) https://github.com/python/cpython/issues/135321
(4) https://github.com/python/cpython/issues/135276
(5) https://github.com/python/cpython/issues/134151
(6) https://datatracker.ietf.org/doc/html/rfc2231.html
(7) https://github.com/python/cpython/issues/134152
(8) https://github.com/python/cpython/issues/127081
(9) https://github.com/python/cpython/issues/131884
File: python.info, Node: Core and Builtins<3>, Next: C API, Prev: Library<3>, Up: Python 3 13 5 final
1.23.3.4 Core and Builtins
..........................
- gh-135171(1): Roll back changes to generator and list
comprehensions that went into 3.13.4 to fix gh-127682(2), but which
involved semantic and bytecode changes not appropriate for a bugfix
release.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135171
(2) https://github.com/python/cpython/issues/127682
File: python.info, Node: C API, Prev: Core and Builtins<3>, Up: Python 3 13 5 final
1.23.3.5 C API
..............
- gh-134989(1): Fix ‘Py_RETURN_NONE’, ‘Py_RETURN_TRUE’ and
‘Py_RETURN_FALSE’ macros in the limited C API 3.11 and older: don’t
treat ‘Py_None’, ‘Py_True’ and ‘Py_False’ as immortal. Patch by
Victor Stinner.
- gh-134989(2): Implement *note PyObject_DelAttr(): 1538. and *note
PyObject_DelAttrString(): 1539. as macros in the limited C API 3.12
and older. Patch by Victor Stinner.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/134989
(2) https://github.com/python/cpython/issues/134989
File: python.info, Node: Python 3 13 4 final, Next: Python 3 13 3 final, Prev: Python 3 13 5 final, Up: Changelog
1.23.4 Python 3.13.4 final
--------------------------
'Release date: 2025-06-03'
* Menu:
* Windows: Windows<3>.
* Tests: Tests<3>.
* Security: Security<2>.
* Library: Library<4>.
* IDLE: IDLE<3>.
* Documentation: Documentation<4>.
* Core and Builtins: Core and Builtins<4>.
* C API: C API<2>.
* Build: Build<2>.
File: python.info, Node: Windows<3>, Next: Tests<3>, Up: Python 3 13 4 final
1.23.4.1 Windows
................
- gh-130727(1): Fix a race in internal calls into WMI that can result
in an “invalid handle” exception under high load. Patch by Chris
Eibl.
- gh-76023(2): Make *note os.path.realpath(): 227. ignore Windows
error 1005 when in non-strict mode.
- gh-133626(3): Ensures packages are not accidentally bundled into
the traditional installer.
- gh-133512(4): Add warnings to *note Python Launcher for Windows:
5ba. about use of subcommands belonging to the Python install
manager.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/130727
(2) https://github.com/python/cpython/issues/76023
(3) https://github.com/python/cpython/issues/133626
(4) https://github.com/python/cpython/issues/133512
File: python.info, Node: Tests<3>, Next: Security<2>, Prev: Windows<3>, Up: Python 3 13 4 final
1.23.4.2 Tests
..............
- gh-133744(1): Fix multiprocessing interrupt test. Add an event to
synchronize the parent process with the child process: wait until
the child process starts sleeping. Patch by Victor Stinner.
- gh-133639(2): Fix ‘TestPyReplAutoindent.test_auto_indent_default()’
doesn’t run ‘input_code’.
- gh-133131(3): The iOS testbed will now select the most recently
released “SE-class” device for testing if a device isn’t explicitly
specified.
- gh-109981(4): The test helper that counts the list of open file
descriptors now uses the optimised ‘/dev/fd’ approach on all Apple
platforms, not just macOS. This avoids crashes caused by guarded
file descriptors.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/133744
(2) https://github.com/python/cpython/issues/133639
(3) https://github.com/python/cpython/issues/133131
(4) https://github.com/python/cpython/issues/109981
File: python.info, Node: Security<2>, Next: Library<4>, Prev: Tests<3>, Up: Python 3 13 4 final
1.23.4.3 Security
.................
- gh-135034(1): Fixes multiple issues that allowed ‘tarfile’
extraction filters (‘filter="data"’ and ‘filter="tar"’) to be
bypassed using crafted symlinks and hard links.
Addresses CVE 2024-12718(2), CVE 2025-4138(3), CVE 2025-4330(4),
and CVE 2025-4517(5).
- gh-133767(6): Fix use-after-free in the “unicode-escape” decoder
with a non-“strict” error handler.
- gh-128840(7): Short-circuit the processing of long IPv6 addresses
early in *note ipaddress: 80. to prevent excessive memory
consumption and a minor denial-of-service.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/135034
(2) https://www.cve.org/CVERecord?id=CVE-2024-12718
(3) https://www.cve.org/CVERecord?id=CVE-2025-4138
(4) https://www.cve.org/CVERecord?id=CVE-2025-4330
(5) https://www.cve.org/CVERecord?id=CVE-2025-4517
(6) https://github.com/python/cpython/issues/133767
(7) https://github.com/python/cpython/issues/128840
File: python.info, Node: Library<4>, Next: IDLE<3>, Prev: Security<2>, Up: Python 3 13 4 final
1.23.4.4 Library
................
- gh-134718(1): *note ast.dump(): 8e3. now only omits ‘None’ and ‘[]’
values if they are default values.
- gh-128840(2): Fix parsing long IPv6 addresses with embedded IPv4
address.
- gh-134696(3): Built-in HACL* and OpenSSL implementations of hash
function constructors now correctly accept the same 'documented'
named arguments. For instance, *note md5(): 153f. could be
previously invoked as ‘md5(data=data)’ or ‘md5(string=string)’
depending on the underlying implementation but these calls were not
compatible. Patch by Bénédikt Tran.
- gh-134210(4): *note curses.window.getch(): 1540. now correctly
handles signals. Patch by Bénédikt Tran.
- gh-80334(5): *note multiprocessing.freeze_support(): 1541. now
checks for work on any “spawn” start method platform rather than
only on Windows.
- gh-114177(6): Fix *note asyncio: a. to not close subprocess pipes
which would otherwise error out when the event loop is already
closed.
- gh-134152(7): Fixed *note UnboundLocalError: 1500. that could occur
during *note email: 3b. header parsing if an expected trailing
delimiter is missing in some contexts.
- gh-62184(8): Remove import of C implementation of *note io.FileIO:
1303. from Python implementation which has its own implementation
- gh-133982(9): Emit *note RuntimeWarning: 1bd. in the Python
implementation of *note io: 7f. when the *note file-like object:
11b5. is not closed explicitly in the presence of multiple I/O
layers.
- gh-133890(10): The *note tarfile: de. module now handles *note
UnicodeEncodeError: 673. in the same way as *note OSError: 413.
when cannot extract a member.
- gh-134097(11): Fix interaction of the new *note REPL: 16a. and
*note -X showrefcount: 176. command line option.
- gh-133889(12): The generated directory listing page in *note
http.server.SimpleHTTPRequestHandler: b43. now only shows the
decoded path component of the requested URL, and not the query and
fragment.
- gh-134098(13): Fix handling paths that end with a percent-encoded
slash (‘%2f’ or ‘%2F’) in *note
http.server.SimpleHTTPRequestHandler: b43.
- gh-134062(14): *note ipaddress: 80.: fix collisions in *note
__hash__(): afb. for *note IPv4Network: 200. and *note IPv6Network:
201. objects.
- gh-133745(15): In 3.13.3 we accidentally changed the signature of
the asyncio ‘create_task()’ family of methods and how it calls a
custom task factory in a backwards incompatible way. Since some
3rd party libraries have already made changes to work around the
issue that might break if we simply reverted the changes, we’re
instead changing things to be backwards compatible with 3.13.2
while still supporting those workarounds for 3.13.3. In
particular, the special-casing of ‘name’ and ‘context’ is back
(until 3.14) and consequently eager tasks may still find that their
name hasn’t been set before they execute their first yielding
await.
- gh-71253(16): Raise *note ValueError: 204. in *note open(): 517. if
'opener' returns a negative file-descriptor in the Python
implementation of *note io: 7f. to match the C implementation.
- gh-77057(17): Fix handling of invalid markup declarations in *note
html.parser.HTMLParser: 874.
- gh-133489(18): *note random.getrandbits(): 1006. can now generate
more that 2^31 bits. *note random.randbytes(): 1542. can now
generate more that 256 MiB.
- gh-133290(19): Fix attribute caching issue when setting *note
ctypes._Pointer._type_: 1543. in the undocumented and deprecated
‘ctypes.SetPointerType()’ function and the undocumented
‘set_type()’ method.
- gh-132876(20): ‘ldexp()’ on Windows doesn’t round subnormal results
before Windows 11, but should. Python’s *note math.ldexp(): 1544.
wrapper now does round them, so results may change slightly, in
rare cases of very small results, on Windows versions before 11.
- gh-133089(21): Use original timeout value for *note
subprocess.TimeoutExpired: 1545. when the func *note
subprocess.run(): b86. is called with a timeout instead of
sometimes a confusing partial remaining time out value used
internally on the final ‘wait()’.
- gh-133009(22): *note xml.etree.ElementTree: 125.: Fix a crash in
*note Element.__deepcopy__: 1546. when the element is concurrently
mutated. Patch by Bénédikt Tran.
- gh-132995(23): Bump the version of pip bundled in ensurepip to
version 25.1.1
- gh-132017(24): Fix error when ‘pyrepl’ is suspended, then resumed
and terminated.
- gh-132673(25): Fix a crash when using ‘_align_ = 0’ and ‘_fields_ =
[]’ in a *note ctypes.Structure: 1d4.
- gh-132527(26): Include the valid typecode ‘w’ in the error message
when an invalid typecode is passed to *note array.array: 1a0.
- gh-132439(27): Fix ‘PyREPL’ on Windows: characters entered via
AltGr are swallowed. Patch by Chris Eibl.
- gh-132429(28): Fix support of Bluetooth sockets on NetBSD and
DragonFly BSD.
- gh-132106(29): *note QueueListener.start: 1547. now raises a *note
RuntimeError: 195. if the listener is already started.
- gh-132417(30): Fix a ‘NULL’ pointer dereference when a C function
called using *note ctypes: 2a. with ‘restype’ *note py_object:
1403. returns ‘NULL’.
- gh-132385(31): Fix instance error suggestions trigger potential
exceptions in *note object.__getattr__(): 4cf. in *note traceback:
fe.
- gh-132308(32): A *note traceback.TracebackException: 25e. now
correctly renders the ‘__context__’ and ‘__cause__’ attributes from
*note falsey: 1548. *note Exception: 9d9, and the ‘exceptions’
attribute from falsey *note ExceptionGroup: 1b6.
- gh-132250(33): Fixed the *note SystemError: 572. in *note cProfile:
27. when locating the actual C function of a method raises an
exception.
- gh-132063(34): Prevent exceptions that evaluate as falsey (namely,
when their ‘__bool__’ method returns ‘False’ or their ‘__len__’
method returns 0) from being ignored by *note
concurrent.futures.ProcessPoolExecutor: 8ed. and *note
concurrent.futures.ThreadPoolExecutor: 73d.
- gh-119605(35): Respect ‘follow_wrapped’ for ‘__init__()’ and
‘__new__()’ methods when getting the class signature for a class
with *note inspect.signature(): 733. Preserve class signature
after wrapping with *note warnings.deprecated(): 160. Patch by
Xuehai Pan.
- gh-91555(36): Ignore log messages generated during handling of log
messages, to avoid deadlock or infinite recursion. [NOTE: This
change has since been reverted.]
- gh-131434(37): Improve error reporting for incorrect format in
*note time.strptime(): 13f8.
- gh-131127(38): Systems using LibreSSL now successfully build.
- gh-130999(39): Avoid exiting the new REPL and offer suggestions
even if there are non-string candidates when errors occur.
- gh-130941(40): Fix *note configparser.ConfigParser: 1c8. parsing
empty interpolation with ‘allow_no_value’ set to ‘True’.
- gh-129098(41): Fix REPL traceback reporting when using *note
compile(): 192. with an inexisting file. Patch by Bénédikt Tran.
- gh-130631(42): ‘http.cookiejar.join_header_words()’ is now more
similar to the original Perl version. It now quotes the same set
of characters and always quote values that end with ‘"\n"’.
- gh-129719(43): Fix missing ‘socket.CAN_RAW_ERR_FILTER’ constant in
the socket module on Linux systems. It was missing since Python
3.11.
- gh-124096(44): Turn on virtual terminal mode and enable bracketed
paste in REPL on Windows console. (If the terminal does not
support bracketed paste, enabling it does nothing.)
- gh-122559(45): Remove ‘__reduce__()’ and ‘__reduce_ex__()’ methods
that always raise *note TypeError: 534. in the C implementation of
*note io.FileIO: 1303, *note io.BufferedReader: 1200, *note
io.BufferedWriter: 1549. and *note io.BufferedRandom: 154a. and
replace them with default ‘__getstate__()’ methods that raise
‘TypeError’. This restores fine details of behavior of Python 3.11
and older versions.
- gh-122179(46): *note hashlib.file_digest(): 640. now raises *note
BlockingIOError: 1050. when no data is available during
non-blocking I/O. Before, it added spurious null bytes to the
digest.
- gh-86155(47): *note html.parser.HTMLParser.close(): 154b. no longer
loses data when the ‘’ and
‘’).
-- Method: HTMLParser.handle_entityref (name)
This method is called to process a named character reference of the
form ‘&name;’ (e.g. ‘>’), where 'name' is a general entity
reference (e.g. ‘'gt'’). This method is never called if
'convert_charrefs' is ‘True’.
-- Method: HTMLParser.handle_charref (name)
This method is called to process decimal and hexadecimal numeric
character references of the form ‘NNN;’ and ‘NNN;’. For
example, the decimal equivalent for ‘>’ is ‘>’, whereas the
hexadecimal is ‘>’; in this case the method will receive
‘'62'’ or ‘'x3E'’. This method is never called if
'convert_charrefs' is ‘True’.
-- Method: HTMLParser.handle_comment (data)
This method is called when a comment is encountered (e.g.
‘’).
For example, the comment ‘’ will cause this method
to be called with the argument ‘' comment '’.
The content of Internet Explorer conditional comments (condcoms)
will also be sent to this method, so, for ‘’, this method will receive
‘'[if IE 9]>IE9-specific content’).
The 'decl' parameter will be the entire contents of the declaration
inside the ‘’ markup (e.g. ‘'DOCTYPE html'’).
-- Method: HTMLParser.handle_pi (data)
Method called when a processing instruction is encountered. The
'data' parameter will contain the entire processing instruction.
For example, for the processing instruction ‘’,
this method would be called as ‘handle_pi("proc color='red'")’. It
is intended to be overridden by a derived class; the base class
implementation does nothing.
Note: The *note HTMLParser: 874. class uses the SGML syntactic
rules for processing instructions. An XHTML processing
instruction using the trailing ‘'?'’ will cause the ‘'?'’ to
be included in 'data'.
-- Method: HTMLParser.unknown_decl (data)
This method is called when an unrecognized declaration is read by
the parser.
The 'data' parameter will be the entire contents of the declaration
inside the ‘’ markup. It is sometimes useful to be
overridden by a derived class. The base class implementation does
nothing.
File: python.info, Node: Examples<24>, Prev: HTMLParser Methods, Up: html parser — Simple HTML and XHTML parser
5.21.2.3 Examples
.................
The following class implements a parser that will be used to illustrate
more examples:
from html.parser import HTMLParser
from html.entities import name2codepoint
class MyHTMLParser(HTMLParser):
def handle_starttag(self, tag, attrs):
print("Start tag:", tag)
for attr in attrs:
print(" attr:", attr)
def handle_endtag(self, tag):
print("End tag :", tag)
def handle_data(self, data):
print("Data :", data)
def handle_comment(self, data):
print("Comment :", data)
def handle_entityref(self, name):
c = chr(name2codepoint[name])
print("Named ent:", c)
def handle_charref(self, name):
if name.startswith('x'):
c = chr(int(name[1:], 16))
else:
c = chr(int(name))
print("Num ent :", c)
def handle_decl(self, data):
print("Decl :", data)
parser = MyHTMLParser()
Parsing a doctype:
>>> parser.feed('')
Decl : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"
Parsing an element with a few attributes and a title:
>>> parser.feed(' ')
Start tag: img
attr: ('src', 'python-logo.png')
attr: ('alt', 'The Python logo')
>>>
>>> parser.feed('Python ')
Start tag: h1
Data : Python
End tag : h1
The content of ‘script’ and ‘style’ elements is returned as is, without
further parsing:
>>> parser.feed('')
Start tag: style
attr: ('type', 'text/css')
Data : #python { color: green }
End tag : style
>>> parser.feed('')
Start tag: script
attr: ('type', 'text/javascript')
Data : alert("hello! ");
End tag : script
Parsing comments:
>>> parser.feed(''
... '')
Comment : a comment
Comment : [if IE 9]>IE-specific content'’):
>>> parser = MyHTMLParser()
>>> parser.feed('>>>')
Data : >>>
>>> parser = MyHTMLParser(convert_charrefs=False)
>>> parser.feed('>>>')
Named ent: >
Num ent : >
Num ent : >
Feeding incomplete chunks to *note feed(): 375c. works, but *note
handle_data(): 3763. might be called more than once (unless
'convert_charrefs' is set to ‘True’):
>>> for chunk in ['buff', 'ered', ' text']:
... parser.feed(chunk)
...
Start tag: span
Data : buff
Data : ered
Data : text
End tag : span
Parsing invalid HTML (e.g. unquoted attributes) also works:
>>> parser.feed('tag soup
')
Start tag: p
Start tag: a
attr: ('class', 'link')
attr: ('href', '#main')
Data : tag soup
End tag : p
End tag : a
File: python.info, Node: html entities — Definitions of HTML general entities, Next: XML Processing Modules, Prev: html parser — Simple HTML and XHTML parser, Up: Structured Markup Processing Tools
5.21.3 ‘html.entities’ — Definitions of HTML general entities
-------------------------------------------------------------
'Source code:' Lib/html/entities.py(1)
__________________________________________________________________
This module defines four dictionaries, *note html5: 10c8, *note
name2codepoint: 376e, *note codepoint2name: 376f, and *note entitydefs:
3770.
-- Data: html.entities.html5
A dictionary that maps HTML5 named character references (2) to the
equivalent Unicode character(s), e.g. ‘html5['gt;'] == '>'’. Note
that the trailing semicolon is included in the name (e.g.
‘'gt;'’), however some of the names are accepted by the standard
even without the semicolon: in this case the name is present with
and without the ‘';'’. See also *note html.unescape(): 957.
Added in version 3.3.
-- Data: html.entities.entitydefs
A dictionary mapping XHTML 1.0 entity definitions to their
replacement text in ISO Latin-1.
-- Data: html.entities.name2codepoint
A dictionary that maps HTML4 entity names to the Unicode code
points.
-- Data: html.entities.codepoint2name
A dictionary that maps Unicode code points to HTML4 entity names.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/html/entities.py
(2) See
‘https://html.spec.whatwg.org/multipage/named-characters.html#named-character-references’
File: python.info, Node: XML Processing Modules, Next: xml etree ElementTree — The ElementTree XML API, Prev: html entities — Definitions of HTML general entities, Up: Structured Markup Processing Tools
5.21.4 XML Processing Modules
-----------------------------
'Source code:' Lib/xml/(1)
__________________________________________________________________
Python’s interfaces for processing XML are grouped in the ‘xml’ package.
Note: If you need to parse untrusted or unauthenticated data, see
*note XML security: 3774.
It is important to note that modules in the *note xml: 120. package
require that there be at least one SAX-compliant XML parser available.
The Expat parser is included with Python, so the *note
xml.parsers.expat: 126. module will always be available.
The documentation for the *note xml.dom: 121. and *note xml.sax: 129.
packages are the definition of the Python bindings for the DOM and SAX
interfaces.
The XML handling submodules are:
* *note xml.etree.ElementTree: 125.: the ElementTree API, a simple
and lightweight XML processor
* *note xml.dom: 121.: the DOM API definition
* *note xml.dom.minidom: 122.: a minimal DOM implementation
* *note xml.dom.pulldom: 123.: support for building partial DOM trees
* *note xml.sax: 129.: SAX2 base classes and convenience functions
* *note xml.parsers.expat: 126.: the Expat parser binding
* Menu:
* XML security::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/xml/
File: python.info, Node: XML security, Up: XML Processing Modules
5.21.4.1 XML security
.....................
An attacker can abuse XML features to carry out denial of service
attacks, access local files, generate network connections to other
machines, or circumvent firewalls.
Expat versions lower that 2.6.0 may be vulnerable to “billion laughs”,
“quadratic blowup” and “large tokens”. Python may be vulnerable if it
uses such older versions of Expat as a system-provided library. Check
‘pyexpat.EXPAT_VERSION’.
*note xmlrpc: 12d. is 'vulnerable' to the “decompression bomb” attack.
billion laughs / exponential entity expansion
The Billion Laughs(1) attack – also known as exponential entity
expansion – uses multiple levels of nested entities. Each entity
refers to another entity several times, and the final entity
definition contains a small string. The exponential expansion
results in several gigabytes of text and consumes lots of memory
and CPU time.
quadratic blowup entity expansion
A quadratic blowup attack is similar to a Billion Laughs(2) attack;
it abuses entity expansion, too. Instead of nested entities it
repeats one large entity with a couple of thousand chars over and
over again. The attack isn’t as efficient as the exponential case
but it avoids triggering parser countermeasures that forbid deeply
nested entities.
decompression bomb
Decompression bombs (aka ZIP bomb(3)) apply to all XML libraries
that can parse compressed XML streams such as gzipped HTTP streams
or LZMA-compressed files. For an attacker it can reduce the amount
of transmitted data by three magnitudes or more.
large tokens
Expat needs to re-parse unfinished tokens; without the protection
introduced in Expat 2.6.0, this can lead to quadratic runtime that
can be used to cause denial of service in the application parsing
XML. The issue is known as CVE 2023-52425(4).
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/Billion_laughs
(2) https://en.wikipedia.org/wiki/Billion_laughs
(3) https://en.wikipedia.org/wiki/Zip_bomb
(4) https://www.cve.org/CVERecord?id=CVE-2023-52425
File: python.info, Node: xml etree ElementTree — The ElementTree XML API, Next: xml dom — The Document Object Model API, Prev: XML Processing Modules, Up: Structured Markup Processing Tools
5.21.5 ‘xml.etree.ElementTree’ — The ElementTree XML API
--------------------------------------------------------
'Source code:' Lib/xml/etree/ElementTree.py(1)
__________________________________________________________________
The *note xml.etree.ElementTree: 125. module implements a simple and
efficient API for parsing and creating XML data.
Changed in version 3.3: This module will use a fast implementation
whenever available.
Deprecated since version 3.3: The ‘xml.etree.cElementTree’ module is
deprecated.
Note: If you need to parse untrusted or unauthenticated data, see
*note XML security: 3774.
* Menu:
* Tutorial: Tutorial<3>.
* XPath support::
* Reference: Reference<4>.
* XInclude support::
* Reference: Reference<5>.
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/xml/etree/ElementTree.py
File: python.info, Node: Tutorial<3>, Next: XPath support, Up: xml etree ElementTree — The ElementTree XML API
5.21.5.1 Tutorial
.................
This is a short tutorial for using *note xml.etree.ElementTree: 125.
(‘ET’ in short). The goal is to demonstrate some of the building blocks
and basic concepts of the module.
* Menu:
* XML tree and elements::
* Parsing XML::
* Pull API for non-blocking parsing::
* Finding interesting elements::
* Modifying an XML File::
* Building XML documents::
* Parsing XML with Namespaces::
File: python.info, Node: XML tree and elements, Next: Parsing XML, Up: Tutorial<3>
5.21.5.2 XML tree and elements
..............................
XML is an inherently hierarchical data format, and the most natural way
to represent it is with a tree. ‘ET’ has two classes for this purpose -
*note ElementTree: 94e. represents the whole XML document as a tree, and
*note Element: 30a. represents a single node in this tree. Interactions
with the whole document (reading and writing to/from files) are usually
done on the *note ElementTree: 94e. level. Interactions with a single
XML element and its sub-elements are done on the *note Element: 30a.
level.
File: python.info, Node: Parsing XML, Next: Pull API for non-blocking parsing, Prev: XML tree and elements, Up: Tutorial<3>
5.21.5.3 Parsing XML
....................
We’ll be using the fictive ‘country_data.xml’ XML document as the sample
data for this section:
1
2008
141100
4
2011
59900
68
2011
13600
We can import this data by reading from a file:
import xml.etree.ElementTree as ET
tree = ET.parse('country_data.xml')
root = tree.getroot()
Or directly from a string:
root = ET.fromstring(country_data_as_string)
*note fromstring(): 377d. parses XML from a string directly into an
*note Element: 30a, which is the root element of the parsed tree. Other
parsing functions may create an *note ElementTree: 94e. Check the
documentation to be sure.
As an *note Element: 30a, ‘root’ has a tag and a dictionary of
attributes:
>>> root.tag
'data'
>>> root.attrib
{}
It also has children nodes over which we can iterate:
>>> for child in root:
... print(child.tag, child.attrib)
...
country {'name': 'Liechtenstein'}
country {'name': 'Singapore'}
country {'name': 'Panama'}
Children are nested, and we can access specific child nodes by index:
>>> root[0][1].text
'2008'
Note: Not all elements of the XML input will end up as elements of
the parsed tree. Currently, this module skips over any XML
comments, processing instructions, and document type declarations
in the input. Nevertheless, trees built using this module’s API
rather than parsing from XML text can have comments and processing
instructions in them; they will be included when generating XML
output. A document type declaration may be accessed by passing a
custom *note TreeBuilder: a54. instance to the *note XMLParser:
a53. constructor.
File: python.info, Node: Pull API for non-blocking parsing, Next: Finding interesting elements, Prev: Parsing XML, Up: Tutorial<3>
5.21.5.4 Pull API for non-blocking parsing
..........................................
Most parsing functions provided by this module require the whole
document to be read at once before returning any result. It is possible
to use an *note XMLParser: a53. and feed data into it incrementally, but
it is a push API that calls methods on a callback target, which is too
low-level and inconvenient for most needs. Sometimes what the user
really wants is to be able to parse XML incrementally, without blocking
operations, while enjoying the convenience of fully constructed *note
Element: 30a. objects.
The most powerful tool for doing this is *note XMLPullParser: fec. It
does not require a blocking read to obtain the XML data, and is instead
fed with data incrementally with *note XMLPullParser.feed(): 377f.
calls. To get the parsed XML elements, call *note
XMLPullParser.read_events(): 3780. Here is an example:
>>> parser = ET.XMLPullParser(['start', 'end'])
>>> parser.feed('sometext')
>>> list(parser.read_events())
[('start', )]
>>> parser.feed(' more text ')
>>> for event, elem in parser.read_events():
... print(event)
... print(elem.tag, 'text=', elem.text)
...
end
mytag text= sometext more text
The obvious use case is applications that operate in a non-blocking
fashion where the XML data is being received from a socket or read
incrementally from some storage device. In such cases, blocking reads
are unacceptable.
Because it’s so flexible, *note XMLPullParser: fec. can be inconvenient
to use for simpler use-cases. If you don’t mind your application
blocking on reading XML data but would still like to have incremental
parsing capabilities, take a look at *note iterparse(): 275. It can be
useful when you’re reading a large XML document and don’t want to hold
it wholly in memory.
Where 'immediate' feedback through events is wanted, calling method
*note XMLPullParser.flush(): 272. can help reduce delay; please make
sure to study the related security notes.
File: python.info, Node: Finding interesting elements, Next: Modifying an XML File, Prev: Pull API for non-blocking parsing, Up: Tutorial<3>
5.21.5.5 Finding interesting elements
.....................................
*note Element: 30a. has some useful methods that help iterate
recursively over all the sub-tree below it (its children, their
children, and so on). For example, *note Element.iter(): 1342.:
>>> for neighbor in root.iter('neighbor'):
... print(neighbor.attrib)
...
{'name': 'Austria', 'direction': 'E'}
{'name': 'Switzerland', 'direction': 'W'}
{'name': 'Malaysia', 'direction': 'N'}
{'name': 'Costa Rica', 'direction': 'W'}
{'name': 'Colombia', 'direction': 'E'}
*note Element.findall(): 1567. finds only elements with a tag which are
direct children of the current element. *note Element.find(): 1565.
finds the 'first' child with a particular tag, and *note Element.text:
3782. accesses the element’s text content. *note Element.get(): 3783.
accesses the element’s attributes:
>>> for country in root.findall('country'):
... rank = country.find('rank').text
... name = country.get('name')
... print(name, rank)
...
Liechtenstein 1
Singapore 4
Panama 68
More sophisticated specification of which elements to look for is
possible by using *note XPath: bad.
File: python.info, Node: Modifying an XML File, Next: Building XML documents, Prev: Finding interesting elements, Up: Tutorial<3>
5.21.5.6 Modifying an XML File
..............................
*note ElementTree: 94e. provides a simple way to build XML documents and
write them to files. The *note ElementTree.write(): ff0. method serves
this purpose.
Once created, an *note Element: 30a. object may be manipulated by
directly changing its fields (such as *note Element.text: 3782.), adding
and modifying attributes (*note Element.set(): 3785. method), as well as
adding new children (for example with *note Element.append(): 3786.).
Let’s say we want to add one to each country’s rank, and add an
‘updated’ attribute to the rank element:
>>> for rank in root.iter('rank'):
... new_rank = int(rank.text) + 1
... rank.text = str(new_rank)
... rank.set('updated', 'yes')
...
>>> tree.write('output.xml')
Our XML now looks like this:
2
2008
141100
5
2011
59900
69
2011
13600
We can remove elements using *note Element.remove(): 1569. Let’s say we
want to remove all countries with a rank higher than 50:
>>> for country in root.findall('country'):
... # using root.findall() to avoid removal during traversal
... rank = int(country.find('rank').text)
... if rank > 50:
... root.remove(country)
...
>>> tree.write('output.xml')
Note that concurrent modification while iterating can lead to problems,
just like when iterating and modifying Python lists or dicts.
Therefore, the example first collects all matching elements with
‘root.findall()’, and only then iterates over the list of matches.
Our XML now looks like this:
2
2008
141100
5
2011
59900
File: python.info, Node: Building XML documents, Next: Parsing XML with Namespaces, Prev: Modifying an XML File, Up: Tutorial<3>
5.21.5.7 Building XML documents
...............................
The *note SubElement(): 3788. function also provides a convenient way to
create new sub-elements for a given element:
>>> a = ET.Element('a')
>>> b = ET.SubElement(a, 'b')
>>> c = ET.SubElement(a, 'c')
>>> d = ET.SubElement(c, 'd')
>>> ET.dump(a)
File: python.info, Node: Parsing XML with Namespaces, Prev: Building XML documents, Up: Tutorial<3>
5.21.5.8 Parsing XML with Namespaces
....................................
If the XML input has namespaces(1), tags and attributes with prefixes in
the form ‘prefix:sometag’ get expanded to ‘{uri}sometag’ where the
'prefix' is replaced by the full 'URI'. Also, if there is a default
namespace(2), that full URI gets prepended to all of the non-prefixed
tags.
Here is an XML example that incorporates two namespaces, one with the
prefix “fictional” and the other serving as the default namespace:
John Cleese
Lancelot
Archie Leach
Eric Idle
Sir Robin
Gunther
Commander Clement
One way to search and explore this XML example is to manually add the
URI to every tag or attribute in the xpath of a *note find(): 1565. or
*note findall(): 1567.:
root = fromstring(xml_text)
for actor in root.findall('{http://people.example.com}actor'):
name = actor.find('{http://people.example.com}name')
print(name.text)
for char in actor.findall('{http://characters.example.com}character'):
print(' |-->', char.text)
A better way to search the namespaced XML example is to create a
dictionary with your own prefixes and use those in the search functions:
ns = {'real_person': 'http://people.example.com',
'role': 'http://characters.example.com'}
for actor in root.findall('real_person:actor', ns):
name = actor.find('real_person:name', ns)
print(name.text)
for char in actor.findall('role:character', ns):
print(' |-->', char.text)
These two approaches both output:
John Cleese
|--> Lancelot
|--> Archie Leach
Eric Idle
|--> Sir Robin
|--> Gunther
|--> Commander Clement
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/XML_namespace
(2) https://www.w3.org/TR/xml-names/#defaulting
File: python.info, Node: XPath support, Next: Reference<4>, Prev: Tutorial<3>, Up: xml etree ElementTree — The ElementTree XML API
5.21.5.9 XPath support
......................
This module provides limited support for XPath expressions(1) for
locating elements in a tree. The goal is to support a small subset of
the abbreviated syntax; a full XPath engine is outside the scope of the
module.
* Menu:
* Example: Example<8>.
* Supported XPath syntax::
---------- Footnotes ----------
(1) https://www.w3.org/TR/xpath
File: python.info, Node: Example<8>, Next: Supported XPath syntax, Up: XPath support
5.21.5.10 Example
.................
Here’s an example that demonstrates some of the XPath capabilities of
the module. We’ll be using the ‘countrydata’ XML document from the
*note Parsing XML: 377b. section:
import xml.etree.ElementTree as ET
root = ET.fromstring(countrydata)
# Top-level elements
root.findall(".")
# All 'neighbor' grand-children of 'country' children of the top-level
# elements
root.findall("./country/neighbor")
# Nodes with name='Singapore' that have a 'year' child
root.findall(".//year/..[@name='Singapore']")
# 'year' nodes that are children of nodes with name='Singapore'
root.findall(".//*[@name='Singapore']/year")
# All 'neighbor' nodes that are the second child of their parent
root.findall(".//neighbor[2]")
For XML with namespaces, use the usual qualified ‘{namespace}tag’
notation:
# All dublin-core "title" tags in the document
root.findall(".//{http://purl.org/dc/elements/1.1/}title")
File: python.info, Node: Supported XPath syntax, Prev: Example<8>, Up: XPath support
5.21.5.11 Supported XPath syntax
................................
Syntax Meaning
---------------------------------------------------------------------------------------
‘tag’ Selects all child elements with the given tag. For
example, ‘spam’ selects all child elements named ‘spam’,
and ‘spam/egg’ selects all grandchildren named ‘egg’ in
all children named ‘spam’. ‘{namespace}*’ selects all
tags in the given namespace, ‘{*}spam’ selects tags
named ‘spam’ in any (or no) namespace, and ‘{}*’ only
selects tags that are not in a namespace.
Changed in version 3.8: Support for star-wildcards was
added.
‘*’ Selects all child elements, including comments and
processing instructions. For example, ‘*/egg’ selects
all grandchildren named ‘egg’.
‘.’ Selects the current node. This is mostly useful at the
beginning of the path, to indicate that it’s a relative
path.
‘//’ Selects all subelements, on all levels beneath the
current element. For example, ‘.//egg’ selects all
‘egg’ elements in the entire tree.
‘..’ Selects the parent element. Returns ‘None’ if the path
attempts to reach the ancestors of the start element
(the element ‘find’ was called on).
‘[@attrib]’ Selects all elements that have the given attribute.
‘[@attrib='value']’ Selects all elements for which the given attribute has
the given value. The value cannot contain quotes.
‘[@attrib!='value']’ Selects all elements for which the given attribute does
not have the given value. The value cannot contain
quotes.
Added in version 3.10.
‘[tag]’ Selects all elements that have a child named ‘tag’.
Only immediate children are supported.
‘[.='text']’ Selects all elements whose complete text content,
including descendants, equals the given ‘text’.
Added in version 3.7.
‘[.!='text']’ Selects all elements whose complete text content,
including descendants, does not equal the given ‘text’.
Added in version 3.10.
‘[tag='text']’ Selects all elements that have a child named ‘tag’ whose
complete text content, including descendants, equals the
given ‘text’.
‘[tag!='text']’ Selects all elements that have a child named ‘tag’ whose
complete text content, including descendants, does not
equal the given ‘text’.
Added in version 3.10.
‘[position]’ Selects all elements that are located at the given
position. The position can be either an integer (1 is
the first position), the expression ‘last()’ (for the
last position), or a position relative to the last
position (e.g. ‘last()-1’).
Predicates (expressions within square brackets) must be preceded by a
tag name, an asterisk, or another predicate. ‘position’ predicates must
be preceded by a tag name.
File: python.info, Node: Reference<4>, Next: XInclude support, Prev: XPath support, Up: xml etree ElementTree — The ElementTree XML API
5.21.5.12 Reference
...................
* Menu:
* Functions: Functions<9>.
File: python.info, Node: Functions<9>, Up: Reference<4>
5.21.5.13 Functions
...................
-- Function: xml.etree.ElementTree.canonicalize (xml_data=None, *,
out=None, from_file=None, **options)
C14N 2.0(1) transformation function.
Canonicalization is a way to normalise XML output in a way that
allows byte-by-byte comparisons and digital signatures. It reduces
the freedom that XML serializers have and instead generates a more
constrained XML representation. The main restrictions regard the
placement of namespace declarations, the ordering of attributes,
and ignorable whitespace.
This function takes an XML data string ('xml_data') or a file path
or file-like object ('from_file') as input, converts it to the
canonical form, and writes it out using the 'out' file(-like)
object, if provided, or returns it as a text string if not. The
output file receives text, not bytes. It should therefore be
opened in text mode with ‘utf-8’ encoding.
Typical uses:
xml_data = "... "
print(canonicalize(xml_data))
with open("c14n_output.xml", mode='w', encoding='utf-8') as out_file:
canonicalize(xml_data, out=out_file)
with open("c14n_output.xml", mode='w', encoding='utf-8') as out_file:
canonicalize(from_file="inputfile.xml", out=out_file)
The configuration 'options' are as follows:
- 'with_comments': set to true to include comments (default:
false)
-
'strip_text': set to true to strip whitespace before and after text content
(default: false)
-
'rewrite_prefixes': set to true to replace namespace prefixes by “n{number}”
(default: false)
-
'qname_aware_tags': a set of qname aware tag names in which prefixes
should be replaced in text content (default: empty)
-
'qname_aware_attrs': a set of qname aware attribute names in which prefixes
should be replaced in text content (default: empty)
- 'exclude_attrs': a set of attribute names that should not be
serialised
- 'exclude_tags': a set of tag names that should not be
serialised
In the option list above, “a set” refers to any collection or
iterable of strings, no ordering is expected.
Added in version 3.8.
-- Function: xml.etree.ElementTree.Comment (text=None)
Comment element factory. This factory function creates a special
element that will be serialized as an XML comment by the standard
serializer. The comment string can be either a bytestring or a
Unicode string. 'text' is a string containing the comment string.
Returns an element instance representing a comment.
Note that *note XMLParser: a53. skips over comments in the input
instead of creating comment objects for them. An *note
ElementTree: 94e. will only contain comment nodes if they have been
inserted into to the tree using one of the *note Element: 30a.
methods.
-- Function: xml.etree.ElementTree.dump (elem)
Writes an element tree or element structure to sys.stdout. This
function should be used for debugging only.
The exact output format is implementation dependent. In this
version, it’s written as an ordinary XML file.
'elem' is an element tree or an individual element.
Changed in version 3.8: The *note dump(): 3792. function now
preserves the attribute order specified by the user.
-- Function: xml.etree.ElementTree.fromstring (text, parser=None)
Parses an XML section from a string constant. Same as *note XML():
3793. 'text' is a string containing XML data. 'parser' is an
optional parser instance. If not given, the standard *note
XMLParser: a53. parser is used. Returns an *note Element: 30a.
instance.
-- Function: xml.etree.ElementTree.fromstringlist (sequence,
parser=None)
Parses an XML document from a sequence of string fragments.
'sequence' is a list or other sequence containing XML data
fragments. 'parser' is an optional parser instance. If not given,
the standard *note XMLParser: a53. parser is used. Returns an
*note Element: 30a. instance.
Added in version 3.2.
-- Function: xml.etree.ElementTree.indent (tree, space=' ', level=0)
Appends whitespace to the subtree to indent the tree visually.
This can be used to generate pretty-printed XML output. 'tree' can
be an Element or ElementTree. 'space' is the whitespace string
that will be inserted for each indentation level, two space
characters by default. For indenting partial subtrees inside of an
already indented tree, pass the initial indentation level as
'level'.
Added in version 3.9.
-- Function: xml.etree.ElementTree.iselement (element)
Check if an object appears to be a valid element object. 'element'
is an element instance. Return ‘True’ if this is an element
object.
-- Function: xml.etree.ElementTree.iterparse (source, events=None,
parser=None)
Parses an XML section into an element tree incrementally, and
reports what’s going on to the user. 'source' is a filename or
*note file object: 11b5. containing XML data. 'events' is a
sequence of events to report back. The supported events are the
strings ‘"start"’, ‘"end"’, ‘"comment"’, ‘"pi"’, ‘"start-ns"’ and
‘"end-ns"’ (the “ns” events are used to get detailed namespace
information). If 'events' is omitted, only ‘"end"’ events are
reported. 'parser' is an optional parser instance. If not given,
the standard *note XMLParser: a53. parser is used. 'parser' must
be a subclass of *note XMLParser: a53. and can only use the default
*note TreeBuilder: a54. as a target. Returns an *note iterator:
1ac. providing ‘(event, elem)’ pairs; it has a ‘root’ attribute
that references the root element of the resulting XML tree once
'source' is fully read. The iterator has the ‘close()’ method that
closes the internal file object if 'source' is a filename.
Note that while *note iterparse(): 275. builds the tree
incrementally, it issues blocking reads on 'source' (or the file it
names). As such, it’s unsuitable for applications where blocking
reads can’t be made. For fully non-blocking parsing, see *note
XMLPullParser: fec.
Note: *note iterparse(): 275. only guarantees that it has seen
the “>” character of a starting tag when it emits a “start”
event, so the attributes are defined, but the contents of the
text and tail attributes are undefined at that point. The
same applies to the element children; they may or may not be
present.
If you need a fully populated element, look for “end” events
instead.
Deprecated since version 3.4: The 'parser' argument.
Changed in version 3.8: The ‘comment’ and ‘pi’ events were added.
Changed in version 3.13: Added the ‘close()’ method.
-- Function: xml.etree.ElementTree.parse (source, parser=None)
Parses an XML section into an element tree. 'source' is a filename
or file object containing XML data. 'parser' is an optional parser
instance. If not given, the standard *note XMLParser: a53. parser
is used. Returns an *note ElementTree: 94e. instance.
-- Function: xml.etree.ElementTree.ProcessingInstruction (target,
text=None)
PI element factory. This factory function creates a special
element that will be serialized as an XML processing instruction.
'target' is a string containing the PI target. 'text' is a string
containing the PI contents, if given. Returns an element instance,
representing a processing instruction.
Note that *note XMLParser: a53. skips over processing instructions
in the input instead of creating PI objects for them. An *note
ElementTree: 94e. will only contain processing instruction nodes if
they have been inserted into to the tree using one of the *note
Element: 30a. methods.
-- Function: xml.etree.ElementTree.register_namespace (prefix, uri)
Registers a namespace prefix. The registry is global, and any
existing mapping for either the given prefix or the namespace URI
will be removed. 'prefix' is a namespace prefix. 'uri' is a
namespace uri. Tags and attributes in this namespace will be
serialized with the given prefix, if at all possible.
Added in version 3.2.
-- Function: xml.etree.ElementTree.SubElement (parent, tag, attrib={},
**extra)
Subelement factory. This function creates an element instance, and
appends it to an existing element.
The element name, attribute names, and attribute values can be
either bytestrings or Unicode strings. 'parent' is the parent
element. 'tag' is the subelement name. 'attrib' is an optional
dictionary, containing element attributes. 'extra' contains
additional attributes, given as keyword arguments. Returns an
element instance.
-- Function: xml.etree.ElementTree.tostring (element,
encoding='us-ascii', method='xml', *, xml_declaration=None,
default_namespace=None, short_empty_elements=True)
Generates a string representation of an XML element, including all
subelements. 'element' is an *note Element: 30a. instance.
'encoding' (2) is the output encoding (default is US-ASCII). Use
‘encoding="unicode"’ to generate a Unicode string (otherwise, a
bytestring is generated). 'method' is either ‘"xml"’, ‘"html"’ or
‘"text"’ (default is ‘"xml"’). 'xml_declaration',
'default_namespace' and 'short_empty_elements' has the same meaning
as in *note ElementTree.write(): ff0. Returns an (optionally)
encoded string containing the XML data.
Changed in version 3.4: Added the 'short_empty_elements' parameter.
Changed in version 3.8: Added the 'xml_declaration' and
'default_namespace' parameters.
Changed in version 3.8: The *note tostring(): fee. function now
preserves the attribute order specified by the user.
-- Function: xml.etree.ElementTree.tostringlist (element,
encoding='us-ascii', method='xml', *, xml_declaration=None,
default_namespace=None, short_empty_elements=True)
Generates a string representation of an XML element, including all
subelements. 'element' is an *note Element: 30a. instance.
'encoding' (3) is the output encoding (default is US-ASCII). Use
‘encoding="unicode"’ to generate a Unicode string (otherwise, a
bytestring is generated). 'method' is either ‘"xml"’, ‘"html"’ or
‘"text"’ (default is ‘"xml"’). 'xml_declaration',
'default_namespace' and 'short_empty_elements' has the same meaning
as in *note ElementTree.write(): ff0. Returns a list of
(optionally) encoded strings containing the XML data. It does not
guarantee any specific sequence, except that
‘b"".join(tostringlist(element)) == tostring(element)’.
Added in version 3.2.
Changed in version 3.4: Added the 'short_empty_elements' parameter.
Changed in version 3.8: Added the 'xml_declaration' and
'default_namespace' parameters.
Changed in version 3.8: The *note tostringlist(): fef. function now
preserves the attribute order specified by the user.
-- Function: xml.etree.ElementTree.XML (text, parser=None)
Parses an XML section from a string constant. This function can be
used to embed “XML literals” in Python code. 'text' is a string
containing XML data. 'parser' is an optional parser instance. If
not given, the standard *note XMLParser: a53. parser is used.
Returns an *note Element: 30a. instance.
-- Function: xml.etree.ElementTree.XMLID (text, parser=None)
Parses an XML section from a string constant, and also returns a
dictionary which maps from element id:s to elements. 'text' is a
string containing XML data. 'parser' is an optional parser
instance. If not given, the standard *note XMLParser: a53. parser
is used. Returns a tuple containing an *note Element: 30a.
instance and a dictionary.
---------- Footnotes ----------
(1) https://www.w3.org/TR/xml-c14n2/
(2) The encoding string included in XML output should conform to the
appropriate standards. For example, “UTF-8” is valid, but “UTF8” is
not. See
‘https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl’ and
‘https://www.iana.org/assignments/character-sets/character-sets.xhtml’.
(3) The encoding string included in XML output should conform to the
appropriate standards. For example, “UTF-8” is valid, but “UTF8” is
not. See
‘https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl’ and
‘https://www.iana.org/assignments/character-sets/character-sets.xhtml’.
File: python.info, Node: XInclude support, Next: Reference<5>, Prev: Reference<4>, Up: xml etree ElementTree — The ElementTree XML API
5.21.5.14 XInclude support
..........................
This module provides limited support for XInclude directives(1), via the
*note xml.etree.ElementInclude: 124. helper module. This module can be
used to insert subtrees and text strings into element trees, based on
information in the tree.
* Menu:
* Example: Example<9>.
---------- Footnotes ----------
(1) https://www.w3.org/TR/xinclude/
File: python.info, Node: Example<9>, Up: XInclude support
5.21.5.15 Example
.................
Here’s an example that demonstrates use of the XInclude module. To
include an XML document in the current document, use the
‘{http://www.w3.org/2001/XInclude}include’ element and set the 'parse'
attribute to ‘"xml"’, and use the 'href' attribute to specify the
document to include.
By default, the 'href' attribute is treated as a file name. You can use
custom loaders to override this behaviour. Also note that the standard
helper does not support XPointer syntax.
To process this file, load it as usual, and pass the root element to the
*note xml.etree.ElementTree: 125. module:
from xml.etree import ElementTree, ElementInclude
tree = ElementTree.parse("document.xml")
root = tree.getroot()
ElementInclude.include(root)
The ElementInclude module replaces the
‘{http://www.w3.org/2001/XInclude}include’ element with the root element
from the 'source.xml' document. The result might look something like
this:
This is a paragraph.
If the 'parse' attribute is omitted, it defaults to “xml”. The href
attribute is required.
To include a text document, use the
‘{http://www.w3.org/2001/XInclude}include’ element, and set the 'parse'
attribute to “text”:
Copyright (c) .
The result might look something like:
Copyright (c) 2003.
File: python.info, Node: Reference<5>, Prev: XInclude support, Up: xml etree ElementTree — The ElementTree XML API
5.21.5.16 Reference
...................
* Menu:
* Functions: Functions<10>.
* Element Objects::
* ElementTree Objects::
* QName Objects::
* TreeBuilder Objects::
* XMLParser Objects::
* XMLPullParser Objects::
* Exceptions: Exceptions<19>.
File: python.info, Node: Functions<10>, Next: Element Objects, Up: Reference<5>
5.21.5.17 Functions
...................
-- Function: xml.etree.ElementInclude.default_loader (href, parse,
encoding=None)
Default loader. This default loader reads an included resource
from disk. 'href' is a URL. 'parse' is for parse mode either “xml”
or “text”. 'encoding' is an optional text encoding. If not given,
encoding is ‘utf-8’. Returns the expanded resource. If the parse
mode is ‘"xml"’, this is an *note Element: 30a. instance. If the
parse mode is ‘"text"’, this is a string. If the loader fails, it
can return ‘None’ or raise an exception.
-- Function: xml.etree.ElementInclude.include (elem, loader=None,
base_url=None, max_depth=6)
This function expands XInclude directives in-place in tree pointed
by 'elem'. 'elem' is either the root *note Element: 30a. or an
*note ElementTree: 94e. instance to find such element. 'loader' is
an optional resource loader. If omitted, it defaults to *note
default_loader(): 379f. If given, it should be a callable that
implements the same interface as *note default_loader(): 379f.
'base_url' is base URL of the original file, to resolve relative
include file references. 'max_depth' is the maximum number of
recursive inclusions. Limited to reduce the risk of malicious
content explosion. Pass ‘None’ to disable the limitation.
Changed in version 3.9: Added the 'base_url' and 'max_depth'
parameters.
File: python.info, Node: Element Objects, Next: ElementTree Objects, Prev: Functions<10>, Up: Reference<5>
5.21.5.18 Element Objects
.........................
-- Class: xml.etree.ElementTree.Element (tag, attrib={}, **extra)
Element class. This class defines the Element interface, and
provides a reference implementation of this interface.
The element name, attribute names, and attribute values can be
either bytestrings or Unicode strings. 'tag' is the element name.
'attrib' is an optional dictionary, containing element attributes.
'extra' contains additional attributes, given as keyword arguments.
-- Attribute: tag
A string identifying what kind of data this element represents
(the element type, in other words).
-- Attribute: text
-- Attribute: tail
These attributes can be used to hold additional data
associated with the element. Their values are usually strings
but may be any application-specific object. If the element is
created from an XML file, the 'text' attribute holds either
the text between the element’s start tag and its first child
or end tag, or ‘None’, and the 'tail' attribute holds either
the text between the element’s end tag and the next tag, or
‘None’. For the XML data
12 3 4
the 'a' element has ‘None’ for both 'text' and 'tail'
attributes, the 'b' element has 'text' ‘"1"’ and 'tail' ‘"4"’,
the 'c' element has 'text' ‘"2"’ and 'tail' ‘None’, and the
'd' element has 'text' ‘None’ and 'tail' ‘"3"’.
To collect the inner text of an element, see *note itertext():
11c7, for example ‘"".join(element.itertext())’.
Applications may store arbitrary objects in these attributes.
-- Attribute: attrib
A dictionary containing the element’s attributes. Note that
while the 'attrib' value is always a real mutable Python
dictionary, an ElementTree implementation may choose to use
another internal representation, and create the dictionary
only if someone asks for it. To take advantage of such
implementations, use the dictionary methods below whenever
possible.
The following dictionary-like methods work on the element
attributes.
-- Method: clear ()
Resets an element. This function removes all subelements,
clears all attributes, and sets the text and tail attributes
to ‘None’.
-- Method: get (key, default=None)
Gets the element attribute named 'key'.
Returns the attribute value, or 'default' if the attribute was
not found.
-- Method: items ()
Returns the element attributes as a sequence of (name, value)
pairs. The attributes are returned in an arbitrary order.
-- Method: keys ()
Returns the elements attribute names as a list. The names are
returned in an arbitrary order.
-- Method: set (key, value)
Set the attribute 'key' on the element to 'value'.
The following methods work on the element’s children (subelements).
-- Method: append (subelement)
Adds the element 'subelement' to the end of this element’s
internal list of subelements. Raises *note TypeError: 534. if
'subelement' is not an *note Element: 30a.
-- Method: extend (subelements)
Appends 'subelements' from an iterable of elements. Raises
*note TypeError: 534. if a subelement is not an *note Element:
30a.
Added in version 3.2.
-- Method: find (match, namespaces=None)
Finds the first subelement matching 'match'. 'match' may be a
tag name or a *note path: bad. Returns an element instance or
‘None’. 'namespaces' is an optional mapping from namespace
prefix to full name. Pass ‘''’ as prefix to move all
unprefixed tag names in the expression into the given
namespace.
-- Method: findall (match, namespaces=None)
Finds all matching subelements, by tag name or *note path:
bad. Returns a list containing all matching elements in
document order. 'namespaces' is an optional mapping from
namespace prefix to full name. Pass ‘''’ as prefix to move
all unprefixed tag names in the expression into the given
namespace.
-- Method: findtext (match, default=None, namespaces=None)
Finds text for the first subelement matching 'match'. 'match'
may be a tag name or a *note path: bad. Returns the text
content of the first matching element, or 'default' if no
element was found. Note that if the matching element has no
text content an empty string is returned. 'namespaces' is an
optional mapping from namespace prefix to full name. Pass
‘''’ as prefix to move all unprefixed tag names in the
expression into the given namespace.
-- Method: insert (index, subelement)
Inserts 'subelement' at the given position in this element.
Raises *note TypeError: 534. if 'subelement' is not an *note
Element: 30a.
-- Method: iter (tag=None)
Creates a tree *note iterator: 1ac. with the current element
as the root. The iterator iterates over this element and all
elements below it, in document (depth first) order. If 'tag'
is not ‘None’ or ‘'*'’, only elements whose tag equals 'tag'
are returned from the iterator. If the tree structure is
modified during iteration, the result is undefined.
Added in version 3.2.
-- Method: iterfind (match, namespaces=None)
Finds all matching subelements, by tag name or *note path:
bad. Returns an iterable yielding all matching elements in
document order. 'namespaces' is an optional mapping from
namespace prefix to full name.
Added in version 3.2.
-- Method: itertext ()
Creates a text iterator. The iterator loops over this element
and all subelements, in document order, and returns all inner
text.
Added in version 3.2.
-- Method: makeelement (tag, attrib)
Creates a new element object of the same type as this element.
Do not call this method, use the *note SubElement(): 3788.
factory function instead.
-- Method: remove (subelement)
Removes 'subelement' from the element. Unlike the find*
methods this method compares elements based on the instance
identity, not on tag value or contents.
*note Element: 30a. objects also support the following sequence
type methods for working with subelements: *note __delitem__():
12bf, *note __getitem__(): 285, *note __setitem__(): 12be, *note
__len__(): 1f63.
Caution: Elements with no subelements will test as ‘False’. In a
future release of Python, all elements will test as ‘True’
regardless of whether subelements exist. Instead, prefer explicit
‘len(elem)’ or ‘elem is not None’ tests.:
element = root.find('foo')
if not element: # careful!
print("element not found, or element has no subelements")
if element is None:
print("element not found")
Changed in version 3.12: Testing the truth value of an Element
emits *note DeprecationWarning: 1a5.
Prior to Python 3.8, the serialisation order of the XML attributes
of elements was artificially made predictable by sorting the
attributes by their name. Based on the now guaranteed ordering of
dicts, this arbitrary reordering was removed in Python 3.8 to
preserve the order in which attributes were originally parsed or
created by user code.
In general, user code should try not to depend on a specific
ordering of attributes, given that the XML Information Set(1)
explicitly excludes the attribute order from conveying information.
Code should be prepared to deal with any ordering on input. In
cases where deterministic XML output is required, e.g. for
cryptographic signing or test data sets, canonical serialisation is
available with the *note canonicalize(): 3790. function.
In cases where canonical output is not applicable but a specific
attribute order is still desirable on output, code should aim for
creating the attributes directly in the desired order, to avoid
perceptual mismatches for readers of the code. In cases where this
is difficult to achieve, a recipe like the following can be applied
prior to serialisation to enforce an order independently from the
Element creation:
def reorder_attributes(root):
for el in root.iter():
attrib = el.attrib
if len(attrib) > 1:
# adjust attribute order, e.g. by sorting
attribs = sorted(attrib.items())
attrib.clear()
attrib.update(attribs)
---------- Footnotes ----------
(1) https://www.w3.org/TR/xml-infoset/
File: python.info, Node: ElementTree Objects, Next: QName Objects, Prev: Element Objects, Up: Reference<5>
5.21.5.19 ElementTree Objects
.............................
-- Class: xml.etree.ElementTree.ElementTree (element=None, file=None)
ElementTree wrapper class. This class represents an entire element
hierarchy, and adds some extra support for serialization to and
from standard XML.
'element' is the root element. The tree is initialized with the
contents of the XML 'file' if given.
-- Method: _setroot (element)
Replaces the root element for this tree. This discards the
current contents of the tree, and replaces it with the given
element. Use with care. 'element' is an element instance.
-- Method: find (match, namespaces=None)
Same as *note Element.find(): 1565, starting at the root of
the tree.
-- Method: findall (match, namespaces=None)
Same as *note Element.findall(): 1567, starting at the root of
the tree.
-- Method: findtext (match, default=None, namespaces=None)
Same as *note Element.findtext(): 1566, starting at the root
of the tree.
-- Method: getroot ()
Returns the root element for this tree.
-- Method: iter (tag=None)
Creates and returns a tree iterator for the root element. The
iterator loops over all elements in this tree, in section
order. 'tag' is the tag to look for (default is to return all
elements).
-- Method: iterfind (match, namespaces=None)
Same as *note Element.iterfind(): 11c6, starting at the root
of the tree.
Added in version 3.2.
-- Method: parse (source, parser=None)
Loads an external XML section into this element tree.
'source' is a file name or *note file object: 11b5. 'parser'
is an optional parser instance. If not given, the standard
*note XMLParser: a53. parser is used. Returns the section
root element.
-- Method: write (file, encoding='us-ascii', xml_declaration=None,
default_namespace=None, method='xml', *,
short_empty_elements=True)
Writes the element tree to a file, as XML. 'file' is a file
name, or a *note file object: 11b5. opened for writing.
'encoding' (1) is the output encoding (default is US-ASCII).
'xml_declaration' controls if an XML declaration should be
added to the file. Use ‘False’ for never, ‘True’ for always,
‘None’ for only if not US-ASCII or UTF-8 or Unicode (default
is ‘None’). 'default_namespace' sets the default XML
namespace (for “xmlns”). 'method' is either ‘"xml"’, ‘"html"’
or ‘"text"’ (default is ‘"xml"’). The keyword-only
'short_empty_elements' parameter controls the formatting of
elements that contain no content. If ‘True’ (the default),
they are emitted as a single self-closed tag, otherwise they
are emitted as a pair of start/end tags.
The output is either a string (*note str: 447.) or binary
(*note bytes: 1c2.). This is controlled by the 'encoding'
argument. If 'encoding' is ‘"unicode"’, the output is a
string; otherwise, it’s binary. Note that this may conflict
with the type of 'file' if it’s an open *note file object:
11b5.; make sure you do not try to write a string to a binary
stream and vice versa.
Changed in version 3.4: Added the 'short_empty_elements'
parameter.
Changed in version 3.8: The *note write(): ff0. method now
preserves the attribute order specified by the user.
This is the XML file that is going to be manipulated:
Example page
Moved to example.org
or example.com .
Example of changing the attribute “target” of every link in first
paragraph:
>>> from xml.etree.ElementTree import ElementTree
>>> tree = ElementTree()
>>> tree.parse("index.xhtml")
>>> p = tree.find("body/p") # Finds first occurrence of tag p in body
>>> p
>>> links = list(p.iter("a")) # Returns list of all links
>>> links
[, ]
>>> for i in links: # Iterates through all found links
... i.attrib["target"] = "blank"
...
>>> tree.write("output.xhtml")
---------- Footnotes ----------
(1) The encoding string included in XML output should conform to the
appropriate standards. For example, “UTF-8” is valid, but “UTF8” is
not. See
‘https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl’ and
‘https://www.iana.org/assignments/character-sets/character-sets.xhtml’.
File: python.info, Node: QName Objects, Next: TreeBuilder Objects, Prev: ElementTree Objects, Up: Reference<5>
5.21.5.20 QName Objects
.......................
-- Class: xml.etree.ElementTree.QName (text_or_uri, tag=None)
QName wrapper. This can be used to wrap a QName attribute value,
in order to get proper namespace handling on output. 'text_or_uri'
is a string containing the QName value, in the form {uri}local, or,
if the tag argument is given, the URI part of a QName. If 'tag' is
given, the first argument is interpreted as a URI, and this
argument is interpreted as a local name. *note QName: 37b6.
instances are opaque.
File: python.info, Node: TreeBuilder Objects, Next: XMLParser Objects, Prev: QName Objects, Up: Reference<5>
5.21.5.21 TreeBuilder Objects
.............................
-- Class: xml.etree.ElementTree.TreeBuilder (element_factory=None, *,
comment_factory=None, pi_factory=None, insert_comments=False,
insert_pis=False)
Generic element structure builder. This builder converts a
sequence of start, data, end, comment and pi method calls to a
well-formed element structure. You can use this class to build an
element structure using a custom XML parser, or a parser for some
other XML-like format.
'element_factory', when given, must be a callable accepting two
positional arguments: a tag and a dict of attributes. It is
expected to return a new element instance.
The 'comment_factory' and 'pi_factory' functions, when given,
should behave like the *note Comment(): 3791. and *note
ProcessingInstruction(): 3797. functions to create comments and
processing instructions. When not given, the default factories
will be used. When 'insert_comments' and/or 'insert_pis' is true,
comments/pis will be inserted into the tree if they appear within
the root element (but not outside of it).
-- Method: close ()
Flushes the builder buffers, and returns the toplevel document
element. Returns an *note Element: 30a. instance.
-- Method: data (data)
Adds text to the current element. 'data' is a string. This
should be either a bytestring, or a Unicode string.
-- Method: end (tag)
Closes the current element. 'tag' is the element name.
Returns the closed element.
-- Method: start (tag, attrs)
Opens a new element. 'tag' is the element name. 'attrs' is a
dictionary containing element attributes. Returns the opened
element.
-- Method: comment (text)
Creates a comment with the given 'text'. If ‘insert_comments’
is true, this will also add it to the tree.
Added in version 3.8.
-- Method: pi (target, text)
Creates a process instruction with the given 'target' name and
'text'. If ‘insert_pis’ is true, this will also add it to the
tree.
Added in version 3.8.
In addition, a custom *note TreeBuilder: a54. object can provide
the following methods:
-- Method: doctype (name, pubid, system)
Handles a doctype declaration. 'name' is the doctype name.
'pubid' is the public identifier. 'system' is the system
identifier. This method does not exist on the default *note
TreeBuilder: a54. class.
Added in version 3.2.
-- Method: start_ns (prefix, uri)
Is called whenever the parser encounters a new namespace
declaration, before the ‘start()’ callback for the opening
element that defines it. 'prefix' is ‘''’ for the default
namespace and the declared namespace prefix name otherwise.
'uri' is the namespace URI.
Added in version 3.8.
-- Method: end_ns (prefix)
Is called after the ‘end()’ callback of an element that
declared a namespace prefix mapping, with the name of the
'prefix' that went out of scope.
Added in version 3.8.
-- Class: xml.etree.ElementTree.C14NWriterTarget (write, *,
with_comments=False, strip_text=False, rewrite_prefixes=False,
qname_aware_tags=None, qname_aware_attrs=None,
exclude_attrs=None, exclude_tags=None)
A C14N 2.0(1) writer. Arguments are the same as for the *note
canonicalize(): 3790. function. This class does not build a tree
but translates the callback events directly into a serialised form
using the 'write' function.
Added in version 3.8.
---------- Footnotes ----------
(1) https://www.w3.org/TR/xml-c14n2/
File: python.info, Node: XMLParser Objects, Next: XMLPullParser Objects, Prev: TreeBuilder Objects, Up: Reference<5>
5.21.5.22 XMLParser Objects
...........................
-- Class: xml.etree.ElementTree.XMLParser (*, target=None,
encoding=None)
This class is the low-level building block of the module. It uses
*note xml.parsers.expat: 126. for efficient, event-based parsing of
XML. It can be fed XML data incrementally with the *note feed():
37c2. method, and parsing events are translated to a push API - by
invoking callbacks on the 'target' object. If 'target' is omitted,
the standard *note TreeBuilder: a54. is used. If 'encoding' (1) is
given, the value overrides the encoding specified in the XML file.
Changed in version 3.8: Parameters are now *note keyword-only: 2a7.
The 'html' argument is no longer supported.
-- Method: close ()
Finishes feeding data to the parser. Returns the result of
calling the ‘close()’ method of the 'target' passed during
construction; by default, this is the toplevel document
element.
-- Method: feed (data)
Feeds data to the parser. 'data' is encoded data.
-- Method: flush ()
Triggers parsing of any previously fed unparsed data, which
can be used to ensure more immediate feedback, in particular
with Expat >=2.6.0. The implementation of *note flush(): 271.
temporarily disables reparse deferral with Expat (if currently
enabled) and triggers a reparse. Disabling reparse deferral
has security consequences; please see *note
xml.parsers.expat.xmlparser.SetReparseDeferralEnabled(): 274.
for details.
Note that *note flush(): 271. has been backported to some
prior releases of CPython as a security fix. Check for
availability of *note flush(): 271. using *note hasattr():
4ce. if used in code running across a variety of Python
versions.
Added in version 3.13.
*note XMLParser.feed(): 37c2. calls 'target"s ‘start(tag,
attrs_dict)’ method for each opening tag, its ‘end(tag)’ method for
each closing tag, and data is processed by method ‘data(data)’.
For further supported callback methods, see the *note TreeBuilder:
a54. class. *note XMLParser.close(): 37c3. calls 'target"s method
‘close()’. *note XMLParser: a53. can be used not only for building
a tree structure. This is an example of counting the maximum depth
of an XML file:
>>> from xml.etree.ElementTree import XMLParser
>>> class MaxDepth: # The target object of the parser
... maxDepth = 0
... depth = 0
... def start(self, tag, attrib): # Called for each opening tag.
... self.depth += 1
... if self.depth > self.maxDepth:
... self.maxDepth = self.depth
... def end(self, tag): # Called for each closing tag.
... self.depth -= 1
... def data(self, data):
... pass # We do not need to do anything with data.
... def close(self): # Called when all data has been parsed.
... return self.maxDepth
...
>>> target = MaxDepth()
>>> parser = XMLParser(target=target)
>>> exampleXml = """
...
...
...
...
...
...
...
...
...
... """
>>> parser.feed(exampleXml)
>>> parser.close()
4
---------- Footnotes ----------
(1) The encoding string included in XML output should conform to the
appropriate standards. For example, “UTF-8” is valid, but “UTF8” is
not. See
‘https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl’ and
‘https://www.iana.org/assignments/character-sets/character-sets.xhtml’.
File: python.info, Node: XMLPullParser Objects, Next: Exceptions<19>, Prev: XMLParser Objects, Up: Reference<5>
5.21.5.23 XMLPullParser Objects
...............................
-- Class: xml.etree.ElementTree.XMLPullParser (events=None)
A pull parser suitable for non-blocking applications. Its
input-side API is similar to that of *note XMLParser: a53, but
instead of pushing calls to a callback target, *note XMLPullParser:
fec. collects an internal list of parsing events and lets the user
read from it. 'events' is a sequence of events to report back.
The supported events are the strings ‘"start"’, ‘"end"’,
‘"comment"’, ‘"pi"’, ‘"start-ns"’ and ‘"end-ns"’ (the “ns” events
are used to get detailed namespace information). If 'events' is
omitted, only ‘"end"’ events are reported.
-- Method: feed (data)
Feed the given bytes data to the parser.
-- Method: flush ()
Triggers parsing of any previously fed unparsed data, which
can be used to ensure more immediate feedback, in particular
with Expat >=2.6.0. The implementation of *note flush(): 272.
temporarily disables reparse deferral with Expat (if currently
enabled) and triggers a reparse. Disabling reparse deferral
has security consequences; please see *note
xml.parsers.expat.xmlparser.SetReparseDeferralEnabled(): 274.
for details.
Note that *note flush(): 272. has been backported to some
prior releases of CPython as a security fix. Check for
availability of *note flush(): 272. using *note hasattr():
4ce. if used in code running across a variety of Python
versions.
Added in version 3.13.
-- Method: close ()
Signal the parser that the data stream is terminated. Unlike
*note XMLParser.close(): 37c3, this method always returns
*note None: 671. Any events not yet retrieved when the parser
is closed can still be read with *note read_events(): 3780.
-- Method: read_events ()
Return an iterator over the events which have been encountered
in the data fed to the parser. The iterator yields ‘(event,
elem)’ pairs, where 'event' is a string representing the type
of event (e.g. ‘"end"’) and 'elem' is the encountered *note
Element: 30a. object, or other context value as follows.
* ‘start’, ‘end’: the current Element.
* ‘comment’, ‘pi’: the current comment / processing
instruction
* ‘start-ns’: a tuple ‘(prefix, uri)’ naming the declared
namespace mapping.
* ‘end-ns’: *note None: 671. (this may change in a future
version)
Events provided in a previous call to *note read_events():
3780. will not be yielded again. Events are consumed from the
internal queue only when they are retrieved from the iterator,
so multiple readers iterating in parallel over iterators
obtained from *note read_events(): 3780. will have
unpredictable results.
Note: *note XMLPullParser: fec. only guarantees that it has
seen the “>” character of a starting tag when it emits a
“start” event, so the attributes are defined, but the contents
of the text and tail attributes are undefined at that point.
The same applies to the element children; they may or may not
be present.
If you need a fully populated element, look for “end” events
instead.
Added in version 3.4.
Changed in version 3.8: The ‘comment’ and ‘pi’ events were added.
File: python.info, Node: Exceptions<19>, Prev: XMLPullParser Objects, Up: Reference<5>
5.21.5.24 Exceptions
....................
-- Class: xml.etree.ElementTree.ParseError
XML parse error, raised by the various parsing methods in this
module when parsing fails. The string representation of an
instance of this exception will contain a user-friendly error
message. In addition, it will have the following attributes
available:
-- Attribute: code
A numeric error code from the expat parser. See the
documentation of *note xml.parsers.expat: 126. for the list of
error codes and their meanings.
-- Attribute: position
A tuple of 'line', 'column' numbers, specifying where the
error occurred.
File: python.info, Node: xml dom — The Document Object Model API, Next: xml dom minidom — Minimal DOM implementation, Prev: xml etree ElementTree — The ElementTree XML API, Up: Structured Markup Processing Tools
5.21.6 ‘xml.dom’ — The Document Object Model API
------------------------------------------------
'Source code:' Lib/xml/dom/__init__.py(1)
__________________________________________________________________
The Document Object Model, or “DOM,” is a cross-language API from the
World Wide Web Consortium (W3C) for accessing and modifying XML
documents. A DOM implementation presents an XML document as a tree
structure, or allows client code to build such a structure from scratch.
It then gives access to the structure through a set of objects which
provided well-known interfaces.
The DOM is extremely useful for random-access applications. SAX only
allows you a view of one bit of the document at a time. If you are
looking at one SAX element, you have no access to another. If you are
looking at a text node, you have no access to a containing element.
When you write a SAX application, you need to keep track of your
program’s position in the document somewhere in your own code. SAX does
not do it for you. Also, if you need to look ahead in the XML document,
you are just out of luck.
Some applications are simply impossible in an event driven model with no
access to a tree. Of course you could build some sort of tree yourself
in SAX events, but the DOM allows you to avoid writing that code. The
DOM is a standard tree representation for XML data.
The Document Object Model is being defined by the W3C in stages, or
“levels” in their terminology. The Python mapping of the API is
substantially based on the DOM Level 2 recommendation.
DOM applications typically start by parsing some XML into a DOM. How
this is accomplished is not covered at all by DOM Level 1, and Level 2
provides only limited improvements: There is a ‘DOMImplementation’
object class which provides access to ‘Document’ creation methods, but
no way to access an XML reader/parser/Document builder in an
implementation-independent way. There is also no well-defined way to
access these methods without an existing ‘Document’ object. In Python,
each DOM implementation will provide a function *note
getDOMImplementation(): 37cc. DOM Level 3 adds a Load/Store
specification, which defines an interface to the reader, but this is not
yet available in the Python standard library.
Once you have a DOM document object, you can access the parts of your
XML document through its properties and methods. These properties are
defined in the DOM specification; this portion of the reference manual
describes the interpretation of the specification in Python.
The specification provided by the W3C defines the DOM API for Java,
ECMAScript, and OMG IDL. The Python mapping defined here is based in
large part on the IDL version of the specification, but strict
compliance is not required (though implementations are free to support
the strict mapping from IDL). See section *note Conformance: 37cd. for a
detailed discussion of mapping requirements.
See also
........
Document Object Model (DOM) Level 2 Specification(2)
The W3C recommendation upon which the Python DOM API is based.
Document Object Model (DOM) Level 1 Specification(3)
The W3C recommendation for the DOM supported by *note
xml.dom.minidom: 122.
Python Language Mapping Specification(4)
This specifies the mapping from OMG IDL to Python.
* Menu:
* Module Contents: Module Contents<4>.
* Objects in the DOM::
* Conformance::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/xml/dom/__init__.py
(2) https://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/
(3) https://www.w3.org/TR/REC-DOM-Level-1/
(4) https://www.omg.org/spec/PYTH/1.2/PDF
File: python.info, Node: Module Contents<4>, Next: Objects in the DOM, Up: xml dom — The Document Object Model API
5.21.6.1 Module Contents
........................
The *note xml.dom: 121. contains the following functions:
-- Function: xml.dom.registerDOMImplementation (name, factory)
Register the 'factory' function with the name 'name'. The factory
function should return an object which implements the
‘DOMImplementation’ interface. The factory function can return the
same object every time, or a new one for each call, as appropriate
for the specific implementation (e.g. if that implementation
supports some customization).
-- Function: xml.dom.getDOMImplementation (name=None, features=())
Return a suitable DOM implementation. The 'name' is either
well-known, the module name of a DOM implementation, or ‘None’. If
it is not ‘None’, imports the corresponding module and returns a
‘DOMImplementation’ object if the import succeeds. If no name is
given, and if the environment variable ‘PYTHON_DOM’ is set, this
variable is used to find the implementation.
If name is not given, this examines the available implementations
to find one with the required feature set. If no implementation
can be found, raise an *note ImportError: 415. The features list
must be a sequence of ‘(feature, version)’ pairs which are passed
to the ‘hasFeature()’ method on available ‘DOMImplementation’
objects.
Some convenience constants are also provided:
-- Data: xml.dom.EMPTY_NAMESPACE
The value used to indicate that no namespace is associated with a
node in the DOM. This is typically found as the ‘namespaceURI’ of a
node, or used as the 'namespaceURI' parameter to a
namespaces-specific method.
-- Data: xml.dom.XML_NAMESPACE
The namespace URI associated with the reserved prefix ‘xml’, as
defined by Namespaces in XML(1) (section 4).
-- Data: xml.dom.XMLNS_NAMESPACE
The namespace URI for namespace declarations, as defined by
Document Object Model (DOM) Level 2 Core Specification(2) (section
1.1.8).
-- Data: xml.dom.XHTML_NAMESPACE
The URI of the XHTML namespace as defined by XHTML 1.0: The
Extensible HyperText Markup Language(3) (section 3.1.1).
In addition, *note xml.dom: 121. contains a base ‘Node’ class and the
DOM exception classes. The ‘Node’ class provided by this module does
not implement any of the methods or attributes defined by the DOM
specification; concrete DOM implementations must provide those. The
‘Node’ class provided as part of this module does provide the constants
used for the ‘nodeType’ attribute on concrete ‘Node’ objects; they are
located within the class rather than at the module level to conform with
the DOM specifications.
---------- Footnotes ----------
(1) https://www.w3.org/TR/REC-xml-names/
(2) https://www.w3.org/TR/DOM-Level-2-Core/core.html
(3) https://www.w3.org/TR/xhtml1/
File: python.info, Node: Objects in the DOM, Next: Conformance, Prev: Module Contents<4>, Up: xml dom — The Document Object Model API
5.21.6.2 Objects in the DOM
...........................
The definitive documentation for the DOM is the DOM specification from
the W3C.
Note that DOM attributes may also be manipulated as nodes instead of as
simple strings. It is fairly rare that you must do this, however, so
this usage is not yet documented.
Interface Section Purpose
-------------------------------------------------------------------------------------------------------------------
‘DOMImplementation’ *note DOMImplementation Objects: 37d6. Interface to the underlying
implementation.
‘Node’ *note Node Objects: 37d7. Base interface for most objects in
a document.
‘NodeList’ *note NodeList Objects: 37d8. Interface for a sequence of nodes.
‘DocumentType’ *note DocumentType Objects: 37d9. Information about the declarations
needed to process a document.
‘Document’ *note Document Objects: 37da. Object which represents an entire
document.
‘Element’ *note Element Objects: 37db. Element nodes in the document
hierarchy.
‘Attr’ *note Attr Objects: 37dc. Attribute value nodes on element
nodes.
‘Comment’ *note Comment Objects: 37dd. Representation of comments in the
source document.
‘Text’ *note Text and CDATASection Objects: 37de.Nodes containing textual content
from the document.
‘ProcessingInstruction’ *note ProcessingInstruction Objects: 37df.Processing instruction
representation.
An additional section describes the exceptions defined for working with
the DOM in Python.
* Menu:
* DOMImplementation Objects::
* Node Objects::
* NodeList Objects::
* DocumentType Objects::
* Document Objects::
* Element Objects: Element Objects<2>.
* Attr Objects::
* NamedNodeMap Objects::
* Comment Objects::
* Text and CDATASection Objects::
* ProcessingInstruction Objects::
* Exceptions: Exceptions<20>.
File: python.info, Node: DOMImplementation Objects, Next: Node Objects, Up: Objects in the DOM
5.21.6.3 DOMImplementation Objects
..................................
The ‘DOMImplementation’ interface provides a way for applications to
determine the availability of particular features in the DOM they are
using. DOM Level 2 added the ability to create new ‘Document’ and
‘DocumentType’ objects using the ‘DOMImplementation’ as well.
-- Method: DOMImplementation.hasFeature (feature, version)
Return ‘True’ if the feature identified by the pair of strings
'feature' and 'version' is implemented.
-- Method: DOMImplementation.createDocument (namespaceUri,
qualifiedName, doctype)
Return a new ‘Document’ object (the root of the DOM), with a child
‘Element’ object having the given 'namespaceUri' and
'qualifiedName'. The 'doctype' must be a ‘DocumentType’ object
created by *note createDocumentType(): 37e3, or ‘None’. In the
Python DOM API, the first two arguments can also be ‘None’ in order
to indicate that no ‘Element’ child is to be created.
-- Method: DOMImplementation.createDocumentType (qualifiedName,
publicId, systemId)
Return a new ‘DocumentType’ object that encapsulates the given
'qualifiedName', 'publicId', and 'systemId' strings, representing
the information contained in an XML document type declaration.
File: python.info, Node: Node Objects, Next: NodeList Objects, Prev: DOMImplementation Objects, Up: Objects in the DOM
5.21.6.4 Node Objects
.....................
All of the components of an XML document are subclasses of ‘Node’.
-- Attribute: Node.nodeType
An integer representing the node type. Symbolic constants for the
types are on the ‘Node’ object: ‘ELEMENT_NODE’, ‘ATTRIBUTE_NODE’,
‘TEXT_NODE’, ‘CDATA_SECTION_NODE’, ‘ENTITY_NODE’,
‘PROCESSING_INSTRUCTION_NODE’, ‘COMMENT_NODE’, ‘DOCUMENT_NODE’,
‘DOCUMENT_TYPE_NODE’, ‘NOTATION_NODE’. This is a read-only
attribute.
-- Attribute: Node.parentNode
The parent of the current node, or ‘None’ for the document node.
The value is always a ‘Node’ object or ‘None’. For ‘Element’
nodes, this will be the parent element, except for the root
element, in which case it will be the ‘Document’ object. For
‘Attr’ nodes, this is always ‘None’. This is a read-only
attribute.
-- Attribute: Node.attributes
A ‘NamedNodeMap’ of attribute objects. Only elements have actual
values for this; others provide ‘None’ for this attribute. This is
a read-only attribute.
-- Attribute: Node.previousSibling
The node that immediately precedes this one with the same parent.
For instance the element with an end-tag that comes just before the
'self' element’s start-tag. Of course, XML documents are made up
of more than just elements so the previous sibling could be text, a
comment, or something else. If this node is the first child of the
parent, this attribute will be ‘None’. This is a read-only
attribute.
-- Attribute: Node.nextSibling
The node that immediately follows this one with the same parent.
See also *note previousSibling: 37e8. If this is the last child of
the parent, this attribute will be ‘None’. This is a read-only
attribute.
-- Attribute: Node.childNodes
A list of nodes contained within this node. This is a read-only
attribute.
-- Attribute: Node.firstChild
The first child of the node, if there are any, or ‘None’. This is
a read-only attribute.
-- Attribute: Node.lastChild
The last child of the node, if there are any, or ‘None’. This is a
read-only attribute.
-- Attribute: Node.localName
The part of the ‘tagName’ following the colon if there is one, else
the entire ‘tagName’. The value is a string.
-- Attribute: Node.prefix
The part of the ‘tagName’ preceding the colon if there is one, else
the empty string. The value is a string, or ‘None’.
-- Attribute: Node.namespaceURI
The namespace associated with the element name. This will be a
string or ‘None’. This is a read-only attribute.
-- Attribute: Node.nodeName
This has a different meaning for each node type; see the DOM
specification for details. You can always get the information you
would get here from another property such as the ‘tagName’ property
for elements or the ‘name’ property for attributes. For all node
types, the value of this attribute will be either a string or
‘None’. This is a read-only attribute.
-- Attribute: Node.nodeValue
This has a different meaning for each node type; see the DOM
specification for details. The situation is similar to that with
*note nodeName: 37f0. The value is a string or ‘None’.
-- Method: Node.hasAttributes ()
Return ‘True’ if the node has any attributes.
-- Method: Node.hasChildNodes ()
Return ‘True’ if the node has any child nodes.
-- Method: Node.isSameNode (other)
Return ‘True’ if 'other' refers to the same node as this node.
This is especially useful for DOM implementations which use any
sort of proxy architecture (because more than one object can refer
to the same node).
Note: This is based on a proposed DOM Level 3 API which is
still in the “working draft” stage, but this particular
interface appears uncontroversial. Changes from the W3C will
not necessarily affect this method in the Python DOM interface
(though any new W3C API for this would also be supported).
-- Method: Node.appendChild (newChild)
Add a new child node to this node at the end of the list of
children, returning 'newChild'. If the node was already in the
tree, it is removed first.
-- Method: Node.insertBefore (newChild, refChild)
Insert a new child node before an existing child. It must be the
case that 'refChild' is a child of this node; if not, *note
ValueError: 204. is raised. 'newChild' is returned. If 'refChild'
is ‘None’, it inserts 'newChild' at the end of the children’s list.
-- Method: Node.removeChild (oldChild)
Remove a child node. 'oldChild' must be a child of this node; if
not, *note ValueError: 204. is raised. 'oldChild' is returned on
success. If 'oldChild' will not be used further, its ‘unlink()’
method should be called.
-- Method: Node.replaceChild (newChild, oldChild)
Replace an existing node with a new node. It must be the case that
'oldChild' is a child of this node; if not, *note ValueError: 204.
is raised.
-- Method: Node.normalize ()
Join adjacent text nodes so that all stretches of text are stored
as single ‘Text’ instances. This simplifies processing text from a
DOM tree for many applications.
-- Method: Node.cloneNode (deep)
Clone this node. Setting 'deep' means to clone all child nodes as
well. This returns the clone.
File: python.info, Node: NodeList Objects, Next: DocumentType Objects, Prev: Node Objects, Up: Objects in the DOM
5.21.6.5 NodeList Objects
.........................
A ‘NodeList’ represents a sequence of nodes. These objects are used in
two ways in the DOM Core recommendation: an ‘Element’ object provides
one as its list of child nodes, and the ‘getElementsByTagName()’ and
‘getElementsByTagNameNS()’ methods of ‘Node’ return objects with this
interface to represent query results.
The DOM Level 2 recommendation defines one method and one attribute for
these objects:
-- Method: NodeList.item (i)
Return the 'i'’th item from the sequence, if there is one, or
‘None’. The index 'i' is not allowed to be less than zero or
greater than or equal to the length of the sequence.
-- Attribute: NodeList.length
The number of nodes in the sequence.
In addition, the Python DOM interface requires that some additional
support is provided to allow ‘NodeList’ objects to be used as Python
sequences. All ‘NodeList’ implementations must include support for
*note __len__(): 1f63. and *note __getitem__(): 285.; this allows
iteration over the ‘NodeList’ in *note for: 2ec. statements and proper
support for the *note len(): 62a. built-in function.
If a DOM implementation supports modification of the document, the
‘NodeList’ implementation must also support the *note __setitem__():
12be. and *note __delitem__(): 12bf. methods.
File: python.info, Node: DocumentType Objects, Next: Document Objects, Prev: NodeList Objects, Up: Objects in the DOM
5.21.6.6 DocumentType Objects
.............................
Information about the notations and entities declared by a document
(including the external subset if the parser uses it and can provide the
information) is available from a ‘DocumentType’ object. The
‘DocumentType’ for a document is available from the ‘Document’ object’s
‘doctype’ attribute; if there is no ‘DOCTYPE’ declaration for the
document, the document’s ‘doctype’ attribute will be set to ‘None’
instead of an instance of this interface.
‘DocumentType’ is a specialization of ‘Node’, and adds the following
attributes:
-- Attribute: DocumentType.publicId
The public identifier for the external subset of the document type
definition. This will be a string or ‘None’.
-- Attribute: DocumentType.systemId
The system identifier for the external subset of the document type
definition. This will be a URI as a string, or ‘None’.
-- Attribute: DocumentType.internalSubset
A string giving the complete internal subset from the document.
This does not include the brackets which enclose the subset. If
the document has no internal subset, this should be ‘None’.
-- Attribute: DocumentType.name
The name of the root element as given in the ‘DOCTYPE’ declaration,
if present.
-- Attribute: DocumentType.entities
This is a ‘NamedNodeMap’ giving the definitions of external
entities. For entity names defined more than once, only the first
definition is provided (others are ignored as required by the XML
recommendation). This may be ‘None’ if the information is not
provided by the parser, or if no entities are defined.
-- Attribute: DocumentType.notations
This is a ‘NamedNodeMap’ giving the definitions of notations. For
notation names defined more than once, only the first definition is
provided (others are ignored as required by the XML
recommendation). This may be ‘None’ if the information is not
provided by the parser, or if no notations are defined.
File: python.info, Node: Document Objects, Next: Element Objects<2>, Prev: DocumentType Objects, Up: Objects in the DOM
5.21.6.7 Document Objects
.........................
A ‘Document’ represents an entire XML document, including its
constituent elements, attributes, processing instructions, comments etc.
Remember that it inherits properties from ‘Node’.
-- Attribute: Document.documentElement
The one and only root element of the document.
-- Method: Document.createElement (tagName)
Create and return a new element node. The element is not inserted
into the document when it is created. You need to explicitly
insert it with one of the other methods such as ‘insertBefore()’ or
‘appendChild()’.
-- Method: Document.createElementNS (namespaceURI, tagName)
Create and return a new element with a namespace. The 'tagName'
may have a prefix. The element is not inserted into the document
when it is created. You need to explicitly insert it with one of
the other methods such as ‘insertBefore()’ or ‘appendChild()’.
-- Method: Document.createTextNode (data)
Create and return a text node containing the data passed as a
parameter. As with the other creation methods, this one does not
insert the node into the tree.
-- Method: Document.createComment (data)
Create and return a comment node containing the data passed as a
parameter. As with the other creation methods, this one does not
insert the node into the tree.
-- Method: Document.createProcessingInstruction (target, data)
Create and return a processing instruction node containing the
'target' and 'data' passed as parameters. As with the other
creation methods, this one does not insert the node into the tree.
-- Method: Document.createAttribute (name)
Create and return an attribute node. This method does not
associate the attribute node with any particular element. You must
use ‘setAttributeNode()’ on the appropriate ‘Element’ object to use
the newly created attribute instance.
-- Method: Document.createAttributeNS (namespaceURI, qualifiedName)
Create and return an attribute node with a namespace. The
'tagName' may have a prefix. This method does not associate the
attribute node with any particular element. You must use
‘setAttributeNode()’ on the appropriate ‘Element’ object to use the
newly created attribute instance.
-- Method: Document.getElementsByTagName (tagName)
Search for all descendants (direct children, children’s children,
etc.) with a particular element type name.
-- Method: Document.getElementsByTagNameNS (namespaceURI, localName)
Search for all descendants (direct children, children’s children,
etc.) with a particular namespace URI and localname. The
localname is the part of the namespace after the prefix.
File: python.info, Node: Element Objects<2>, Next: Attr Objects, Prev: Document Objects, Up: Objects in the DOM
5.21.6.8 Element Objects
........................
‘Element’ is a subclass of ‘Node’, so inherits all the attributes of
that class.
-- Attribute: Element.tagName
The element type name. In a namespace-using document it may have
colons in it. The value is a string.
-- Method: Element.getElementsByTagName (tagName)
Same as equivalent method in the ‘Document’ class.
-- Method: Element.getElementsByTagNameNS (namespaceURI, localName)
Same as equivalent method in the ‘Document’ class.
-- Method: Element.hasAttribute (name)
Return ‘True’ if the element has an attribute named by 'name'.
-- Method: Element.hasAttributeNS (namespaceURI, localName)
Return ‘True’ if the element has an attribute named by
'namespaceURI' and 'localName'.
-- Method: Element.getAttribute (name)
Return the value of the attribute named by 'name' as a string. If
no such attribute exists, an empty string is returned, as if the
attribute had no value.
-- Method: Element.getAttributeNode (attrname)
Return the ‘Attr’ node for the attribute named by 'attrname'.
-- Method: Element.getAttributeNS (namespaceURI, localName)
Return the value of the attribute named by 'namespaceURI' and
'localName' as a string. If no such attribute exists, an empty
string is returned, as if the attribute had no value.
-- Method: Element.getAttributeNodeNS (namespaceURI, localName)
Return an attribute value as a node, given a 'namespaceURI' and
'localName'.
-- Method: Element.removeAttribute (name)
Remove an attribute by name. If there is no matching attribute, a
*note NotFoundErr: 381b. is raised.
-- Method: Element.removeAttributeNode (oldAttr)
Remove and return 'oldAttr' from the attribute list, if present.
If 'oldAttr' is not present, *note NotFoundErr: 381b. is raised.
-- Method: Element.removeAttributeNS (namespaceURI, localName)
Remove an attribute by name. Note that it uses a localName, not a
qname. No exception is raised if there is no matching attribute.
-- Method: Element.setAttribute (name, value)
Set an attribute value from a string.
-- Method: Element.setAttributeNode (newAttr)
Add a new attribute node to the element, replacing an existing
attribute if necessary if the ‘name’ attribute matches. If a
replacement occurs, the old attribute node will be returned. If
'newAttr' is already in use, *note InuseAttributeErr: 3820. will be
raised.
-- Method: Element.setAttributeNodeNS (newAttr)
Add a new attribute node to the element, replacing an existing
attribute if necessary if the ‘namespaceURI’ and ‘localName’
attributes match. If a replacement occurs, the old attribute node
will be returned. If 'newAttr' is already in use, *note
InuseAttributeErr: 3820. will be raised.
-- Method: Element.setAttributeNS (namespaceURI, qname, value)
Set an attribute value from a string, given a 'namespaceURI' and a
'qname'. Note that a qname is the whole attribute name. This is
different than above.
File: python.info, Node: Attr Objects, Next: NamedNodeMap Objects, Prev: Element Objects<2>, Up: Objects in the DOM
5.21.6.9 Attr Objects
.....................
‘Attr’ inherits from ‘Node’, so inherits all its attributes.
-- Attribute: Attr.name
The attribute name. In a namespace-using document it may include a
colon.
-- Attribute: Attr.localName
The part of the name following the colon if there is one, else the
entire name. This is a read-only attribute.
-- Attribute: Attr.prefix
The part of the name preceding the colon if there is one, else the
empty string.
-- Attribute: Attr.value
The text value of the attribute. This is a synonym for the
‘nodeValue’ attribute.
File: python.info, Node: NamedNodeMap Objects, Next: Comment Objects, Prev: Attr Objects, Up: Objects in the DOM
5.21.6.10 NamedNodeMap Objects
..............................
‘NamedNodeMap’ does 'not' inherit from ‘Node’.
-- Attribute: NamedNodeMap.length
The length of the attribute list.
-- Method: NamedNodeMap.item (index)
Return an attribute with a particular index. The order you get the
attributes in is arbitrary but will be consistent for the life of a
DOM. Each item is an attribute node. Get its value with the
‘value’ attribute.
There are also experimental methods that give this class more mapping
behavior. You can use them or you can use the standardized
‘getAttribute*()’ family of methods on the ‘Element’ objects.
File: python.info, Node: Comment Objects, Next: Text and CDATASection Objects, Prev: NamedNodeMap Objects, Up: Objects in the DOM
5.21.6.11 Comment Objects
.........................
‘Comment’ represents a comment in the XML document. It is a subclass of
‘Node’, but cannot have child nodes.
-- Attribute: Comment.data
The content of the comment as a string. The attribute contains all
characters between the leading ‘’, but
does not include them.
File: python.info, Node: Text and CDATASection Objects, Next: ProcessingInstruction Objects, Prev: Comment Objects, Up: Objects in the DOM
5.21.6.12 Text and CDATASection Objects
.......................................
The ‘Text’ interface represents text in the XML document. If the parser
and DOM implementation support the DOM’s XML extension, portions of the
text enclosed in CDATA marked sections are stored in ‘CDATASection’
objects. These two interfaces are identical, but provide different
values for the ‘nodeType’ attribute.
These interfaces extend the ‘Node’ interface. They cannot have child
nodes.
-- Attribute: Text.data
The content of the text node as a string.
Note: The use of a ‘CDATASection’ node does not indicate that the
node represents a complete CDATA marked section, only that the
content of the node was part of a CDATA section. A single CDATA
section may be represented by more than one node in the document
tree. There is no way to determine whether two adjacent
‘CDATASection’ nodes represent different CDATA marked sections.
File: python.info, Node: ProcessingInstruction Objects, Next: Exceptions<20>, Prev: Text and CDATASection Objects, Up: Objects in the DOM
5.21.6.13 ProcessingInstruction Objects
.......................................
Represents a processing instruction in the XML document; this inherits
from the ‘Node’ interface and cannot have child nodes.
-- Attribute: ProcessingInstruction.target
The content of the processing instruction up to the first
whitespace character. This is a read-only attribute.
-- Attribute: ProcessingInstruction.data
The content of the processing instruction following the first
whitespace character.
File: python.info, Node: Exceptions<20>, Prev: ProcessingInstruction Objects, Up: Objects in the DOM
5.21.6.14 Exceptions
....................
The DOM Level 2 recommendation defines a single exception, *note
DOMException: 3835, and a number of constants that allow applications to
determine what sort of error occurred. *note DOMException: 3835.
instances carry a *note code: 1a. attribute that provides the
appropriate value for the specific exception.
The Python DOM interface provides the constants, but also expands the
set of exceptions so that a specific exception exists for each of the
exception codes defined by the DOM. The implementations must raise the
appropriate specific exception, each of which carries the appropriate
value for the *note code: 1a. attribute.
-- Exception: xml.dom.DOMException
Base exception class used for all specific DOM exceptions. This
exception class cannot be directly instantiated.
-- Exception: xml.dom.DomstringSizeErr
Raised when a specified range of text does not fit into a string.
This is not known to be used in the Python DOM implementations, but
may be received from DOM implementations not written in Python.
-- Exception: xml.dom.HierarchyRequestErr
Raised when an attempt is made to insert a node where the node type
is not allowed.
-- Exception: xml.dom.IndexSizeErr
Raised when an index or size parameter to a method is negative or
exceeds the allowed values.
-- Exception: xml.dom.InuseAttributeErr
Raised when an attempt is made to insert an ‘Attr’ node that is
already present elsewhere in the document.
-- Exception: xml.dom.InvalidAccessErr
Raised if a parameter or an operation is not supported on the
underlying object.
-- Exception: xml.dom.InvalidCharacterErr
This exception is raised when a string parameter contains a
character that is not permitted in the context it’s being used in
by the XML 1.0 recommendation. For example, attempting to create
an ‘Element’ node with a space in the element type name will cause
this error to be raised.
-- Exception: xml.dom.InvalidModificationErr
Raised when an attempt is made to modify the type of a node.
-- Exception: xml.dom.InvalidStateErr
Raised when an attempt is made to use an object that is not defined
or is no longer usable.
-- Exception: xml.dom.NamespaceErr
If an attempt is made to change any object in a way that is not
permitted with regard to the Namespaces in XML(1) recommendation,
this exception is raised.
-- Exception: xml.dom.NotFoundErr
Exception when a node does not exist in the referenced context.
For example, ‘NamedNodeMap.removeNamedItem()’ will raise this if
the node passed in does not exist in the map.
-- Exception: xml.dom.NotSupportedErr
Raised when the implementation does not support the requested type
of object or operation.
-- Exception: xml.dom.NoDataAllowedErr
This is raised if data is specified for a node which does not
support data.
-- Exception: xml.dom.NoModificationAllowedErr
Raised on attempts to modify an object where modifications are not
allowed (such as for read-only nodes).
-- Exception: xml.dom.SyntaxErr
Raised when an invalid or illegal string is specified.
-- Exception: xml.dom.WrongDocumentErr
Raised when a node is inserted in a different document than it
currently belongs to, and the implementation does not support
migrating the node from one document to the other.
The exception codes defined in the DOM recommendation map to the
exceptions described above according to this table:
Constant Exception
---------------------------------------------------------------------------------
‘DOMSTRING_SIZE_ERR’ *note DomstringSizeErr: 3836.
‘HIERARCHY_REQUEST_ERR’ *note HierarchyRequestErr: 3837.
‘INDEX_SIZE_ERR’ *note IndexSizeErr: 3838.
‘INUSE_ATTRIBUTE_ERR’ *note InuseAttributeErr: 3820.
‘INVALID_ACCESS_ERR’ *note InvalidAccessErr: 3839.
‘INVALID_CHARACTER_ERR’ *note InvalidCharacterErr: 383a.
‘INVALID_MODIFICATION_ERR’ *note InvalidModificationErr: 383b.
‘INVALID_STATE_ERR’ *note InvalidStateErr: 383c.
‘NAMESPACE_ERR’ *note NamespaceErr: 383d.
‘NOT_FOUND_ERR’ *note NotFoundErr: 381b.
‘NOT_SUPPORTED_ERR’ *note NotSupportedErr: 383e.
‘NO_DATA_ALLOWED_ERR’ *note NoDataAllowedErr: 383f.
‘NO_MODIFICATION_ALLOWED_ERR’ *note NoModificationAllowedErr: 3840.
‘SYNTAX_ERR’ *note SyntaxErr: 3841.
‘WRONG_DOCUMENT_ERR’ *note WrongDocumentErr: 3842.
---------- Footnotes ----------
(1) https://www.w3.org/TR/REC-xml-names/
File: python.info, Node: Conformance, Prev: Objects in the DOM, Up: xml dom — The Document Object Model API
5.21.6.15 Conformance
.....................
This section describes the conformance requirements and relationships
between the Python DOM API, the W3C DOM recommendations, and the OMG IDL
mapping for Python.
* Menu:
* Type Mapping::
* Accessor Methods::
File: python.info, Node: Type Mapping, Next: Accessor Methods, Up: Conformance
5.21.6.16 Type Mapping
......................
The IDL types used in the DOM specification are mapped to Python types
according to the following table.
IDL Type Python Type
-----------------------------------------------------------------------
‘boolean’ ‘bool’ or ‘int’
‘int’ ‘int’
‘long int’ ‘int’
‘unsigned int’ ‘int’
‘DOMString’ ‘str’ or ‘bytes’
‘null’ ‘None’
File: python.info, Node: Accessor Methods, Prev: Type Mapping, Up: Conformance
5.21.6.17 Accessor Methods
..........................
The mapping from OMG IDL to Python defines accessor functions for IDL
‘attribute’ declarations in much the way the Java mapping does. Mapping
the IDL declarations
readonly attribute string someValue;
attribute string anotherValue;
yields three accessor functions: a “get” method for ‘someValue’
(‘_get_someValue()’), and “get” and “set” methods for ‘anotherValue’
(‘_get_anotherValue()’ and ‘_set_anotherValue()’). The mapping, in
particular, does not require that the IDL attributes are accessible as
normal Python attributes: ‘object.someValue’ is 'not' required to work,
and may raise an *note AttributeError: 348.
The Python DOM API, however, 'does' require that normal attribute access
work. This means that the typical surrogates generated by Python IDL
compilers are not likely to work, and wrapper objects may be needed on
the client if the DOM objects are accessed via CORBA. While this does
require some additional consideration for CORBA DOM clients, the
implementers with experience using DOM over CORBA from Python do not
consider this a problem. Attributes that are declared ‘readonly’ may
not restrict write access in all DOM implementations.
In the Python DOM API, accessor functions are not required. If
provided, they should take the form defined by the Python IDL mapping,
but these methods are considered unnecessary since the attributes are
accessible directly from Python. “Set” accessors should never be
provided for ‘readonly’ attributes.
The IDL definitions do not fully embody the requirements of the W3C DOM
API, such as the notion of certain objects, such as the return value of
‘getElementsByTagName()’, being “live”. The Python DOM API does not
require implementations to enforce such requirements.
File: python.info, Node: xml dom minidom — Minimal DOM implementation, Next: xml dom pulldom — Support for building partial DOM trees, Prev: xml dom — The Document Object Model API, Up: Structured Markup Processing Tools
5.21.7 ‘xml.dom.minidom’ — Minimal DOM implementation
-----------------------------------------------------
'Source code:' Lib/xml/dom/minidom.py(1)
__________________________________________________________________
*note xml.dom.minidom: 122. is a minimal implementation of the Document
Object Model interface, with an API similar to that in other languages.
It is intended to be simpler than the full DOM and also significantly
smaller. Users who are not already proficient with the DOM should
consider using the *note xml.etree.ElementTree: 125. module for their
XML processing instead.
Note: If you need to parse untrusted or unauthenticated data, see
*note XML security: 3774.
DOM applications typically start by parsing some XML into a DOM. With
*note xml.dom.minidom: 122, this is done through the parse functions:
from xml.dom.minidom import parse, parseString
dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name
datasource = open('c:\\temp\\mydata.xml')
dom2 = parse(datasource) # parse an open file
dom3 = parseString('Some data some more data ')
The *note parse(): 384a. function can take either a filename or an open
file object.
-- Function: xml.dom.minidom.parse (filename_or_file, parser=None,
bufsize=None)
Return a ‘Document’ from the given input. 'filename_or_file' may
be either a file name, or a file-like object. 'parser', if given,
must be a SAX2 parser object. This function will change the
document handler of the parser and activate namespace support;
other parser configuration (like setting an entity resolver) must
have been done in advance.
If you have XML in a string, you can use the *note parseString(): 384b.
function instead:
-- Function: xml.dom.minidom.parseString (string, parser=None)
Return a ‘Document’ that represents the 'string'. This method
creates an *note io.StringIO: f22. object for the string and passes
that on to *note parse(): 384a.
Both functions return a ‘Document’ object representing the content of
the document.
What the *note parse(): 384a. and *note parseString(): 384b. functions
do is connect an XML parser with a “DOM builder” that can accept parse
events from any SAX parser and convert them into a DOM tree. The name
of the functions are perhaps misleading, but are easy to grasp when
learning the interfaces. The parsing of the document will be completed
before these functions return; it’s simply that these functions do not
provide a parser implementation themselves.
You can also create a ‘Document’ by calling a method on a “DOM
Implementation” object. You can get this object either by calling the
‘getDOMImplementation()’ function in the *note xml.dom: 121. package or
the *note xml.dom.minidom: 122. module. Once you have a ‘Document’, you
can add child nodes to it to populate the DOM:
from xml.dom.minidom import getDOMImplementation
impl = getDOMImplementation()
newdoc = impl.createDocument(None, "some_tag", None)
top_element = newdoc.documentElement
text = newdoc.createTextNode('Some textual content.')
top_element.appendChild(text)
Once you have a DOM document object, you can access the parts of your
XML document through its properties and methods. These properties are
defined in the DOM specification. The main property of the document
object is the ‘documentElement’ property. It gives you the main element
in the XML document: the one that holds all others. Here is an example
program:
dom3 = parseString("Some data ")
assert dom3.documentElement.tagName == "myxml"
When you are finished with a DOM tree, you may optionally call the
‘unlink()’ method to encourage early cleanup of the now-unneeded
objects. ‘unlink()’ is an *note xml.dom.minidom: 122.-specific
extension to the DOM API that renders the node and its descendants
essentially useless. Otherwise, Python’s garbage collector will
eventually take care of the objects in the tree.
See also
........
Document Object Model (DOM) Level 1 Specification(2)
The W3C recommendation for the DOM supported by *note
xml.dom.minidom: 122.
* Menu:
* DOM Objects::
* DOM Example::
* minidom and the DOM standard::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/xml/dom/minidom.py
(2) https://www.w3.org/TR/REC-DOM-Level-1/
File: python.info, Node: DOM Objects, Next: DOM Example, Up: xml dom minidom — Minimal DOM implementation
5.21.7.1 DOM Objects
....................
The definition of the DOM API for Python is given as part of the *note
xml.dom: 121. module documentation. This section lists the differences
between the API and *note xml.dom.minidom: 122.
-- Method: Node.unlink ()
Break internal references within the DOM so that it will be garbage
collected on versions of Python without cyclic GC. Even when cyclic
GC is available, using this can make large amounts of memory
available sooner, so calling this on DOM objects as soon as they
are no longer needed is good practice. This only needs to be
called on the ‘Document’ object, but may be called on child nodes
to discard children of that node.
You can avoid calling this method explicitly by using the *note
with: 5ce. statement. The following code will automatically unlink
'dom' when the ‘with’ block is exited:
with xml.dom.minidom.parse(datasource) as dom:
... # Work with dom.
-- Method: Node.writexml (writer, indent='', addindent='', newl='',
encoding=None, standalone=None)
Write XML to the writer object. The writer receives texts but not
bytes as input, it should have a ‘write()’ method which matches
that of the file object interface. The 'indent' parameter is the
indentation of the current node. The 'addindent' parameter is the
incremental indentation to use for subnodes of the current one.
The 'newl' parameter specifies the string to use to terminate
newlines.
For the ‘Document’ node, an additional keyword argument 'encoding'
can be used to specify the encoding field of the XML header.
Similarly, explicitly stating the 'standalone' argument causes the
standalone document declarations to be added to the prologue of the
XML document. If the value is set to ‘True’, ‘standalone="yes"’ is
added, otherwise it is set to ‘"no"’. Not stating the argument
will omit the declaration from the document.
Changed in version 3.8: The *note writexml(): 384f. method now
preserves the attribute order specified by the user.
Changed in version 3.9: The 'standalone' parameter was added.
-- Method: Node.toxml (encoding=None, standalone=None)
Return a string or byte string containing the XML represented by
the DOM node.
With an explicit 'encoding' (1) argument, the result is a byte
string in the specified encoding. With no 'encoding' argument, the
result is a Unicode string, and the XML declaration in the
resulting string does not specify an encoding. Encoding this
string in an encoding other than UTF-8 is likely incorrect, since
UTF-8 is the default encoding of XML.
The 'standalone' argument behaves exactly as in *note writexml():
384f.
Changed in version 3.8: The *note toxml(): 1492. method now
preserves the attribute order specified by the user.
Changed in version 3.9: The 'standalone' parameter was added.
-- Method: Node.toprettyxml (indent='\t', newl='\n', encoding=None,
standalone=None)
Return a pretty-printed version of the document. 'indent'
specifies the indentation string and defaults to a tabulator;
'newl' specifies the string emitted at the end of each line and
defaults to ‘\n’.
The 'encoding' argument behaves like the corresponding argument of
*note toxml(): 1492.
The 'standalone' argument behaves exactly as in *note writexml():
384f.
Changed in version 3.8: The *note toprettyxml(): 1493. method now
preserves the attribute order specified by the user.
Changed in version 3.9: The 'standalone' parameter was added.
---------- Footnotes ----------
(1) The encoding name included in the XML output should conform to
the appropriate standards. For example, “UTF-8” is valid, but “UTF8” is
not valid in an XML document’s declaration, even though Python accepts
it as an encoding name. See
‘https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl’ and
‘https://www.iana.org/assignments/character-sets/character-sets.xhtml’.
File: python.info, Node: DOM Example, Next: minidom and the DOM standard, Prev: DOM Objects, Up: xml dom minidom — Minimal DOM implementation
5.21.7.2 DOM Example
....................
This example program is a fairly realistic example of a simple program.
In this particular case, we do not take much advantage of the
flexibility of the DOM.
import xml.dom.minidom
document = """\
Demo slideshow
Slide title
This is a demo
Of a program for processing slides
Another demo slide
It is important
To have more than
one slide
"""
dom = xml.dom.minidom.parseString(document)
def getText(nodelist):
rc = []
for node in nodelist:
if node.nodeType == node.TEXT_NODE:
rc.append(node.data)
return ''.join(rc)
def handleSlideshow(slideshow):
print("")
handleSlideshowTitle(slideshow.getElementsByTagName("title")[0])
slides = slideshow.getElementsByTagName("slide")
handleToc(slides)
handleSlides(slides)
print("")
def handleSlides(slides):
for slide in slides:
handleSlide(slide)
def handleSlide(slide):
handleSlideTitle(slide.getElementsByTagName("title")[0])
handlePoints(slide.getElementsByTagName("point"))
def handleSlideshowTitle(title):
print(f"{getText(title.childNodes)} ")
def handleSlideTitle(title):
print(f"{getText(title.childNodes)} ")
def handlePoints(points):
print("")
for point in points:
handlePoint(point)
print(" ")
def handlePoint(point):
print(f"{getText(point.childNodes)} ")
def handleToc(slides):
for slide in slides:
title = slide.getElementsByTagName("title")[0]
print(f"{getText(title.childNodes)}
")
handleSlideshow(dom)
File: python.info, Node: minidom and the DOM standard, Prev: DOM Example, Up: xml dom minidom — Minimal DOM implementation
5.21.7.3 minidom and the DOM standard
.....................................
The *note xml.dom.minidom: 122. module is essentially a DOM
1.0-compatible DOM with some DOM 2 features (primarily namespace
features).
Usage of the DOM interface in Python is straight-forward. The following
mapping rules apply:
* Interfaces are accessed through instance objects. Applications
should not instantiate the classes themselves; they should use the
creator functions available on the ‘Document’ object. Derived
interfaces support all operations (and attributes) from the base
interfaces, plus any new operations.
* Operations are used as methods. Since the DOM uses only *note in:
2ee. parameters, the arguments are passed in normal order (from
left to right). There are no optional arguments. ‘void’
operations return ‘None’.
* IDL attributes map to instance attributes. For compatibility with
the OMG IDL language mapping for Python, an attribute ‘foo’ can
also be accessed through accessor methods ‘_get_foo()’ and
‘_set_foo()’. ‘readonly’ attributes must not be changed; this is
not enforced at runtime.
* The types ‘short int’, ‘unsigned int’, ‘unsigned long long’, and
‘boolean’ all map to Python integer objects.
* The type ‘DOMString’ maps to Python strings. *note
xml.dom.minidom: 122. supports either bytes or strings, but will
normally produce strings. Values of type ‘DOMString’ may also be
‘None’ where allowed to have the IDL ‘null’ value by the DOM
specification from the W3C.
* ‘const’ declarations map to variables in their respective scope
(e.g. ‘xml.dom.minidom.Node.PROCESSING_INSTRUCTION_NODE’); they
must not be changed.
* ‘DOMException’ is currently not supported in *note xml.dom.minidom:
122. Instead, *note xml.dom.minidom: 122. uses standard Python
exceptions such as *note TypeError: 534. and *note AttributeError:
348.
* ‘NodeList’ objects are implemented using Python’s built-in list
type. These objects provide the interface defined in the DOM
specification, but with earlier versions of Python they do not
support the official API. They are, however, much more “Pythonic”
than the interface defined in the W3C recommendations.
The following interfaces have no implementation in *note
xml.dom.minidom: 122.:
* ‘DOMTimeStamp’
* ‘EntityReference’
Most of these reflect information in the XML document that is not of
general utility to most DOM users.
File: python.info, Node: xml dom pulldom — Support for building partial DOM trees, Next: xml sax — Support for SAX2 parsers, Prev: xml dom minidom — Minimal DOM implementation, Up: Structured Markup Processing Tools
5.21.8 ‘xml.dom.pulldom’ — Support for building partial DOM trees
-----------------------------------------------------------------
'Source code:' Lib/xml/dom/pulldom.py(1)
__________________________________________________________________
The *note xml.dom.pulldom: 123. module provides a “pull parser” which
can also be asked to produce DOM-accessible fragments of the document
where necessary. The basic concept involves pulling “events” from a
stream of incoming XML and processing them. In contrast to SAX which
also employs an event-driven processing model together with callbacks,
the user of a pull parser is responsible for explicitly pulling events
from the stream, looping over those events until either processing is
finished or an error condition occurs.
Note: If you need to parse untrusted or unauthenticated data, see
*note XML security: 3774.
Changed in version 3.7.1: The SAX parser no longer processes general
external entities by default to increase security by default. To enable
processing of external entities, pass a custom parser instance in:
from xml.dom.pulldom import parse
from xml.sax import make_parser
from xml.sax.handler import feature_external_ges
parser = make_parser()
parser.setFeature(feature_external_ges, True)
parse(filename, parser=parser)
Example:
from xml.dom import pulldom
doc = pulldom.parse('sales_items.xml')
for event, node in doc:
if event == pulldom.START_ELEMENT and node.tagName == 'item':
if int(node.getAttribute('price')) > 50:
doc.expandNode(node)
print(node.toxml())
‘event’ is a constant and can be one of:
* ‘START_ELEMENT’
* ‘END_ELEMENT’
* ‘COMMENT’
* ‘START_DOCUMENT’
* ‘END_DOCUMENT’
* ‘CHARACTERS’
* ‘PROCESSING_INSTRUCTION’
* ‘IGNORABLE_WHITESPACE’
‘node’ is an object of type ‘xml.dom.minidom.Document’,
‘xml.dom.minidom.Element’ or ‘xml.dom.minidom.Text’.
Since the document is treated as a “flat” stream of events, the document
“tree” is implicitly traversed and the desired elements are found
regardless of their depth in the tree. In other words, one does not
need to consider hierarchical issues such as recursive searching of the
document nodes, although if the context of elements were important, one
would either need to maintain some context-related state (i.e.
remembering where one is in the document at any given point) or to make
use of the *note DOMEventStream.expandNode(): 3856. method and switch to
DOM-related processing.
-- Class: xml.dom.pulldom.PullDom (documentFactory=None)
Subclass of *note xml.sax.handler.ContentHandler: 3858.
-- Class: xml.dom.pulldom.SAX2DOM (documentFactory=None)
Subclass of *note xml.sax.handler.ContentHandler: 3858.
-- Function: xml.dom.pulldom.parse (stream_or_string, parser=None,
bufsize=None)
Return a *note DOMEventStream: 730. from the given input.
'stream_or_string' may be either a file name, or a file-like
object. 'parser', if given, must be an *note XMLReader: 385b.
object. This function will change the document handler of the
parser and activate namespace support; other parser configuration
(like setting an entity resolver) must have been done in advance.
If you have XML in a string, you can use the *note parseString(): 385c.
function instead:
-- Function: xml.dom.pulldom.parseString (string, parser=None)
Return a *note DOMEventStream: 730. that represents the (Unicode)
'string'.
-- Data: xml.dom.pulldom.default_bufsize
Default value for the 'bufsize' parameter to *note parse(): 385a.
The value of this variable can be changed before calling *note
parse(): 385a. and the new value will take effect.
* Menu:
* DOMEventStream Objects::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/xml/dom/pulldom.py
File: python.info, Node: DOMEventStream Objects, Up: xml dom pulldom — Support for building partial DOM trees
5.21.8.1 DOMEventStream Objects
...............................
-- Class: xml.dom.pulldom.DOMEventStream (stream, parser, bufsize)
Changed in version 3.11: Support for *note __getitem__(): 285.
method has been removed.
-- Method: getEvent ()
Return a tuple containing 'event' and the current 'node' as
‘xml.dom.minidom.Document’ if event equals ‘START_DOCUMENT’,
‘xml.dom.minidom.Element’ if event equals ‘START_ELEMENT’ or
‘END_ELEMENT’ or ‘xml.dom.minidom.Text’ if event equals
‘CHARACTERS’. The current node does not contain information
about its children, unless *note expandNode(): 3856. is
called.
-- Method: expandNode (node)
Expands all children of 'node' into 'node'. Example:
from xml.dom import pulldom
xml = 'Foo Some text
and more
'
doc = pulldom.parseString(xml)
for event, node in doc:
if event == pulldom.START_ELEMENT and node.tagName == 'p':
# Following statement only prints '
'
print(node.toxml())
doc.expandNode(node)
# Following statement prints node with all its children 'Some text
and more
'
print(node.toxml())
-- Method: reset ()
File: python.info, Node: xml sax — Support for SAX2 parsers, Next: xml sax handler — Base classes for SAX handlers, Prev: xml dom pulldom — Support for building partial DOM trees, Up: Structured Markup Processing Tools
5.21.9 ‘xml.sax’ — Support for SAX2 parsers
-------------------------------------------
'Source code:' Lib/xml/sax/__init__.py(1)
__________________________________________________________________
The *note xml.sax: 129. package provides a number of modules which
implement the Simple API for XML (SAX) interface for Python. The
package itself provides the SAX exceptions and the convenience functions
which will be most used by users of the SAX API.
Note: If you need to parse untrusted or unauthenticated data, see
*note XML security: 3774.
Changed in version 3.7.1: The SAX parser no longer processes general
external entities by default to increase security. Before, the parser
created network connections to fetch remote files or loaded local files
from the file system for DTD and entities. The feature can be enabled
again with method *note setFeature(): 3864. on the parser object and
argument *note feature_external_ges: 3865.
The convenience functions are:
-- Function: xml.sax.make_parser (parser_list=[])
Create and return a SAX *note XMLReader: 385b. object. The first
parser found will be used. If 'parser_list' is provided, it must
be an iterable of strings which name modules that have a function
named ‘create_parser()’. Modules listed in 'parser_list' will be
used before modules in the default list of parsers.
Changed in version 3.8: The 'parser_list' argument can be any
iterable, not just a list.
-- Function: xml.sax.parse (filename_or_stream, handler,
error_handler=handler.ErrorHandler())
Create a SAX parser and use it to parse a document. The document,
passed in as 'filename_or_stream', can be a filename or a file
object. The 'handler' parameter needs to be a SAX *note
ContentHandler: 3858. instance. If 'error_handler' is given, it
must be a SAX *note ErrorHandler: 3866. instance; if omitted, *note
SAXParseException: 3867. will be raised on all errors. There is no
return value; all work must be done by the 'handler' passed in.
-- Function: xml.sax.parseString (string, handler,
error_handler=handler.ErrorHandler())
Similar to *note parse(): 1a03, but parses from a buffer 'string'
received as a parameter. 'string' must be a *note str: 447.
instance or a *note bytes-like object: d2d.
Changed in version 3.5: Added support of *note str: 447. instances.
A typical SAX application uses three kinds of objects: readers, handlers
and input sources. “Reader” in this context is another term for parser,
i.e. some piece of code that reads the bytes or characters from the
input source, and produces a sequence of events. The events then get
distributed to the handler objects, i.e. the reader invokes a method on
the handler. A SAX application must therefore obtain a reader object,
create or open the input sources, create the handlers, and connect these
objects all together. As the final step of preparation, the reader is
called to parse the input. During parsing, methods on the handler
objects are called based on structural and syntactic events from the
input data.
For these objects, only the interfaces are relevant; they are normally
not instantiated by the application itself. Since Python does not have
an explicit notion of interface, they are formally introduced as
classes, but applications may use implementations which do not inherit
from the provided classes. The *note InputSource: e94, *note Locator:
3868, ‘Attributes’, ‘AttributesNS’, and *note XMLReader: 385b.
interfaces are defined in the module *note xml.sax.xmlreader: 12c. The
handler interfaces are defined in *note xml.sax.handler: 12a. For
convenience, *note InputSource: e94. (which is often instantiated
directly) and the handler classes are also available from *note xml.sax:
129. These interfaces are described below.
In addition to these classes, *note xml.sax: 129. provides the following
exception classes.
-- Exception: xml.sax.SAXException (msg, exception=None)
Encapsulate an XML error or warning. This class can contain basic
error or warning information from either the XML parser or the
application: it can be subclassed to provide additional
functionality or to add localization. Note that although the
handlers defined in the *note ErrorHandler: 3866. interface receive
instances of this exception, it is not required to actually raise
the exception — it is also useful as a container for information.
When instantiated, 'msg' should be a human-readable description of
the error. The optional 'exception' parameter, if given, should be
‘None’ or an exception that was caught by the parsing code and is
being passed along as information.
This is the base class for the other SAX exception classes.
-- Exception: xml.sax.SAXParseException (msg, exception, locator)
Subclass of *note SAXException: 3869. raised on parse errors.
Instances of this class are passed to the methods of the SAX *note
ErrorHandler: 3866. interface to provide information about the
parse error. This class supports the SAX *note Locator: 3868.
interface as well as the *note SAXException: 3869. interface.
-- Exception: xml.sax.SAXNotRecognizedException (msg, exception=None)
Subclass of *note SAXException: 3869. raised when a SAX *note
XMLReader: 385b. is confronted with an unrecognized feature or
property. SAX applications and extensions may use this class for
similar purposes.
-- Exception: xml.sax.SAXNotSupportedException (msg, exception=None)
Subclass of *note SAXException: 3869. raised when a SAX *note
XMLReader: 385b. is asked to enable a feature that is not
supported, or to set a property to a value that the implementation
does not support. SAX applications and extensions may use this
class for similar purposes.
See also
........
SAX: The Simple API for XML(2)
This site is the focal point for the definition of the SAX API. It
provides a Java implementation and online documentation. Links to
implementations and historical information are also available.
Module *note xml.sax.handler: 12a.
Definitions of the interfaces for application-provided objects.
Module *note xml.sax.saxutils: 12b.
Convenience functions for use in SAX applications.
Module *note xml.sax.xmlreader: 12c.
Definitions of the interfaces for parser-provided objects.
* Menu:
* SAXException Objects::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/xml/sax/__init__.py
(2) http://www.saxproject.org/
File: python.info, Node: SAXException Objects, Up: xml sax — Support for SAX2 parsers
5.21.9.1 SAXException Objects
.............................
The *note SAXException: 3869. exception class supports the following
methods:
-- Method: SAXException.getMessage ()
Return a human-readable message describing the error condition.
-- Method: SAXException.getException ()
Return an encapsulated exception object, or ‘None’.
File: python.info, Node: xml sax handler — Base classes for SAX handlers, Next: xml sax saxutils — SAX Utilities, Prev: xml sax — Support for SAX2 parsers, Up: Structured Markup Processing Tools
5.21.10 ‘xml.sax.handler’ — Base classes for SAX handlers
---------------------------------------------------------
'Source code:' Lib/xml/sax/handler.py(1)
__________________________________________________________________
The SAX API defines five kinds of handlers: content handlers, DTD
handlers, error handlers, entity resolvers and lexical handlers.
Applications normally only need to implement those interfaces whose
events they are interested in; they can implement the interfaces in a
single object or in multiple objects. Handler implementations should
inherit from the base classes provided in the module *note
xml.sax.handler: 12a, so that all methods get default implementations.
-- Class: xml.sax.handler.ContentHandler
This is the main callback interface in SAX, and the one most
important to applications. The order of events in this interface
mirrors the order of the information in the document.
-- Class: xml.sax.handler.DTDHandler
Handle DTD events.
This interface specifies only those DTD events required for basic
parsing (unparsed entities and attributes).
-- Class: xml.sax.handler.EntityResolver
Basic interface for resolving entities. If you create an object
implementing this interface, then register the object with your
Parser, the parser will call the method in your object to resolve
all external entities.
-- Class: xml.sax.handler.ErrorHandler
Interface used by the parser to present error and warning messages
to the application. The methods of this object control whether
errors are immediately converted to exceptions or are handled in
some other way.
-- Class: xml.sax.handler.LexicalHandler
Interface used by the parser to represent low frequency events
which may not be of interest to many applications.
In addition to these classes, *note xml.sax.handler: 12a. provides
symbolic constants for the feature and property names.
-- Data: xml.sax.handler.feature_namespaces
value: ‘"http://xml.org/sax/features/namespaces"’
true: Perform Namespace processing.
false: Optionally do not perform Namespace processing (implies namespace-prefixes; default).
access: (parsing) read-only; (not parsing) read/write
-- Data: xml.sax.handler.feature_namespace_prefixes
value: ‘"http://xml.org/sax/features/namespace-prefixes"’
true: Report the original prefixed names and attributes used for Namespace declarations.
false: Do not report attributes used for Namespace declarations, and optionally do not report original prefixed names (default).
access: (parsing) read-only; (not parsing) read/write
-- Data: xml.sax.handler.feature_string_interning
value: ‘"http://xml.org/sax/features/string-interning"’
true: All element names, prefixes, attribute names, Namespace URIs, and local names are interned using the built-in intern function.
false: Names are not necessarily interned, although they may be (default).
access: (parsing) read-only; (not parsing) read/write
-- Data: xml.sax.handler.feature_validation
value: ‘"http://xml.org/sax/features/validation"’
true: Report all validation errors (implies external-general-entities and external-parameter-entities).
false: Do not report validation errors.
access: (parsing) read-only; (not parsing) read/write
-- Data: xml.sax.handler.feature_external_ges
value: ‘"http://xml.org/sax/features/external-general-entities"’
true: Include all external general (text) entities.
false: Do not include external general entities.
access: (parsing) read-only; (not parsing) read/write
-- Data: xml.sax.handler.feature_external_pes
value: ‘"http://xml.org/sax/features/external-parameter-entities"’
true: Include all external parameter entities, including the external DTD subset.
false: Do not include any external parameter entities, even the external DTD subset.
access: (parsing) read-only; (not parsing) read/write
-- Data: xml.sax.handler.all_features
List of all features.
-- Data: xml.sax.handler.property_lexical_handler
value: ‘"http://xml.org/sax/properties/lexical-handler"’
data type: xml.sax.handler.LexicalHandler (not supported in Python 2)
description: An optional extension handler for lexical events like comments.
access: read/write
-- Data: xml.sax.handler.property_declaration_handler
value: ‘"http://xml.org/sax/properties/declaration-handler"’
data type: xml.sax.sax2lib.DeclHandler (not supported in Python 2)
description: An optional extension handler for DTD-related events other than notations and unparsed entities.
access: read/write
-- Data: xml.sax.handler.property_dom_node
value: ‘"http://xml.org/sax/properties/dom-node"’
data type: org.w3c.dom.Node (not supported in Python 2)
description: When parsing, the current DOM node being visited if this is a DOM iterator; when not parsing, the root DOM node for iteration.
access: (parsing) read-only; (not parsing) read/write
-- Data: xml.sax.handler.property_xml_string
value: ‘"http://xml.org/sax/properties/xml-string"’
data type: Bytes
description: The literal string of characters that was the source for the current event.
access: read-only
-- Data: xml.sax.handler.all_properties
List of all known property names.
* Menu:
* ContentHandler Objects::
* DTDHandler Objects::
* EntityResolver Objects::
* ErrorHandler Objects::
* LexicalHandler Objects::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/xml/sax/handler.py
File: python.info, Node: ContentHandler Objects, Next: DTDHandler Objects, Up: xml sax handler — Base classes for SAX handlers
5.21.10.1 ContentHandler Objects
................................
Users are expected to subclass *note ContentHandler: 3858. to support
their application. The following methods are called by the parser on
the appropriate events in the input document:
-- Method: ContentHandler.setDocumentLocator (locator)
Called by the parser to give the application a locator for locating
the origin of document events.
SAX parsers are strongly encouraged (though not absolutely
required) to supply a locator: if it does so, it must supply the
locator to the application by invoking this method before invoking
any of the other methods in the DocumentHandler interface.
The locator allows the application to determine the end position of
any document-related event, even if the parser is not reporting an
error. Typically, the application will use this information for
reporting its own errors (such as character content that does not
match an application’s business rules). The information returned
by the locator is probably not sufficient for use with a search
engine.
Note that the locator will return correct information only during
the invocation of the events in this interface. The application
should not attempt to use it at any other time.
-- Method: ContentHandler.startDocument ()
Receive notification of the beginning of a document.
The SAX parser will invoke this method only once, before any other
methods in this interface or in DTDHandler (except for *note
setDocumentLocator(): 3881.).
-- Method: ContentHandler.endDocument ()
Receive notification of the end of a document.
The SAX parser will invoke this method only once, and it will be
the last method invoked during the parse. The parser shall not
invoke this method until it has either abandoned parsing (because
of an unrecoverable error) or reached the end of input.
-- Method: ContentHandler.startPrefixMapping (prefix, uri)
Begin the scope of a prefix-URI Namespace mapping.
The information from this event is not necessary for normal
Namespace processing: the SAX XML reader will automatically replace
prefixes for element and attribute names when the
‘feature_namespaces’ feature is enabled (the default).
There are cases, however, when applications need to use prefixes in
character data or in attribute values, where they cannot safely be
expanded automatically; the *note startPrefixMapping(): 3884. and
*note endPrefixMapping(): 3885. events supply the information to
the application to expand prefixes in those contexts itself, if
necessary.
Note that *note startPrefixMapping(): 3884. and *note
endPrefixMapping(): 3885. events are not guaranteed to be properly
nested relative to each-other: all *note startPrefixMapping():
3884. events will occur before the corresponding *note
startElement(): 150d. event, and all *note endPrefixMapping():
3885. events will occur after the corresponding *note endElement():
150e. event, but their order is not guaranteed.
-- Method: ContentHandler.endPrefixMapping (prefix)
End the scope of a prefix-URI mapping.
See *note startPrefixMapping(): 3884. for details. This event will
always occur after the corresponding *note endElement(): 150e.
event, but the order of *note endPrefixMapping(): 3885. events is
not otherwise guaranteed.
-- Method: ContentHandler.startElement (name, attrs)
Signals the start of an element in non-namespace mode.
The 'name' parameter contains the raw XML 1.0 name of the element
type as a string and the 'attrs' parameter holds an object of the
‘Attributes’ interface (see *note The Attributes Interface: 3886.)
containing the attributes of the element. The object passed as
'attrs' may be re-used by the parser; holding on to a reference to
it is not a reliable way to keep a copy of the attributes. To keep
a copy of the attributes, use the *note copy(): 25. method of the
'attrs' object.
-- Method: ContentHandler.endElement (name)
Signals the end of an element in non-namespace mode.
The 'name' parameter contains the name of the element type, just as
with the *note startElement(): 150d. event.
-- Method: ContentHandler.startElementNS (name, qname, attrs)
Signals the start of an element in namespace mode.
The 'name' parameter contains the name of the element type as a
‘(uri, localname)’ tuple, the 'qname' parameter contains the raw
XML 1.0 name used in the source document, and the 'attrs' parameter
holds an instance of the ‘AttributesNS’ interface (see *note The
AttributesNS Interface: 3888.) containing the attributes of the
element. If no namespace is associated with the element, the 'uri'
component of 'name' will be ‘None’. The object passed as 'attrs'
may be re-used by the parser; holding on to a reference to it is
not a reliable way to keep a copy of the attributes. To keep a
copy of the attributes, use the *note copy(): 25. method of the
'attrs' object.
Parsers may set the 'qname' parameter to ‘None’, unless the
‘feature_namespace_prefixes’ feature is activated.
-- Method: ContentHandler.endElementNS (name, qname)
Signals the end of an element in namespace mode.
The 'name' parameter contains the name of the element type, just as
with the *note startElementNS(): 3887. method, likewise the 'qname'
parameter.
-- Method: ContentHandler.characters (content)
Receive notification of character data.
The Parser will call this method to report each chunk of character
data. SAX parsers may return all contiguous character data in a
single chunk, or they may split it into several chunks; however,
all of the characters in any single event must come from the same
external entity so that the Locator provides useful information.
'content' may be a string or bytes instance; the ‘expat’ reader
module always produces strings.
Note: The earlier SAX 1 interface provided by the Python XML
Special Interest Group used a more Java-like interface for
this method. Since most parsers used from Python did not take
advantage of the older interface, the simpler signature was
chosen to replace it. To convert old code to the new
interface, use 'content' instead of slicing content with the
old 'offset' and 'length' parameters.
-- Method: ContentHandler.ignorableWhitespace (whitespace)
Receive notification of ignorable whitespace in element content.
Validating Parsers must use this method to report each chunk of
ignorable whitespace (see the W3C XML 1.0 recommendation, section
2.10): non-validating parsers may also use this method if they are
capable of parsing and using content models.
SAX parsers may return all contiguous whitespace in a single chunk,
or they may split it into several chunks; however, all of the
characters in any single event must come from the same external
entity, so that the Locator provides useful information.
-- Method: ContentHandler.processingInstruction (target, data)
Receive notification of a processing instruction.
The Parser will invoke this method once for each processing
instruction found: note that processing instructions may occur
before or after the main document element.
A SAX parser should never report an XML declaration (XML 1.0,
section 2.8) or a text declaration (XML 1.0, section 4.3.1) using
this method.
-- Method: ContentHandler.skippedEntity (name)
Receive notification of a skipped entity.
The Parser will invoke this method once for each entity skipped.
Non-validating processors may skip entities if they have not seen
the declarations (because, for example, the entity was declared in
an external DTD subset). All processors may skip external
entities, depending on the values of the ‘feature_external_ges’ and
the ‘feature_external_pes’ properties.
File: python.info, Node: DTDHandler Objects, Next: EntityResolver Objects, Prev: ContentHandler Objects, Up: xml sax handler — Base classes for SAX handlers
5.21.10.2 DTDHandler Objects
............................
*note DTDHandler: 3872. instances provide the following methods:
-- Method: DTDHandler.notationDecl (name, publicId, systemId)
Handle a notation declaration event.
-- Method: DTDHandler.unparsedEntityDecl (name, publicId, systemId,
ndata)
Handle an unparsed entity declaration event.
File: python.info, Node: EntityResolver Objects, Next: ErrorHandler Objects, Prev: DTDHandler Objects, Up: xml sax handler — Base classes for SAX handlers
5.21.10.3 EntityResolver Objects
................................
-- Method: EntityResolver.resolveEntity (publicId, systemId)
Resolve the system identifier of an entity and return either the
system identifier to read from as a string, or an InputSource to
read from. The default implementation returns 'systemId'.
File: python.info, Node: ErrorHandler Objects, Next: LexicalHandler Objects, Prev: EntityResolver Objects, Up: xml sax handler — Base classes for SAX handlers
5.21.10.4 ErrorHandler Objects
..............................
Objects with this interface are used to receive error and warning
information from the *note XMLReader: 385b. If you create an object
that implements this interface, then register the object with your *note
XMLReader: 385b, the parser will call the methods in your object to
report all warnings and errors. There are three levels of errors
available: warnings, (possibly) recoverable errors, and unrecoverable
errors. All methods take a *note SAXParseException: 3867. as the only
parameter. Errors and warnings may be converted to an exception by
raising the passed-in exception object.
-- Method: ErrorHandler.error (exception)
Called when the parser encounters a recoverable error. If this
method does not raise an exception, parsing may continue, but
further document information should not be expected by the
application. Allowing the parser to continue may allow additional
errors to be discovered in the input document.
-- Method: ErrorHandler.fatalError (exception)
Called when the parser encounters an error it cannot recover from;
parsing is expected to terminate when this method returns.
-- Method: ErrorHandler.warning (exception)
Called when the parser presents minor warning information to the
application. Parsing is expected to continue when this method
returns, and document information will continue to be passed to the
application. Raising an exception in this method will cause
parsing to end.
File: python.info, Node: LexicalHandler Objects, Prev: ErrorHandler Objects, Up: xml sax handler — Base classes for SAX handlers
5.21.10.5 LexicalHandler Objects
................................
Optional SAX2 handler for lexical events.
This handler is used to obtain lexical information about an XML
document. Lexical information includes information describing the
document encoding used and XML comments embedded in the document, as
well as section boundaries for the DTD and for any CDATA sections. The
lexical handlers are used in the same manner as content handlers.
Set the LexicalHandler of an XMLReader by using the setProperty method
with the property identifier
‘'http://xml.org/sax/properties/lexical-handler'’.
-- Method: LexicalHandler.comment (content)
Reports a comment anywhere in the document (including the DTD and
outside the document element).
-- Method: LexicalHandler.startDTD (name, public_id, system_id)
Reports the start of the DTD declarations if the document has an
associated DTD.
-- Method: LexicalHandler.endDTD ()
Reports the end of DTD declaration.
-- Method: LexicalHandler.startCDATA ()
Reports the start of a CDATA marked section.
The contents of the CDATA marked section will be reported through
the characters handler.
-- Method: LexicalHandler.endCDATA ()
Reports the end of a CDATA marked section.
File: python.info, Node: xml sax saxutils — SAX Utilities, Next: xml sax xmlreader — Interface for XML parsers, Prev: xml sax handler — Base classes for SAX handlers, Up: Structured Markup Processing Tools
5.21.11 ‘xml.sax.saxutils’ — SAX Utilities
------------------------------------------
'Source code:' Lib/xml/sax/saxutils.py(1)
__________________________________________________________________
The module *note xml.sax.saxutils: 12b. contains a number of classes and
functions that are commonly useful when creating SAX applications,
either in direct use, or as base classes.
-- Function: xml.sax.saxutils.escape (data, entities={})
Escape ‘'&'’, ‘'<'’, and ‘'>'’ in a string of data.
You can escape other strings of data by passing a dictionary as the
optional 'entities' parameter. The keys and values must all be
strings; each key will be replaced with its corresponding value.
The characters ‘'&'’, ‘'<'’ and ‘'>'’ are always escaped, even if
'entities' is provided.
Note: This function should only be used to escape characters
that can’t be used directly in XML. Do not use this function
as a general string translation function.
-- Function: xml.sax.saxutils.unescape (data, entities={})
Unescape ‘'&'’, ‘'<'’, and ‘'>'’ in a string of data.
You can unescape other strings of data by passing a dictionary as
the optional 'entities' parameter. The keys and values must all be
strings; each key will be replaced with its corresponding value.
‘'&'’, ‘'<'’, and ‘'>'’ are always unescaped, even if
'entities' is provided.
-- Function: xml.sax.saxutils.quoteattr (data, entities={})
Similar to *note escape(): 38a2, but also prepares 'data' to be
used as an attribute value. The return value is a quoted version
of 'data' with any additional required replacements. *note
quoteattr(): 38a4. will select a quote character based on the
content of 'data', attempting to avoid encoding any quote
characters in the string. If both single- and double-quote
characters are already in 'data', the double-quote characters will
be encoded and 'data' will be wrapped in double-quotes. The
resulting string can be used directly as an attribute value:
>>> print("" % quoteattr("ab ' cd \" ef"))
This function is useful when generating attribute values for HTML
or any SGML using the reference concrete syntax.
-- Class: xml.sax.saxutils.XMLGenerator (out=None,
encoding='iso-8859-1', short_empty_elements=False)
This class implements the *note ContentHandler: 3858. interface by
writing SAX events back into an XML document. In other words,
using an *note XMLGenerator: 38a5. as the content handler will
reproduce the original document being parsed. 'out' should be a
file-like object which will default to 'sys.stdout'. 'encoding' is
the encoding of the output stream which defaults to ‘'iso-8859-1'’.
'short_empty_elements' controls the formatting of elements that
contain no content: if ‘False’ (the default) they are emitted as a
pair of start/end tags, if set to ‘True’ they are emitted as a
single self-closed tag.
Changed in version 3.2: Added the 'short_empty_elements' parameter.
-- Class: xml.sax.saxutils.XMLFilterBase (base)
This class is designed to sit between an *note XMLReader: 385b. and
the client application’s event handlers. By default, it does
nothing but pass requests up to the reader and events on to the
handlers unmodified, but subclasses can override specific methods
to modify the event stream or the configuration requests as they
pass through.
-- Function: xml.sax.saxutils.prepare_input_source (source, base='')
This function takes an input source and an optional base URL and
returns a fully resolved *note InputSource: e94. object ready for
reading. The input source can be given as a string, a file-like
object, or an *note InputSource: e94. object; parsers will use this
function to implement the polymorphic 'source' argument to their
*note parse(): 38a8. method.
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/xml/sax/saxutils.py
File: python.info, Node: xml sax xmlreader — Interface for XML parsers, Next: xml parsers expat — Fast XML parsing using Expat, Prev: xml sax saxutils — SAX Utilities, Up: Structured Markup Processing Tools
5.21.12 ‘xml.sax.xmlreader’ — Interface for XML parsers
-------------------------------------------------------
'Source code:' Lib/xml/sax/xmlreader.py(1)
__________________________________________________________________
SAX parsers implement the *note XMLReader: 385b. interface. They are
implemented in a Python module, which must provide a function
‘create_parser()’. This function is invoked by *note
xml.sax.make_parser(): 1a3a. with no arguments to create a new parser
object.
-- Class: xml.sax.xmlreader.XMLReader
Base class which can be inherited by SAX parsers.
-- Class: xml.sax.xmlreader.IncrementalParser
In some cases, it is desirable not to parse an input source at
once, but to feed chunks of the document as they get available.
Note that the reader will normally not read the entire file, but
read it in chunks as well; still ‘parse()’ won’t return until the
entire document is processed. So these interfaces should be used
if the blocking behaviour of ‘parse()’ is not desirable.
When the parser is instantiated it is ready to begin accepting data
from the feed method immediately. After parsing has been finished
with a call to close the reset method must be called to make the
parser ready to accept new data, either from feed or using the
parse method.
Note that these methods must 'not' be called during parsing, that
is, after parse has been called and before it returns.
By default, the class also implements the parse method of the
XMLReader interface using the feed, close and reset methods of the
IncrementalParser interface as a convenience to SAX 2.0 driver
writers.
-- Class: xml.sax.xmlreader.Locator
Interface for associating a SAX event with a document location. A
locator object will return valid results only during calls to
DocumentHandler methods; at any other time, the results are
unpredictable. If information is not available, methods may return
‘None’.
-- Class: xml.sax.xmlreader.InputSource (system_id=None)
Encapsulation of the information needed by the *note XMLReader:
385b. to read entities.
This class may include information about the public identifier,
system identifier, byte stream (possibly with character encoding
information) and/or the character stream of an entity.
Applications will create objects of this class for use in the *note
XMLReader.parse(): 38a8. method and for returning from
EntityResolver.resolveEntity.
An *note InputSource: e94. belongs to the application, the *note
XMLReader: 385b. is not allowed to modify *note InputSource: e94.
objects passed to it from the application, although it may make
copies and modify those.
-- Class: xml.sax.xmlreader.AttributesImpl (attrs)
This is an implementation of the ‘Attributes’ interface (see
section *note The Attributes Interface: 3886.). This is a
dictionary-like object which represents the element attributes in a
‘startElement()’ call. In addition to the most useful dictionary
operations, it supports a number of other methods as described by
the interface. Objects of this class should be instantiated by
readers; 'attrs' must be a dictionary-like object containing a
mapping from attribute names to attribute values.
-- Class: xml.sax.xmlreader.AttributesNSImpl (attrs, qnames)
Namespace-aware variant of *note AttributesImpl: 38ac, which will
be passed to ‘startElementNS()’. It is derived from *note
AttributesImpl: 38ac, but understands attribute names as two-tuples
of 'namespaceURI' and 'localname'. In addition, it provides a
number of methods expecting qualified names as they appear in the
original document. This class implements the ‘AttributesNS’
interface (see section *note The AttributesNS Interface: 3888.).
* Menu:
* XMLReader Objects::
* IncrementalParser Objects::
* Locator Objects::
* InputSource Objects::
* The Attributes Interface::
* The AttributesNS Interface::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/xml/sax/xmlreader.py
File: python.info, Node: XMLReader Objects, Next: IncrementalParser Objects, Up: xml sax xmlreader — Interface for XML parsers
5.21.12.1 XMLReader Objects
...........................
The *note XMLReader: 385b. interface supports the following methods:
-- Method: XMLReader.parse (source)
Process an input source, producing SAX events. The 'source' object
can be a system identifier (a string identifying the input source –
typically a file name or a URL), a *note pathlib.Path: 22b. or
*note path-like: 2a2. object, or an *note InputSource: e94. object.
When *note parse(): 38a8. returns, the input is completely
processed, and the parser object can be discarded or reset.
Changed in version 3.5: Added support of character streams.
Changed in version 3.8: Added support of path-like objects.
-- Method: XMLReader.getContentHandler ()
Return the current *note ContentHandler: 3858.
-- Method: XMLReader.setContentHandler (handler)
Set the current *note ContentHandler: 3858. If no *note
ContentHandler: 3858. is set, content events will be discarded.
-- Method: XMLReader.getDTDHandler ()
Return the current *note DTDHandler: 3872.
-- Method: XMLReader.setDTDHandler (handler)
Set the current *note DTDHandler: 3872. If no *note DTDHandler:
3872. is set, DTD events will be discarded.
-- Method: XMLReader.getEntityResolver ()
Return the current *note EntityResolver: 3873.
-- Method: XMLReader.setEntityResolver (handler)
Set the current *note EntityResolver: 3873. If no *note
EntityResolver: 3873. is set, attempts to resolve an external
entity will result in opening the system identifier for the entity,
and fail if it is not available.
-- Method: XMLReader.getErrorHandler ()
Return the current *note ErrorHandler: 3866.
-- Method: XMLReader.setErrorHandler (handler)
Set the current error handler. If no *note ErrorHandler: 3866. is
set, errors will be raised as exceptions, and warnings will be
printed.
-- Method: XMLReader.setLocale (locale)
Allow an application to set the locale for errors and warnings.
SAX parsers are not required to provide localization for errors and
warnings; if they cannot support the requested locale, however,
they must raise a SAX exception. Applications may request a locale
change in the middle of a parse.
-- Method: XMLReader.getFeature (featurename)
Return the current setting for feature 'featurename'. If the
feature is not recognized, ‘SAXNotRecognizedException’ is raised.
The well-known featurenames are listed in the module *note
xml.sax.handler: 12a.
-- Method: XMLReader.setFeature (featurename, value)
Set the 'featurename' to 'value'. If the feature is not
recognized, ‘SAXNotRecognizedException’ is raised. If the feature
or its setting is not supported by the parser,
'SAXNotSupportedException' is raised.
-- Method: XMLReader.getProperty (propertyname)
Return the current setting for property 'propertyname'. If the
property is not recognized, a ‘SAXNotRecognizedException’ is
raised. The well-known propertynames are listed in the module
*note xml.sax.handler: 12a.
-- Method: XMLReader.setProperty (propertyname, value)
Set the 'propertyname' to 'value'. If the property is not
recognized, ‘SAXNotRecognizedException’ is raised. If the property
or its setting is not supported by the parser,
'SAXNotSupportedException' is raised.
File: python.info, Node: IncrementalParser Objects, Next: Locator Objects, Prev: XMLReader Objects, Up: xml sax xmlreader — Interface for XML parsers
5.21.12.2 IncrementalParser Objects
...................................
Instances of *note IncrementalParser: 38ab. offer the following
additional methods:
-- Method: IncrementalParser.feed (data)
Process a chunk of 'data'.
-- Method: IncrementalParser.close ()
Assume the end of the document. That will check well-formedness
conditions that can be checked only at the end, invoke handlers,
and may clean up resources allocated during parsing.
-- Method: IncrementalParser.reset ()
This method is called after close has been called to reset the
parser so that it is ready to parse new documents. The results of
calling parse or feed after close without calling reset are
undefined.
File: python.info, Node: Locator Objects, Next: InputSource Objects, Prev: IncrementalParser Objects, Up: xml sax xmlreader — Interface for XML parsers
5.21.12.3 Locator Objects
.........................
Instances of *note Locator: 3868. provide these methods:
-- Method: Locator.getColumnNumber ()
Return the column number where the current event begins.
-- Method: Locator.getLineNumber ()
Return the line number where the current event begins.
-- Method: Locator.getPublicId ()
Return the public identifier for the current event.
-- Method: Locator.getSystemId ()
Return the system identifier for the current event.
File: python.info, Node: InputSource Objects, Next: The Attributes Interface, Prev: Locator Objects, Up: xml sax xmlreader — Interface for XML parsers
5.21.12.4 InputSource Objects
.............................
-- Method: InputSource.setPublicId (id)
Sets the public identifier of this *note InputSource: e94.
-- Method: InputSource.getPublicId ()
Returns the public identifier of this *note InputSource: e94.
-- Method: InputSource.setSystemId (id)
Sets the system identifier of this *note InputSource: e94.
-- Method: InputSource.getSystemId ()
Returns the system identifier of this *note InputSource: e94.
-- Method: InputSource.setEncoding (encoding)
Sets the character encoding of this *note InputSource: e94.
The encoding must be a string acceptable for an XML encoding
declaration (see section 4.3.3 of the XML recommendation).
The encoding attribute of the *note InputSource: e94. is ignored if
the *note InputSource: e94. also contains a character stream.
-- Method: InputSource.getEncoding ()
Get the character encoding of this InputSource.
-- Method: InputSource.setByteStream (bytefile)
Set the byte stream (a *note binary file: 1c87.) for this input
source.
The SAX parser will ignore this if there is also a character stream
specified, but it will use a byte stream in preference to opening a
URI connection itself.
If the application knows the character encoding of the byte stream,
it should set it with the setEncoding method.
-- Method: InputSource.getByteStream ()
Get the byte stream for this input source.
The getEncoding method will return the character encoding for this
byte stream, or ‘None’ if unknown.
-- Method: InputSource.setCharacterStream (charfile)
Set the character stream (a *note text file: 1c86.) for this input
source.
If there is a character stream specified, the SAX parser will
ignore any byte stream and will not attempt to open a URI
connection to the system identifier.
-- Method: InputSource.getCharacterStream ()
Get the character stream for this input source.
File: python.info, Node: The Attributes Interface, Next: The AttributesNS Interface, Prev: InputSource Objects, Up: xml sax xmlreader — Interface for XML parsers
5.21.12.5 The ‘Attributes’ Interface
....................................
‘Attributes’ objects implement a portion of the *note mapping protocol:
11ae, including the methods ‘copy()’, ‘get()’, *note __contains__():
1f5e, ‘items()’, ‘keys()’, and ‘values()’. The following methods are
also provided:
-- Method: Attributes.getLength ()
Return the number of attributes.
-- Method: Attributes.getNames ()
Return the names of the attributes.
-- Method: Attributes.getType (name)
Returns the type of the attribute 'name', which is normally
‘'CDATA'’.
-- Method: Attributes.getValue (name)
Return the value of attribute 'name'.
File: python.info, Node: The AttributesNS Interface, Prev: The Attributes Interface, Up: xml sax xmlreader — Interface for XML parsers
5.21.12.6 The ‘AttributesNS’ Interface
......................................
This interface is a subtype of the ‘Attributes’ interface (see section
*note The Attributes Interface: 3886.). All methods supported by that
interface are also available on ‘AttributesNS’ objects.
The following methods are also available:
-- Method: AttributesNS.getValueByQName (name)
Return the value for a qualified name.
-- Method: AttributesNS.getNameByQName (name)
Return the ‘(namespace, localname)’ pair for a qualified 'name'.
-- Method: AttributesNS.getQNameByName (name)
Return the qualified name for a ‘(namespace, localname)’ pair.
-- Method: AttributesNS.getQNames ()
Return the qualified names of all attributes.
File: python.info, Node: xml parsers expat — Fast XML parsing using Expat, Prev: xml sax xmlreader — Interface for XML parsers, Up: Structured Markup Processing Tools
5.21.13 ‘xml.parsers.expat’ — Fast XML parsing using Expat
----------------------------------------------------------
__________________________________________________________________
Note: If you need to parse untrusted or unauthenticated data, see
*note XML security: 3774.
The *note xml.parsers.expat: 126. module is a Python interface to the
Expat non-validating XML parser. The module provides a single extension
type, ‘xmlparser’, that represents the current state of an XML parser.
After an ‘xmlparser’ object has been created, various attributes of the
object can be set to handler functions. When an XML document is then
fed to the parser, the handler functions are called for the character
data and markup in the XML document.
This module uses the ‘pyexpat’ module to provide access to the Expat
parser. Direct use of the ‘pyexpat’ module is deprecated.
This module provides one exception and one type object:
-- Exception: xml.parsers.expat.ExpatError
The exception raised when Expat reports an error. See section
*note ExpatError Exceptions: 38df. for more information on
interpreting Expat errors.
-- Exception: xml.parsers.expat.error
Alias for *note ExpatError: 126c.
-- Data: xml.parsers.expat.XMLParserType
The type of the return values from the *note ParserCreate(): 38e2.
function.
The *note xml.parsers.expat: 126. module contains two functions:
-- Function: xml.parsers.expat.ErrorString (errno)
Returns an explanatory string for a given error number 'errno'.
-- Function: xml.parsers.expat.ParserCreate (encoding=None,
namespace_separator=None)
Creates and returns a new ‘xmlparser’ object. 'encoding', if
specified, must be a string naming the encoding used by the XML
data. Expat doesn’t support as many encodings as Python does, and
its repertoire of encodings can’t be extended; it supports UTF-8,
UTF-16, ISO-8859-1 (Latin1), and ASCII. If 'encoding' (1) is given
it will override the implicit or explicit encoding of the document.
Expat can optionally do XML namespace processing for you, enabled
by providing a value for 'namespace_separator'. The value must be
a one-character string; a *note ValueError: 204. will be raised if
the string has an illegal length (‘None’ is considered the same as
omission). When namespace processing is enabled, element type
names and attribute names that belong to a namespace will be
expanded. The element name passed to the element handlers
‘StartElementHandler’ and ‘EndElementHandler’ will be the
concatenation of the namespace URI, the namespace separator
character, and the local part of the name. If the namespace
separator is a zero byte (‘chr(0)’) then the namespace URI and the
local part will be concatenated without any separator.
For example, if 'namespace_separator' is set to a space character
(‘' '’) and the following document is parsed:
‘StartElementHandler’ will receive the following strings for each
element:
http://default-namespace.org/ root
http://www.python.org/ns/ elem1
elem2
Due to limitations in the ‘Expat’ library used by ‘pyexpat’, the
‘xmlparser’ instance returned can only be used to parse a single
XML document. Call ‘ParserCreate’ for each document to provide
unique parser instances.
See also
........
The Expat XML Parser(2)
Home page of the Expat project.
* Menu:
* XMLParser Objects: XMLParser Objects<2>.
* ExpatError Exceptions::
* Example: Example<10>.
* Content Model Descriptions::
* Expat error constants::
---------- Footnotes ----------
(1) The encoding string included in XML output should conform to the
appropriate standards. For example, “UTF-8” is valid, but “UTF8” is
not. See
‘https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl’ and
‘https://www.iana.org/assignments/character-sets/character-sets.xhtml’.
(2) http://www.libexpat.org/
File: python.info, Node: XMLParser Objects<2>, Next: ExpatError Exceptions, Up: xml parsers expat — Fast XML parsing using Expat
5.21.13.1 XMLParser Objects
...........................
‘xmlparser’ objects have the following methods:
-- Method: xmlparser.Parse (data[, isfinal])
Parses the contents of the string 'data', calling the appropriate
handler functions to process the parsed data. 'isfinal' must be
true on the final call to this method; it allows the parsing of a
single file in fragments, not the submission of multiple files.
'data' can be the empty string at any time.
-- Method: xmlparser.ParseFile (file)
Parse XML data reading from the object 'file'. 'file' only needs
to provide the ‘read(nbytes)’ method, returning the empty string
when there’s no more data.
-- Method: xmlparser.SetBase (base)
Sets the base to be used for resolving relative URIs in system
identifiers in declarations. Resolving relative identifiers is
left to the application: this value will be passed through as the
'base' argument to the *note ExternalEntityRefHandler(): 38e9,
*note NotationDeclHandler(): 38ea, and *note
UnparsedEntityDeclHandler(): 38eb. functions.
-- Method: xmlparser.GetBase ()
Returns a string containing the base set by a previous call to
*note SetBase(): 38e8, or ‘None’ if *note SetBase(): 38e8. hasn’t
been called.
-- Method: xmlparser.GetInputContext ()
Returns the input data that generated the current event as a
string. The data is in the encoding of the entity which contains
the text. When called while an event handler is not active, the
return value is ‘None’.
-- Method: xmlparser.ExternalEntityParserCreate (context[, encoding])
Create a “child” parser which can be used to parse an external
parsed entity referred to by content parsed by the parent parser.
The 'context' parameter should be the string passed to the *note
ExternalEntityRefHandler(): 38e9. handler function, described
below. The child parser is created with the *note
ordered_attributes: 38ef. and *note specified_attributes: 38f0. set
to the values of this parser.
-- Method: xmlparser.SetParamEntityParsing (flag)
Control parsing of parameter entities (including the external DTD
subset). Possible 'flag' values are
‘XML_PARAM_ENTITY_PARSING_NEVER’,
‘XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE’ and
‘XML_PARAM_ENTITY_PARSING_ALWAYS’. Return true if setting the flag
was successful.
-- Method: xmlparser.UseForeignDTD ([flag])
Calling this with a true value for 'flag' (the default) will cause
Expat to call the *note ExternalEntityRefHandler: 38e9. with *note
None: 671. for all arguments to allow an alternate DTD to be
loaded. If the document does not contain a document type
declaration, the *note ExternalEntityRefHandler: 38e9. will still
be called, but the *note StartDoctypeDeclHandler: 38f3. and *note
EndDoctypeDeclHandler: 38f4. will not be called.
Passing a false value for 'flag' will cancel a previous call that
passed a true value, but otherwise has no effect.
This method can only be called before the *note Parse(): 38e6. or
*note ParseFile(): 38e7. methods are called; calling it after
either of those have been called causes *note ExpatError: 126c. to
be raised with the *note code: 1a. attribute set to
‘errors.codes[errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING]’.
-- Method: xmlparser.SetReparseDeferralEnabled (enabled)
Warning: Calling ‘SetReparseDeferralEnabled(False)’ has
security implications, as detailed below; please make sure to
understand these consequences prior to using the
‘SetReparseDeferralEnabled’ method.
Expat 2.6.0 introduced a security mechanism called “reparse
deferral” where instead of causing denial of service through
quadratic runtime from reparsing large tokens, reparsing of
unfinished tokens is now delayed by default until a sufficient
amount of input is reached. Due to this delay, registered handlers
may — depending of the sizing of input chunks pushed to Expat — no
longer be called right after pushing new input to the parser.
Where immediate feedback and taking over responsibility of
protecting against denial of service from large tokens are both
wanted, calling ‘SetReparseDeferralEnabled(False)’ disables reparse
deferral for the current Expat parser instance, temporarily or
altogether. Calling ‘SetReparseDeferralEnabled(True)’ allows
re-enabling reparse deferral.
Note that *note SetReparseDeferralEnabled(): 274. has been
backported to some prior releases of CPython as a security fix.
Check for availability of *note SetReparseDeferralEnabled(): 274.
using *note hasattr(): 4ce. if used in code running across a
variety of Python versions.
Added in version 3.13.
-- Method: xmlparser.GetReparseDeferralEnabled ()
Returns whether reparse deferral is currently enabled for the given
Expat parser instance.
Added in version 3.13.
‘xmlparser’ objects have the following attributes:
-- Attribute: xmlparser.buffer_size
The size of the buffer used when *note buffer_text: 148b. is true.
A new buffer size can be set by assigning a new integer value to
this attribute. When the size is changed, the buffer will be
flushed.
-- Attribute: xmlparser.buffer_text
Setting this to true causes the ‘xmlparser’ object to buffer
textual content returned by Expat to avoid multiple calls to the
*note CharacterDataHandler(): 38f6. callback whenever possible.
This can improve performance substantially since Expat normally
breaks character data into chunks at every line ending. This
attribute is false by default, and may be changed at any time.
Note that when it is false, data that does not contain newlines may
be chunked too.
-- Attribute: xmlparser.buffer_used
If *note buffer_text: 148b. is enabled, the number of bytes stored
in the buffer. These bytes represent UTF-8 encoded text. This
attribute has no meaningful interpretation when *note buffer_text:
148b. is false.
-- Attribute: xmlparser.ordered_attributes
Setting this attribute to a non-zero integer causes the attributes
to be reported as a list rather than a dictionary. The attributes
are presented in the order found in the document text. For each
attribute, two list entries are presented: the attribute name and
the attribute value. (Older versions of this module also used this
format.) By default, this attribute is false; it may be changed at
any time.
-- Attribute: xmlparser.specified_attributes
If set to a non-zero integer, the parser will report only those
attributes which were specified in the document instance and not
those which were derived from attribute declarations. Applications
which set this need to be especially careful to use what additional
information is available from the declarations as needed to comply
with the standards for the behavior of XML processors. By default,
this attribute is false; it may be changed at any time.
The following attributes contain values relating to the most recent
error encountered by an ‘xmlparser’ object, and will only have correct
values once a call to ‘Parse()’ or ‘ParseFile()’ has raised an *note
xml.parsers.expat.ExpatError: 126c. exception.
-- Attribute: xmlparser.ErrorByteIndex
Byte index at which an error occurred.
-- Attribute: xmlparser.ErrorCode
Numeric code specifying the problem. This value can be passed to
the *note ErrorString(): 38e3. function, or compared to one of the
constants defined in the ‘errors’ object.
-- Attribute: xmlparser.ErrorColumnNumber
Column number at which an error occurred.
-- Attribute: xmlparser.ErrorLineNumber
Line number at which an error occurred.
The following attributes contain values relating to the current parse
location in an ‘xmlparser’ object. During a callback reporting a parse
event they indicate the location of the first of the sequence of
characters that generated the event. When called outside of a callback,
the position indicated will be just past the last parse event
(regardless of whether there was an associated callback).
-- Attribute: xmlparser.CurrentByteIndex
Current byte index in the parser input.
-- Attribute: xmlparser.CurrentColumnNumber
Current column number in the parser input.
-- Attribute: xmlparser.CurrentLineNumber
Current line number in the parser input.
Here is the list of handlers that can be set. To set a handler on an
‘xmlparser’ object 'o', use ‘o.handlername = func’. 'handlername' must
be taken from the following list, and 'func' must be a callable object
accepting the correct number of arguments. The arguments are all
strings, unless otherwise stated.
-- Method: xmlparser.XmlDeclHandler (version, encoding, standalone)
Called when the XML declaration is parsed. The XML declaration is
the (optional) declaration of the applicable version of the XML
recommendation, the encoding of the document text, and an optional
“standalone” declaration. 'version' and 'encoding' will be
strings, and 'standalone' will be ‘1’ if the document is declared
standalone, ‘0’ if it is declared not to be standalone, or ‘-1’ if
the standalone clause was omitted. This is only available with
Expat version 1.95.0 or newer.
-- Method: xmlparser.StartDoctypeDeclHandler (doctypeName, systemId,
publicId, has_internal_subset)
Called when Expat begins parsing the document type declaration
(‘'’.
-- Method: xmlparser.StartCdataSectionHandler ()
Called at the start of a CDATA section. This and *note
EndCdataSectionHandler: 3906. are needed to be able to identify the
syntactical start and end for CDATA sections.
-- Method: xmlparser.EndCdataSectionHandler ()
Called at the end of a CDATA section.
-- Method: xmlparser.DefaultHandler (data)
Called for any characters in the XML document for which no
applicable handler has been specified. This means characters that
are part of a construct which could be reported, but for which no
handler has been supplied.
-- Method: xmlparser.DefaultHandlerExpand (data)
This is the same as the *note DefaultHandler(): 390b, but doesn’t
inhibit expansion of internal entities. The entity reference will
not be passed to the default handler.
-- Method: xmlparser.NotStandaloneHandler ()
Called if the XML document hasn’t been declared as being a
standalone document. This happens when there is an external subset
or a reference to a parameter entity, but the XML declaration does
not set standalone to ‘yes’ in an XML declaration. If this handler
returns ‘0’, then the parser will raise an
‘XML_ERROR_NOT_STANDALONE’ error. If this handler is not set, no
exception is raised by the parser for this condition.
-- Method: xmlparser.ExternalEntityRefHandler (context, base, systemId,
publicId)
Called for references to external entities. 'base' is the current
base, as set by a previous call to *note SetBase(): 38e8. The
public and system identifiers, 'systemId' and 'publicId', are
strings if given; if the public identifier is not given, 'publicId'
will be ‘None’. The 'context' value is opaque and should only be
used as described below.
For external entities to be parsed, this handler must be
implemented. It is responsible for creating the sub-parser using
‘ExternalEntityParserCreate(context)’, initializing it with the
appropriate callbacks, and parsing the entity. This handler should
return an integer; if it returns ‘0’, the parser will raise an
‘XML_ERROR_EXTERNAL_ENTITY_HANDLING’ error, otherwise parsing will
continue.
If this handler is not provided, external entities are reported by
the *note DefaultHandler: 390b. callback, if provided.
File: python.info, Node: ExpatError Exceptions, Next: Example<10>, Prev: XMLParser Objects<2>, Up: xml parsers expat — Fast XML parsing using Expat
5.21.13.2 ExpatError Exceptions
...............................
*note ExpatError: 126c. exceptions have a number of interesting
attributes:
-- Attribute: ExpatError.code
Expat’s internal error number for the specific error. The *note
errors.messages: 3910. dictionary maps these error numbers to
Expat’s error messages. For example:
from xml.parsers.expat import ParserCreate, ExpatError, errors
p = ParserCreate()
try:
p.Parse(some_xml_document)
except ExpatError as err:
print("Error:", errors.messages[err.code])
The *note errors: 127. module also provides error message constants
and a dictionary *note codes: 3911. mapping these messages back to
the error codes, see below.
-- Attribute: ExpatError.lineno
Line number on which the error was detected. The first line is
numbered ‘1’.
-- Attribute: ExpatError.offset
Character offset into the line where the error occurred. The first
column is numbered ‘0’.
File: python.info, Node: Example<10>, Next: Content Model Descriptions, Prev: ExpatError Exceptions, Up: xml parsers expat — Fast XML parsing using Expat
5.21.13.3 Example
.................
The following program defines three handlers that just print out their
arguments.
import xml.parsers.expat
# 3 handler functions
def start_element(name, attrs):
print('Start element:', name, attrs)
def end_element(name):
print('End element:', name)
def char_data(data):
print('Character data:', repr(data))
p = xml.parsers.expat.ParserCreate()
p.StartElementHandler = start_element
p.EndElementHandler = end_element
p.CharacterDataHandler = char_data
p.Parse("""
Text goes here
More text
""", 1)
The output from this program is:
Start element: parent {'id': 'top'}
Start element: child1 {'name': 'paul'}
Character data: 'Text goes here'
End element: child1
Character data: '\n'
Start element: child2 {'name': 'fred'}
Character data: 'More text'
End element: child2
Character data: '\n'
End element: parent
File: python.info, Node: Content Model Descriptions, Next: Expat error constants, Prev: Example<10>, Up: xml parsers expat — Fast XML parsing using Expat
5.21.13.4 Content Model Descriptions
....................................
Content models are described using nested tuples. Each tuple contains
four values: the type, the quantifier, the name, and a tuple of
children. Children are simply additional content model descriptions.
The values of the first two fields are constants defined in the *note
xml.parsers.expat.model: 128. module. These constants can be collected
in two groups: the model type group and the quantifier group.
The constants in the model type group are:
-- Data: xml.parsers.expat.model.XML_CTYPE_ANY
The element named by the model name was declared to have a content
model of ‘ANY’.
-- Data: xml.parsers.expat.model.XML_CTYPE_CHOICE
The named element allows a choice from a number of options; this is
used for content models such as ‘(A | B | C)’.
-- Data: xml.parsers.expat.model.XML_CTYPE_EMPTY
Elements which are declared to be ‘EMPTY’ have this model type.
-- Data: xml.parsers.expat.model.XML_CTYPE_MIXED
-- Data: xml.parsers.expat.model.XML_CTYPE_NAME
-- Data: xml.parsers.expat.model.XML_CTYPE_SEQ
Models which represent a series of models which follow one after
the other are indicated with this model type. This is used for
models such as ‘(A, B, C)’.
The constants in the quantifier group are:
-- Data: xml.parsers.expat.model.XML_CQUANT_NONE
No modifier is given, so it can appear exactly once, as for ‘A’.
-- Data: xml.parsers.expat.model.XML_CQUANT_OPT
The model is optional: it can appear once or not at all, as for
‘A?’.
-- Data: xml.parsers.expat.model.XML_CQUANT_PLUS
The model must occur one or more times (like ‘A+’).
-- Data: xml.parsers.expat.model.XML_CQUANT_REP
The model must occur zero or more times, as for ‘A*’.
File: python.info, Node: Expat error constants, Prev: Content Model Descriptions, Up: xml parsers expat — Fast XML parsing using Expat
5.21.13.5 Expat error constants
...............................
The following constants are provided in the *note
xml.parsers.expat.errors: 127. module. These constants are useful in
interpreting some of the attributes of the ‘ExpatError’ exception
objects raised when an error has occurred. Since for backwards
compatibility reasons, the constants’ value is the error 'message' and
not the numeric error 'code', you do this by comparing its *note code:
1a. attribute with ‘errors.codes[errors.XML_ERROR_CONSTANT_NAME]’.
The ‘errors’ module has the following attributes:
-- Data: xml.parsers.expat.errors.codes
A dictionary mapping string descriptions to their error codes.
Added in version 3.2.
-- Data: xml.parsers.expat.errors.messages
A dictionary mapping numeric error codes to their string
descriptions.
Added in version 3.2.
-- Data: xml.parsers.expat.errors.XML_ERROR_ASYNC_ENTITY
-- Data:
xml.parsers.expat.errors.XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF
An entity reference in an attribute value referred to an external
entity instead of an internal entity.
-- Data: xml.parsers.expat.errors.XML_ERROR_BAD_CHAR_REF
A character reference referred to a character which is illegal in
XML (for example, character ‘0’, or ‘‘’’).
-- Data: xml.parsers.expat.errors.XML_ERROR_BINARY_ENTITY_REF
An entity reference referred to an entity which was declared with a
notation, so cannot be parsed.
-- Data: xml.parsers.expat.errors.XML_ERROR_DUPLICATE_ATTRIBUTE
An attribute was used more than once in a start tag.
-- Data: xml.parsers.expat.errors.XML_ERROR_INCORRECT_ENCODING
-- Data: xml.parsers.expat.errors.XML_ERROR_INVALID_TOKEN
Raised when an input byte could not properly be assigned to a
character; for example, a NUL byte (value ‘0’) in a UTF-8 input
stream.
-- Data: xml.parsers.expat.errors.XML_ERROR_JUNK_AFTER_DOC_ELEMENT
Something other than whitespace occurred after the document
element.
-- Data: xml.parsers.expat.errors.XML_ERROR_MISPLACED_XML_PI
An XML declaration was found somewhere other than the start of the
input data.
-- Data: xml.parsers.expat.errors.XML_ERROR_NO_ELEMENTS
The document contains no elements (XML requires all documents to
contain exactly one top-level element)..
-- Data: xml.parsers.expat.errors.XML_ERROR_NO_MEMORY
Expat was not able to allocate memory internally.
-- Data: xml.parsers.expat.errors.XML_ERROR_PARAM_ENTITY_REF
A parameter entity reference was found where it was not allowed.
-- Data: xml.parsers.expat.errors.XML_ERROR_PARTIAL_CHAR
An incomplete character was found in the input.
-- Data: xml.parsers.expat.errors.XML_ERROR_RECURSIVE_ENTITY_REF
An entity reference contained another reference to the same entity;
possibly via a different name, and possibly indirectly.
-- Data: xml.parsers.expat.errors.XML_ERROR_SYNTAX
Some unspecified syntax error was encountered.
-- Data: xml.parsers.expat.errors.XML_ERROR_TAG_MISMATCH
An end tag did not match the innermost open start tag.
-- Data: xml.parsers.expat.errors.XML_ERROR_UNCLOSED_TOKEN
Some token (such as a start tag) was not closed before the end of
the stream or the next token was encountered.
-- Data: xml.parsers.expat.errors.XML_ERROR_UNDEFINED_ENTITY
A reference was made to an entity which was not defined.
-- Data: xml.parsers.expat.errors.XML_ERROR_UNKNOWN_ENCODING
The document encoding is not supported by Expat.
-- Data: xml.parsers.expat.errors.XML_ERROR_UNCLOSED_CDATA_SECTION
A CDATA marked section was not closed.
-- Data: xml.parsers.expat.errors.XML_ERROR_EXTERNAL_ENTITY_HANDLING
-- Data: xml.parsers.expat.errors.XML_ERROR_NOT_STANDALONE
The parser determined that the document was not “standalone” though
it declared itself to be in the XML declaration, and the
‘NotStandaloneHandler’ was set and returned ‘0’.
-- Data: xml.parsers.expat.errors.XML_ERROR_UNEXPECTED_STATE
-- Data: xml.parsers.expat.errors.XML_ERROR_ENTITY_DECLARED_IN_PE
-- Data: xml.parsers.expat.errors.XML_ERROR_FEATURE_REQUIRES_XML_DTD
An operation was requested that requires DTD support to be compiled
in, but Expat was configured without DTD support. This should
never be reported by a standard build of the *note
xml.parsers.expat: 126. module.
-- Data:
xml.parsers.expat.errors.XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING
A behavioral change was requested after parsing started that can
only be changed before parsing has started. This is (currently)
only raised by ‘UseForeignDTD()’.
-- Data: xml.parsers.expat.errors.XML_ERROR_UNBOUND_PREFIX
An undeclared prefix was found when namespace processing was
enabled.
-- Data: xml.parsers.expat.errors.XML_ERROR_UNDECLARING_PREFIX
The document attempted to remove the namespace declaration
associated with a prefix.
-- Data: xml.parsers.expat.errors.XML_ERROR_INCOMPLETE_PE
A parameter entity contained incomplete markup.
-- Data: xml.parsers.expat.errors.XML_ERROR_XML_DECL
The document contained no document element at all.
-- Data: xml.parsers.expat.errors.XML_ERROR_TEXT_DECL
There was an error parsing a text declaration in an external
entity.
-- Data: xml.parsers.expat.errors.XML_ERROR_PUBLICID
Characters were found in the public id that are not allowed.
-- Data: xml.parsers.expat.errors.XML_ERROR_SUSPENDED
The requested operation was made on a suspended parser, but isn’t
allowed. This includes attempts to provide additional input or to
stop the parser.
-- Data: xml.parsers.expat.errors.XML_ERROR_NOT_SUSPENDED
An attempt to resume the parser was made when the parser had not
been suspended.
-- Data: xml.parsers.expat.errors.XML_ERROR_ABORTED
This should not be reported to Python applications.
-- Data: xml.parsers.expat.errors.XML_ERROR_FINISHED
The requested operation was made on a parser which was finished
parsing input, but isn’t allowed. This includes attempts to
provide additional input or to stop the parser.
-- Data: xml.parsers.expat.errors.XML_ERROR_SUSPEND_PE
-- Data: xml.parsers.expat.errors.XML_ERROR_RESERVED_PREFIX_XML
An attempt was made to undeclare reserved namespace prefix ‘xml’ or
to bind it to another namespace URI.
-- Data: xml.parsers.expat.errors.XML_ERROR_RESERVED_PREFIX_XMLNS
An attempt was made to declare or undeclare reserved namespace
prefix ‘xmlns’.
-- Data: xml.parsers.expat.errors.XML_ERROR_RESERVED_NAMESPACE_URI
An attempt was made to bind the URI of one the reserved namespace
prefixes ‘xml’ and ‘xmlns’ to another namespace prefix.
-- Data: xml.parsers.expat.errors.XML_ERROR_INVALID_ARGUMENT
This should not be reported to Python applications.
-- Data: xml.parsers.expat.errors.XML_ERROR_NO_BUFFER
This should not be reported to Python applications.
-- Data: xml.parsers.expat.errors.XML_ERROR_AMPLIFICATION_LIMIT_BREACH
The limit on input amplification factor (from DTD and entities) has
been breached.
File: python.info, Node: Internet Protocols and Support, Next: Multimedia Services, Prev: Structured Markup Processing Tools, Up: The Python Standard Library
5.22 Internet Protocols and Support
===================================
The modules described in this chapter implement internet protocols and
support for related technology. They are all implemented in Python.
Most of these modules require the presence of the system-dependent
module *note socket: cc, which is currently supported on most popular
platforms. Here is an overview:
* Menu:
* webbrowser — Convenient web-browser controller::
* wsgiref — WSGI Utilities and Reference Implementation::
* urllib — URL handling modules::
* urllib.request — Extensible library for opening URLs: urllib request — Extensible library for opening URLs.
* urllib.response — Response classes used by urllib: urllib response — Response classes used by urllib.
* urllib.parse — Parse URLs into components: urllib parse — Parse URLs into components.
* urllib.error — Exception classes raised by urllib.request: urllib error — Exception classes raised by urllib request.
* urllib.robotparser — Parser for robots.txt: urllib robotparser — Parser for robots txt.
* http — HTTP modules::
* http.client — HTTP protocol client: http client — HTTP protocol client.
* ftplib — FTP protocol client::
* poplib — POP3 protocol client::
* imaplib — IMAP4 protocol client::
* smtplib — SMTP protocol client::
* uuid — UUID objects according to RFC 4122::
* socketserver — A framework for network servers::
* http.server — HTTP servers: http server — HTTP servers.
* http.cookies — HTTP state management: http cookies — HTTP state management.
* http.cookiejar — Cookie handling for HTTP clients: http cookiejar — Cookie handling for HTTP clients.
* xmlrpc — XMLRPC server and client modules::
* xmlrpc.client — XML-RPC client access: xmlrpc client — XML-RPC client access.
* xmlrpc.server — Basic XML-RPC servers: xmlrpc server — Basic XML-RPC servers.
* ipaddress — IPv4/IPv6 manipulation library::
File: python.info, Node: webbrowser — Convenient web-browser controller, Next: wsgiref — WSGI Utilities and Reference Implementation, Up: Internet Protocols and Support
5.22.1 ‘webbrowser’ — Convenient web-browser controller
-------------------------------------------------------
'Source code:' Lib/webbrowser.py(1)
__________________________________________________________________
The *note webbrowser: 115. module provides a high-level interface to
allow displaying web-based documents to users. Under most
circumstances, simply calling the *note open(): 394a. function from this
module will do the right thing.
Under Unix, graphical browsers are preferred under X11, but text-mode
browsers will be used if graphical browsers are not available or an X11
display isn’t available. If text-mode browsers are used, the calling
process will block until the user exits the browser.
If the environment variable ‘BROWSER’ exists, it is interpreted as the
*note os.pathsep: 1d44.-separated list of browsers to try ahead of the
platform defaults. When the value of a list part contains the string
‘%s’, then it is interpreted as a literal browser command line to be
used with the argument URL substituted for ‘%s’; if the part does not
contain ‘%s’, it is simply interpreted as the name of the browser to
launch. (2)
For non-Unix platforms, or when a remote browser is available on Unix,
the controlling process will not wait for the user to finish with the
browser, but allow the remote browser to maintain its own windows on the
display. If remote browsers are not available on Unix, the controlling
process will launch a new browser and wait.
On iOS, the ‘BROWSER’ environment variable, as well as any arguments
controlling autoraise, browser preference, and new tab/window creation
will be ignored. Web pages will 'always' be opened in the user’s
preferred browser, in a new tab, with the browser being brought to the
foreground. The use of the *note webbrowser: 115. module on iOS
requires the *note ctypes: 2a. module. If *note ctypes: 2a. isn’t
available, calls to *note open(): 394a. will fail.
The script ‘webbrowser’ can be used as a command-line interface for the
module. It accepts a URL as the argument. It accepts the following
optional parameters:
-- Option: -n, --new-window
Opens the URL in a new browser window, if possible.
-- Option: -t, --new-tab
Opens the URL in a new browser tab.
The options are, naturally, mutually exclusive. Usage example:
python -m webbrowser -t "https://www.python.org"
*note Availability: 1d54.: not WASI, not Android.
The following exception is defined:
-- Exception: webbrowser.Error
Exception raised when a browser control error occurs.
The following functions are defined:
-- Function: webbrowser.open (url, new=0, autoraise=True)
Display 'url' using the default browser. If 'new' is 0, the 'url'
is opened in the same browser window if possible. If 'new' is 1, a
new browser window is opened if possible. If 'new' is 2, a new
browser page (“tab”) is opened if possible. If 'autoraise' is
‘True’, the window is raised if possible (note that under many
window managers this will occur regardless of the setting of this
variable).
Returns ‘True’ if a browser was successfully launched, ‘False’
otherwise.
Note that on some platforms, trying to open a filename using this
function, may work and start the operating system’s associated
program. However, this is neither supported nor portable.
Raises an *note auditing event: 18ba. ‘webbrowser.open’ with
argument ‘url’.
-- Function: webbrowser.open_new (url)
Open 'url' in a new window of the default browser, if possible,
otherwise, open 'url' in the only browser window.
Returns ‘True’ if a browser was successfully launched, ‘False’
otherwise.
-- Function: webbrowser.open_new_tab (url)
Open 'url' in a new page (“tab”) of the default browser, if
possible, otherwise equivalent to *note open_new(): 3950.
Returns ‘True’ if a browser was successfully launched, ‘False’
otherwise.
-- Function: webbrowser.get (using=None)
Return a controller object for the browser type 'using'. If
'using' is ‘None’, return a controller for a default browser
appropriate to the caller’s environment.
-- Function: webbrowser.register (name, constructor, instance=None, *,
preferred=False)
Register the browser type 'name'. Once a browser type is
registered, the *note get(): 3952. function can return a controller
for that browser type. If 'instance' is not provided, or is
‘None’, 'constructor' will be called without parameters to create
an instance when needed. If 'instance' is provided, 'constructor'
will never be called, and may be ‘None’.
Setting 'preferred' to ‘True’ makes this browser a preferred result
for a *note get(): 3952. call with no argument. Otherwise, this
entry point is only useful if you plan to either set the ‘BROWSER’
variable or call *note get(): 3952. with a nonempty argument
matching the name of a handler you declare.
Changed in version 3.7: 'preferred' keyword-only parameter was
added.
A number of browser types are predefined. This table gives the type
names that may be passed to the *note get(): 3952. function and the
corresponding instantiations for the controller classes, all defined in
this module.
Type Name Class Name Notes
---------------------------------------------------------------------------------------
‘'mozilla'’ ‘Mozilla('mozilla')’
‘'firefox'’ ‘Mozilla('mozilla')’
‘'epiphany'’ ‘Epiphany('epiphany')’
‘'kfmclient'’ ‘Konqueror()’ (1)
‘'konqueror'’ ‘Konqueror()’ (1)
‘'kfm'’ ‘Konqueror()’ (1)
‘'opera'’ ‘Opera()’
‘'links'’ ‘GenericBrowser('links')’
‘'elinks'’ ‘Elinks('elinks')’
‘'lynx'’ ‘GenericBrowser('lynx')’
‘'w3m'’ ‘GenericBrowser('w3m')’
‘'windows-default'’ ‘WindowsDefault’ (2)
‘'macosx'’ ‘MacOSXOSAScript('default')’ (3)
‘'safari'’ ‘MacOSXOSAScript('safari')’ (3)
‘'google-chrome'’ ‘Chrome('google-chrome')’
‘'chrome'’ ‘Chrome('chrome')’
‘'chromium'’ ‘Chromium('chromium')’
‘'chromium-browser'’ ‘Chromium('chromium-browser')’
‘'iosbrowser'’ ‘IOSBrowser’ (4)
Notes:
1. “Konqueror” is the file manager for the KDE desktop environment for
Unix, and only makes sense to use if KDE is running. Some way of
reliably detecting KDE would be nice; the ‘KDEDIR’ variable is not
sufficient. Note also that the name “kfm” is used even when using
the ‘konqueror’ command with KDE 2 — the implementation selects the
best strategy for running Konqueror.
2. Only on Windows platforms.
3. Only on macOS.
4. Only on iOS.
Added in version 3.2: A new ‘MacOSXOSAScript’ class has been added and
is used on Mac instead of the previous ‘MacOSX’ class. This adds
support for opening browsers not currently set as the OS default.
Added in version 3.3: Support for Chrome/Chromium has been added.
Changed in version 3.12: Support for several obsolete browsers has been
removed. Removed browsers include Grail, Mosaic, Netscape, Galeon,
Skipstone, Iceape, and Firefox versions 35 and below.
Changed in version 3.13: Support for iOS has been added.
Here are some simple examples:
url = 'https://docs.python.org/'
# Open URL in a new tab, if a browser window is already open.
webbrowser.open_new_tab(url)
# Open URL in new window, raising the window if possible.
webbrowser.open_new(url)
* Menu:
* Browser Controller Objects::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/webbrowser.py
(2) Executables named here without a full path will be searched in
the directories given in the ‘PATH’ environment variable.
File: python.info, Node: Browser Controller Objects, Up: webbrowser — Convenient web-browser controller
5.22.1.1 Browser Controller Objects
...................................
Browser controllers provide the *note name: 299. attribute, and the
following three methods which parallel module-level convenience
functions:
-- Attribute: controller.name
System-dependent name for the browser.
-- Method: controller.open (url, new=0, autoraise=True)
Display 'url' using the browser handled by this controller. If
'new' is 1, a new browser window is opened if possible. If 'new'
is 2, a new browser page (“tab”) is opened if possible.
-- Method: controller.open_new (url)
Open 'url' in a new window of the browser handled by this
controller, if possible, otherwise, open 'url' in the only browser
window. Alias *note open_new(): 3950.
-- Method: controller.open_new_tab (url)
Open 'url' in a new page (“tab”) of the browser handled by this
controller, if possible, otherwise equivalent to *note open_new():
3950.
File: python.info, Node: wsgiref — WSGI Utilities and Reference Implementation, Next: urllib — URL handling modules, Prev: webbrowser — Convenient web-browser controller, Up: Internet Protocols and Support
5.22.2 ‘wsgiref’ — WSGI Utilities and Reference Implementation
--------------------------------------------------------------
'Source code:' Lib/wsgiref(1)
__________________________________________________________________
The Web Server Gateway Interface (WSGI) is a standard interface between
web server software and web applications written in Python. Having a
standard interface makes it easy to use an application that supports
WSGI with a number of different web servers.
Only authors of web servers and programming frameworks need to know
every detail and corner case of the WSGI design. You don’t need to
understand every detail of WSGI just to install a WSGI application or to
write a web application using an existing framework.
*note wsgiref: 118. is a reference implementation of the WSGI
specification that can be used to add WSGI support to a web server or
framework. It provides utilities for manipulating WSGI environment
variables and response headers, base classes for implementing WSGI
servers, a demo HTTP server that serves WSGI applications, types for
static type checking, and a validation tool that checks WSGI servers and
applications for conformance to the WSGI specification ( PEP 3333(2)).
See wsgi.readthedocs.io(3) for more information about WSGI, and links to
tutorials and other resources.
* Menu:
* wsgiref.util – WSGI environment utilities: wsgiref util – WSGI environment utilities.
* wsgiref.headers – WSGI response header tools: wsgiref headers – WSGI response header tools.
* wsgiref.simple_server – a simple WSGI HTTP server: wsgiref simple_server – a simple WSGI HTTP server.
* wsgiref.validate — WSGI conformance checker: wsgiref validate — WSGI conformance checker.
* wsgiref.handlers – server/gateway base classes: wsgiref handlers – server/gateway base classes.
* wsgiref.types – WSGI types for static type checking: wsgiref types – WSGI types for static type checking.
* Examples: Examples<25>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/wsgiref
(2) https://peps.python.org/pep-3333/
(3) https://wsgi.readthedocs.io/
File: python.info, Node: wsgiref util – WSGI environment utilities, Next: wsgiref headers – WSGI response header tools, Up: wsgiref — WSGI Utilities and Reference Implementation
5.22.2.1 ‘wsgiref.util’ – WSGI environment utilities
....................................................
This module provides a variety of utility functions for working with
WSGI environments. A WSGI environment is a dictionary containing HTTP
request variables as described in PEP 3333(1). All of the functions
taking an 'environ' parameter expect a WSGI-compliant dictionary to be
supplied; please see PEP 3333(2) for a detailed specification and *note
WSGIEnvironment: 395c. for a type alias that can be used in type
annotations.
-- Function: wsgiref.util.guess_scheme (environ)
Return a guess for whether ‘wsgi.url_scheme’ should be “http” or
“https”, by checking for a ‘HTTPS’ environment variable in the
'environ' dictionary. The return value is a string.
This function is useful when creating a gateway that wraps CGI or a
CGI-like protocol such as FastCGI. Typically, servers providing
such protocols will include a ‘HTTPS’ variable with a value of “1”,
“yes”, or “on” when a request is received via SSL. So, this
function returns “https” if such a value is found, and “http”
otherwise.
-- Function: wsgiref.util.request_uri (environ, include_query=True)
Return the full request URI, optionally including the query string,
using the algorithm found in the “URL Reconstruction” section of
PEP 3333(3). If 'include_query' is false, the query string is not
included in the resulting URI.
-- Function: wsgiref.util.application_uri (environ)
Similar to *note request_uri(): 395e, except that the ‘PATH_INFO’
and ‘QUERY_STRING’ variables are ignored. The result is the base
URI of the application object addressed by the request.
-- Function: wsgiref.util.shift_path_info (environ)
Shift a single name from ‘PATH_INFO’ to ‘SCRIPT_NAME’ and return
the name. The 'environ' dictionary is 'modified' in-place; use a
copy if you need to keep the original ‘PATH_INFO’ or ‘SCRIPT_NAME’
intact.
If there are no remaining path segments in ‘PATH_INFO’, ‘None’ is
returned.
Typically, this routine is used to process each portion of a
request URI path, for example to treat the path as a series of
dictionary keys. This routine modifies the passed-in environment
to make it suitable for invoking another WSGI application that is
located at the target URI. For example, if there is a WSGI
application at ‘/foo’, and the request URI path is ‘/foo/bar/baz’,
and the WSGI application at ‘/foo’ calls *note shift_path_info():
3960, it will receive the string “bar”, and the environment will be
updated to be suitable for passing to a WSGI application at
‘/foo/bar’. That is, ‘SCRIPT_NAME’ will change from ‘/foo’ to
‘/foo/bar’, and ‘PATH_INFO’ will change from ‘/bar/baz’ to ‘/baz’.
When ‘PATH_INFO’ is just a “/”, this routine returns an empty
string and appends a trailing slash to ‘SCRIPT_NAME’, even though
empty path segments are normally ignored, and ‘SCRIPT_NAME’ doesn’t
normally end in a slash. This is intentional behavior, to ensure
that an application can tell the difference between URIs ending in
‘/x’ from ones ending in ‘/x/’ when using this routine to do object
traversal.
-- Function: wsgiref.util.setup_testing_defaults (environ)
Update 'environ' with trivial defaults for testing purposes.
This routine adds various parameters required for WSGI, including
‘HTTP_HOST’, ‘SERVER_NAME’, ‘SERVER_PORT’, ‘REQUEST_METHOD’,
‘SCRIPT_NAME’, ‘PATH_INFO’, and all of the PEP 3333(4)-defined
‘wsgi.*’ variables. It only supplies default values, and does not
replace any existing settings for these variables.
This routine is intended to make it easier for unit tests of WSGI
servers and applications to set up dummy environments. It should
NOT be used by actual WSGI servers or applications, since the data
is fake!
Example usage (see also *note demo_app(): 3962. for another
example):
from wsgiref.util import setup_testing_defaults
from wsgiref.simple_server import make_server
# A relatively simple WSGI application. It's going to print out the
# environment dictionary after being updated by setup_testing_defaults
def simple_app(environ, start_response):
setup_testing_defaults(environ)
status = '200 OK'
headers = [('Content-type', 'text/plain; charset=utf-8')]
start_response(status, headers)
ret = [("%s: %s\n" % (key, value)).encode("utf-8")
for key, value in environ.items()]
return ret
with make_server('', 8000, simple_app) as httpd:
print("Serving on port 8000...")
httpd.serve_forever()
In addition to the environment functions above, the *note wsgiref.util:
11d. module also provides these miscellaneous utilities:
-- Function: wsgiref.util.is_hop_by_hop (header_name)
Return ‘True’ if ‘header_name’ is an HTTP/1.1 “Hop-by-Hop” header,
as defined by RFC 2616(5).
-- Class: wsgiref.util.FileWrapper (filelike, blksize=8192)
A concrete implementation of the *note wsgiref.types.FileWrapper:
3963. protocol used to convert a file-like object to an *note
iterator: 1ac. The resulting objects are *note iterable: 121a.s.
As the object is iterated over, the optional 'blksize' parameter
will be repeatedly passed to the 'filelike' object’s ‘read()’
method to obtain bytestrings to yield. When ‘read()’ returns an
empty bytestring, iteration is ended and is not resumable.
If 'filelike' has a ‘close()’ method, the returned object will also
have a ‘close()’ method, and it will invoke the 'filelike' object’s
‘close()’ method when called.
Example usage:
from io import StringIO
from wsgiref.util import FileWrapper
# We're using a StringIO-buffer for as the file-like object
filelike = StringIO("This is an example file-like object"*10)
wrapper = FileWrapper(filelike, blksize=5)
for chunk in wrapper:
print(chunk)
Changed in version 3.11: Support for *note __getitem__(): 285.
method has been removed.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3333/
(2) https://peps.python.org/pep-3333/
(3) https://peps.python.org/pep-3333/
(4) https://peps.python.org/pep-3333/
(5) https://datatracker.ietf.org/doc/html/rfc2616.html
File: python.info, Node: wsgiref headers – WSGI response header tools, Next: wsgiref simple_server – a simple WSGI HTTP server, Prev: wsgiref util – WSGI environment utilities, Up: wsgiref — WSGI Utilities and Reference Implementation
5.22.2.2 ‘wsgiref.headers’ – WSGI response header tools
.......................................................
This module provides a single class, *note Headers: e91, for convenient
manipulation of WSGI response headers using a mapping-like interface.
-- Class: wsgiref.headers.Headers ([headers])
Create a mapping-like object wrapping 'headers', which must be a
list of header name/value tuples as described in PEP 3333(1). The
default value of 'headers' is an empty list.
*note Headers: e91. objects support typical mapping operations
including *note __getitem__(): 285, *note get(): 2242, *note
__setitem__(): 12be, *note setdefault(): 1072, *note __delitem__():
12bf. and *note __contains__(): 1f5e. For each of these methods,
the key is the header name (treated case-insensitively), and the
value is the first value associated with that header name. Setting
a header deletes any existing values for that header, then adds a
new value at the end of the wrapped header list. Headers’ existing
order is generally maintained, with new headers added to the end of
the wrapped list.
Unlike a dictionary, *note Headers: e91. objects do not raise an
error when you try to get or delete a key that isn’t in the wrapped
header list. Getting a nonexistent header just returns ‘None’, and
deleting a nonexistent header does nothing.
*note Headers: e91. objects also support ‘keys()’, ‘values()’, and
‘items()’ methods. The lists returned by ‘keys()’ and ‘items()’
can include the same key more than once if there is a multi-valued
header. The ‘len()’ of a *note Headers: e91. object is the same as
the length of its ‘items()’, which is the same as the length of the
wrapped header list. In fact, the ‘items()’ method just returns a
copy of the wrapped header list.
Calling ‘bytes()’ on a *note Headers: e91. object returns a
formatted bytestring suitable for transmission as HTTP response
headers. Each header is placed on a line with its value, separated
by a colon and a space. Each line is terminated by a carriage
return and line feed, and the bytestring is terminated with a blank
line.
In addition to their mapping interface and formatting features,
*note Headers: e91. objects also have the following methods for
querying and adding multi-valued headers, and for adding headers
with MIME parameters:
-- Method: get_all (name)
Return a list of all the values for the named header.
The returned list will be sorted in the order they appeared in
the original header list or were added to this instance, and
may contain duplicates. Any fields deleted and re-inserted
are always appended to the header list. If no fields exist
with the given name, returns an empty list.
-- Method: add_header (name, value, **_params)
Add a (possibly multi-valued) header, with optional MIME
parameters specified via keyword arguments.
'name' is the header field to add. Keyword arguments can be
used to set MIME parameters for the header field. Each
parameter must be a string or ‘None’. Underscores in
parameter names are converted to dashes, since dashes are
illegal in Python identifiers, but many MIME parameter names
include dashes. If the parameter value is a string, it is
added to the header value parameters in the form
‘name="value"’. If it is ‘None’, only the parameter name is
added. (This is used for MIME parameters without a value.)
Example usage:
h.add_header('content-disposition', 'attachment', filename='bud.gif')
The above will add a header that looks like this:
Content-Disposition: attachment; filename="bud.gif"
Changed in version 3.5: 'headers' parameter is optional.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3333/
File: python.info, Node: wsgiref simple_server – a simple WSGI HTTP server, Next: wsgiref validate — WSGI conformance checker, Prev: wsgiref headers – WSGI response header tools, Up: wsgiref — WSGI Utilities and Reference Implementation
5.22.2.3 ‘wsgiref.simple_server’ – a simple WSGI HTTP server
............................................................
This module implements a simple HTTP server (based on *note http.server:
72.) that serves WSGI applications. Each server instance serves a
single WSGI application on a given host and port. If you want to serve
multiple applications on a single host and port, you should create a
WSGI application that parses ‘PATH_INFO’ to select which application to
invoke for each request. (E.g., using the ‘shift_path_info()’ function
from *note wsgiref.util: 11d.)
-- Function: wsgiref.simple_server.make_server (host, port, app,
server_class=WSGIServer, handler_class=WSGIRequestHandler)
Create a new WSGI server listening on 'host' and 'port', accepting
connections for 'app'. The return value is an instance of the
supplied 'server_class', and will process requests using the
specified 'handler_class'. 'app' must be a WSGI application
object, as defined by PEP 3333(1).
Example usage:
from wsgiref.simple_server import make_server, demo_app
with make_server('', 8000, demo_app) as httpd:
print("Serving HTTP on port 8000...")
# Respond to requests until process is killed
httpd.serve_forever()
# Alternative: serve one request, then exit
httpd.handle_request()
-- Function: wsgiref.simple_server.demo_app (environ, start_response)
This function is a small but complete WSGI application that returns
a text page containing the message “Hello world!” and a list of the
key/value pairs provided in the 'environ' parameter. It’s useful
for verifying that a WSGI server (such as *note
wsgiref.simple_server: 11b.) is able to run a simple WSGI
application correctly.
The 'start_response' callable should follow the *note
StartResponse: 3969. protocol.
-- Class: wsgiref.simple_server.WSGIServer (server_address,
RequestHandlerClass)
Create a *note WSGIServer: 396a. instance. 'server_address' should
be a ‘(host,port)’ tuple, and 'RequestHandlerClass' should be the
subclass of *note http.server.BaseHTTPRequestHandler: 10c1. that
will be used to process requests.
You do not normally need to call this constructor, as the *note
make_server(): 3968. function can handle all the details for you.
*note WSGIServer: 396a. is a subclass of *note
http.server.HTTPServer: 396b, so all of its methods (such as
‘serve_forever()’ and ‘handle_request()’) are available. *note
WSGIServer: 396a. also provides these WSGI-specific methods:
-- Method: set_app (application)
Sets the callable 'application' as the WSGI application that
will receive requests.
-- Method: get_app ()
Returns the currently set application callable.
Normally, however, you do not need to use these additional methods,
as *note set_app(): 396c. is normally called by *note
make_server(): 3968, and the *note get_app(): 396d. exists mainly
for the benefit of request handler instances.
-- Class: wsgiref.simple_server.WSGIRequestHandler (request,
client_address, server)
Create an HTTP handler for the given 'request' (i.e. a socket),
'client_address' (a ‘(host,port)’ tuple), and 'server' (*note
WSGIServer: 396a. instance).
You do not need to create instances of this class directly; they
are automatically created as needed by *note WSGIServer: 396a.
objects. You can, however, subclass this class and supply it as a
'handler_class' to the *note make_server(): 3968. function. Some
possibly relevant methods for overriding in subclasses:
-- Method: get_environ ()
Return a *note WSGIEnvironment: 395c. dictionary for a
request. The default implementation copies the contents of
the *note WSGIServer: 396a. object’s ‘base_environ’ dictionary
attribute and then adds various headers derived from the HTTP
request. Each call to this method should return a new
dictionary containing all of the relevant CGI environment
variables as specified in PEP 3333(2).
-- Method: get_stderr ()
Return the object that should be used as the ‘wsgi.errors’
stream. The default implementation just returns ‘sys.stderr’.
-- Method: handle ()
Process the HTTP request. The default implementation creates
a handler instance using a *note wsgiref.handlers: 119. class
to implement the actual WSGI application interface.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3333/
(2) https://peps.python.org/pep-3333/
File: python.info, Node: wsgiref validate — WSGI conformance checker, Next: wsgiref handlers – server/gateway base classes, Prev: wsgiref simple_server – a simple WSGI HTTP server, Up: wsgiref — WSGI Utilities and Reference Implementation
5.22.2.4 ‘wsgiref.validate’ — WSGI conformance checker
......................................................
When creating new WSGI application objects, frameworks, servers, or
middleware, it can be useful to validate the new code’s conformance
using *note wsgiref.validate: 11e. This module provides a function that
creates WSGI application objects that validate communications between a
WSGI server or gateway and a WSGI application object, to check both
sides for protocol conformance.
Note that this utility does not guarantee complete PEP 3333(1)
compliance; an absence of errors from this module does not necessarily
mean that errors do not exist. However, if this module does produce an
error, then it is virtually certain that either the server or
application is not 100% compliant.
This module is based on the ‘paste.lint’ module from Ian Bicking’s
“Python Paste” library.
-- Function: wsgiref.validate.validator (application)
Wrap 'application' and return a new WSGI application object. The
returned application will forward all requests to the original
'application', and will check that both the 'application' and the
server invoking it are conforming to the WSGI specification and to
RFC 2616(2).
Any detected nonconformance results in an *note AssertionError:
6a5. being raised; note, however, that how these errors are handled
is server-dependent. For example, *note wsgiref.simple_server:
11b. and other servers based on *note wsgiref.handlers: 119. (that
don’t override the error handling methods to do something else)
will simply output a message that an error has occurred, and dump
the traceback to ‘sys.stderr’ or some other error stream.
This wrapper may also generate output using the *note warnings:
112. module to indicate behaviors that are questionable but which
may not actually be prohibited by PEP 3333(3). Unless they are
suppressed using Python command-line options or the *note warnings:
112. API, any such warnings will be written to ‘sys.stderr’ ('not'
‘wsgi.errors’, unless they happen to be the same object).
Example usage:
from wsgiref.validate import validator
from wsgiref.simple_server import make_server
# Our callable object which is intentionally not compliant to the
# standard, so the validator is going to break
def simple_app(environ, start_response):
status = '200 OK' # HTTP Status
headers = [('Content-type', 'text/plain')] # HTTP Headers
start_response(status, headers)
# This is going to break because we need to return a list, and
# the validator is going to inform us
return b"Hello World"
# This is the application wrapped in a validator
validator_app = validator(simple_app)
with make_server('', 8000, validator_app) as httpd:
print("Listening on port 8000....")
httpd.serve_forever()
---------- Footnotes ----------
(1) https://peps.python.org/pep-3333/
(2) https://datatracker.ietf.org/doc/html/rfc2616.html
(3) https://peps.python.org/pep-3333/
File: python.info, Node: wsgiref handlers – server/gateway base classes, Next: wsgiref types – WSGI types for static type checking, Prev: wsgiref validate — WSGI conformance checker, Up: wsgiref — WSGI Utilities and Reference Implementation
5.22.2.5 ‘wsgiref.handlers’ – server/gateway base classes
.........................................................
This module provides base handler classes for implementing WSGI servers
and gateways. These base classes handle most of the work of
communicating with a WSGI application, as long as they are given a
CGI-like environment, along with input, output, and error streams.
-- Class: wsgiref.handlers.CGIHandler
CGI-based invocation via ‘sys.stdin’, ‘sys.stdout’, ‘sys.stderr’
and ‘os.environ’. This is useful when you have a WSGI application
and want to run it as a CGI script. Simply invoke
‘CGIHandler().run(app)’, where ‘app’ is the WSGI application object
you wish to invoke.
This class is a subclass of *note BaseCGIHandler: 3976. that sets
‘wsgi.run_once’ to true, ‘wsgi.multithread’ to false, and
‘wsgi.multiprocess’ to true, and always uses *note sys: d9. and
*note os: a1. to obtain the necessary CGI streams and environment.
-- Class: wsgiref.handlers.IISCGIHandler
A specialized alternative to *note CGIHandler: 3975, for use when
deploying on Microsoft’s IIS web server, without having set the
config allowPathInfo option (IIS>=7) or metabase
allowPathInfoForScriptMappings (IIS<7).
By default, IIS gives a ‘PATH_INFO’ that duplicates the
‘SCRIPT_NAME’ at the front, causing problems for WSGI applications
that wish to implement routing. This handler strips any such
duplicated path.
IIS can be configured to pass the correct ‘PATH_INFO’, but this
causes another bug where ‘PATH_TRANSLATED’ is wrong. Luckily this
variable is rarely used and is not guaranteed by WSGI. On IIS<7,
though, the setting can only be made on a vhost level, affecting
all other script mappings, many of which break when exposed to the
‘PATH_TRANSLATED’ bug. For this reason IIS<7 is almost never
deployed with the fix (Even IIS7 rarely uses it because there is
still no UI for it.).
There is no way for CGI code to tell whether the option was set, so
a separate handler class is provided. It is used in the same way
as *note CGIHandler: 3975, i.e., by calling
‘IISCGIHandler().run(app)’, where ‘app’ is the WSGI application
object you wish to invoke.
Added in version 3.2.
-- Class: wsgiref.handlers.BaseCGIHandler (stdin, stdout, stderr,
environ, multithread=True, multiprocess=False)
Similar to *note CGIHandler: 3975, but instead of using the *note
sys: d9. and *note os: a1. modules, the CGI environment and I/O
streams are specified explicitly. The 'multithread' and
'multiprocess' values are used to set the ‘wsgi.multithread’ and
‘wsgi.multiprocess’ flags for any applications run by the handler
instance.
This class is a subclass of *note SimpleHandler: 3978. intended for
use with software other than HTTP “origin servers”. If you are
writing a gateway protocol implementation (such as CGI, FastCGI,
SCGI, etc.) that uses a ‘Status:’ header to send an HTTP status,
you probably want to subclass this instead of *note SimpleHandler:
3978.
-- Class: wsgiref.handlers.SimpleHandler (stdin, stdout, stderr,
environ, multithread=True, multiprocess=False)
Similar to *note BaseCGIHandler: 3976, but designed for use with
HTTP origin servers. If you are writing an HTTP server
implementation, you will probably want to subclass this instead of
*note BaseCGIHandler: 3976.
This class is a subclass of *note BaseHandler: 19ef. It overrides
the ‘__init__()’, *note get_stdin(): 3979, *note get_stderr():
397a, *note add_cgi_vars(): 397b, *note _write(): 397c, and *note
_flush(): 397d. methods to support explicitly setting the
environment and streams via the constructor. The supplied
environment and streams are stored in the ‘stdin’, ‘stdout’,
‘stderr’, and ‘environ’ attributes.
The *note write(): cdb. method of 'stdout' should write each chunk
in full, like *note io.BufferedIOBase: 690.
-- Class: wsgiref.handlers.BaseHandler
This is an abstract base class for running WSGI applications. Each
instance will handle a single HTTP request, although in principle
you could create a subclass that was reusable for multiple
requests.
*note BaseHandler: 19ef. instances have only one method intended
for external use:
-- Method: run (app)
Run the specified WSGI application, 'app'.
All of the other *note BaseHandler: 19ef. methods are invoked by
this method in the process of running the application, and thus
exist primarily to allow customizing the process.
The following methods MUST be overridden in a subclass:
-- Method: _write (data)
Buffer the bytes 'data' for transmission to the client. It’s
okay if this method actually transmits the data; *note
BaseHandler: 19ef. just separates write and flush operations
for greater efficiency when the underlying system actually has
such a distinction.
-- Method: _flush ()
Force buffered data to be transmitted to the client. It’s
okay if this method is a no-op (i.e., if *note _write(): 397c.
actually sends the data).
-- Method: get_stdin ()
Return an object compatible with *note InputStream: 397f.
suitable for use as the ‘wsgi.input’ of the request currently
being processed.
-- Method: get_stderr ()
Return an object compatible with *note ErrorStream: 3980.
suitable for use as the ‘wsgi.errors’ of the request currently
being processed.
-- Method: add_cgi_vars ()
Insert CGI variables for the current request into the
‘environ’ attribute.
Here are some other methods and attributes you may wish to
override. This list is only a summary, however, and does not
include every method that can be overridden. You should consult
the docstrings and source code for additional information before
attempting to create a customized *note BaseHandler: 19ef.
subclass.
Attributes and methods for customizing the WSGI environment:
-- Attribute: wsgi_multithread
The value to be used for the ‘wsgi.multithread’ environment
variable. It defaults to true in *note BaseHandler: 19ef, but
may have a different default (or be set by the constructor) in
the other subclasses.
-- Attribute: wsgi_multiprocess
The value to be used for the ‘wsgi.multiprocess’ environment
variable. It defaults to true in *note BaseHandler: 19ef, but
may have a different default (or be set by the constructor) in
the other subclasses.
-- Attribute: wsgi_run_once
The value to be used for the ‘wsgi.run_once’ environment
variable. It defaults to false in *note BaseHandler: 19ef,
but *note CGIHandler: 3975. sets it to true by default.
-- Attribute: os_environ
The default environment variables to be included in every
request’s WSGI environment. By default, this is a copy of
‘os.environ’ at the time that *note wsgiref.handlers: 119. was
imported, but subclasses can either create their own at the
class or instance level. Note that the dictionary should be
considered read-only, since the default value is shared
between multiple classes and instances.
-- Attribute: server_software
If the *note origin_server: 3986. attribute is set, this
attribute’s value is used to set the default ‘SERVER_SOFTWARE’
WSGI environment variable, and also to set a default ‘Server:’
header in HTTP responses. It is ignored for handlers (such as
*note BaseCGIHandler: 3976. and *note CGIHandler: 3975.) that
are not HTTP origin servers.
Changed in version 3.3: The term “Python” is replaced with
implementation specific term like “CPython”, “Jython” etc.
-- Method: get_scheme ()
Return the URL scheme being used for the current request. The
default implementation uses the ‘guess_scheme()’ function from
*note wsgiref.util: 11d. to guess whether the scheme should be
“http” or “https”, based on the current request’s ‘environ’
variables.
-- Method: setup_environ ()
Set the ‘environ’ attribute to a fully populated WSGI
environment. The default implementation uses all of the above
methods and attributes, plus the *note get_stdin(): 3979,
*note get_stderr(): 397a, and *note add_cgi_vars(): 397b.
methods and the *note wsgi_file_wrapper: 3989. attribute. It
also inserts a ‘SERVER_SOFTWARE’ key if not present, as long
as the *note origin_server: 3986. attribute is a true value
and the *note server_software: 3985. attribute is set.
Methods and attributes for customizing exception handling:
-- Method: log_exception (exc_info)
Log the 'exc_info' tuple in the server log. 'exc_info' is a
‘(type, value, traceback)’ tuple. The default implementation
simply writes the traceback to the request’s ‘wsgi.errors’
stream and flushes it. Subclasses can override this method to
change the format or retarget the output, mail the traceback
to an administrator, or whatever other action may be deemed
suitable.
-- Attribute: traceback_limit
The maximum number of frames to include in tracebacks output
by the default *note log_exception(): 398a. method. If
‘None’, all frames are included.
-- Method: error_output (environ, start_response)
This method is a WSGI application to generate an error page
for the user. It is only invoked if an error occurs before
headers are sent to the client.
This method can access the current error using
‘sys.exception()’, and should pass that information to
'start_response' when calling it (as described in the “Error
Handling” section of PEP 3333(1)). In particular, the
'start_response' callable should follow the *note
StartResponse: 3969. protocol.
The default implementation just uses the *note error_status:
398d, *note error_headers: 398e, and *note error_body: 398f.
attributes to generate an output page. Subclasses can
override this to produce more dynamic error output.
Note, however, that it’s not recommended from a security
perspective to spit out diagnostics to any old user; ideally,
you should have to do something special to enable diagnostic
output, which is why the default implementation doesn’t
include any.
-- Attribute: error_status
The HTTP status used for error responses. This should be a
status string as defined in PEP 3333(2); it defaults to a 500
code and message.
-- Attribute: error_headers
The HTTP headers used for error responses. This should be a
list of WSGI response headers (‘(name, value)’ tuples), as
described in PEP 3333(3). The default list just sets the
content type to ‘text/plain’.
-- Attribute: error_body
The error response body. This should be an HTTP response body
bytestring. It defaults to the plain text, “A server error
occurred. Please contact the administrator.”
Methods and attributes for PEP 3333(4)’s “Optional
Platform-Specific File Handling” feature:
-- Attribute: wsgi_file_wrapper
A ‘wsgi.file_wrapper’ factory, compatible with *note
wsgiref.types.FileWrapper: 3963, or ‘None’. The default value
of this attribute is the *note wsgiref.util.FileWrapper: 731.
class.
-- Method: sendfile ()
Override to implement platform-specific file transmission.
This method is called only if the application’s return value
is an instance of the class specified by the *note
wsgi_file_wrapper: 3989. attribute. It should return a true
value if it was able to successfully transmit the file, so
that the default transmission code will not be executed. The
default implementation of this method just returns a false
value.
Miscellaneous methods and attributes:
-- Attribute: origin_server
This attribute should be set to a true value if the handler’s
*note _write(): 397c. and *note _flush(): 397d. are being used
to communicate directly to the client, rather than via a
CGI-like gateway protocol that wants the HTTP status in a
special ‘Status:’ header.
This attribute’s default value is true in *note BaseHandler:
19ef, but false in *note BaseCGIHandler: 3976. and *note
CGIHandler: 3975.
-- Attribute: http_version
If *note origin_server: 3986. is true, this string attribute
is used to set the HTTP version of the response set to the
client. It defaults to ‘"1.0"’.
-- Function: wsgiref.handlers.read_environ ()
Transcode CGI variables from ‘os.environ’ to PEP 3333(5) “bytes in
unicode” strings, returning a new dictionary. This function is
used by *note CGIHandler: 3975. and *note IISCGIHandler: 3977. in
place of directly using ‘os.environ’, which is not necessarily
WSGI-compliant on all platforms and web servers using Python 3 –
specifically, ones where the OS’s actual environment is Unicode
(i.e. Windows), or ones where the environment is bytes, but the
system encoding used by Python to decode it is anything other than
ISO-8859-1 (e.g. Unix systems using UTF-8).
If you are implementing a CGI-based handler of your own, you
probably want to use this routine instead of just copying values
out of ‘os.environ’ directly.
Added in version 3.2.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3333/
(2) https://peps.python.org/pep-3333/
(3) https://peps.python.org/pep-3333/
(4) https://peps.python.org/pep-3333/
(5) https://peps.python.org/pep-3333/
File: python.info, Node: wsgiref types – WSGI types for static type checking, Next: Examples<25>, Prev: wsgiref handlers – server/gateway base classes, Up: wsgiref — WSGI Utilities and Reference Implementation
5.22.2.6 ‘wsgiref.types’ – WSGI types for static type checking
..............................................................
This module provides various types for static type checking as described
in PEP 3333(1).
Added in version 3.11.
-- Class: wsgiref.types.StartResponse
A *note typing.Protocol: 266. describing start_response()(2)
callables ( PEP 3333(3)).
-- Data: wsgiref.types.WSGIEnvironment
A type alias describing a WSGI environment dictionary.
-- Data: wsgiref.types.WSGIApplication
A type alias describing a WSGI application callable.
-- Class: wsgiref.types.InputStream
A *note typing.Protocol: 266. describing a WSGI Input Stream(4).
-- Class: wsgiref.types.ErrorStream
A *note typing.Protocol: 266. describing a WSGI Error Stream(5).
-- Class: wsgiref.types.FileWrapper
A *note typing.Protocol: 266. describing a file wrapper(6). See
*note wsgiref.util.FileWrapper: 731. for a concrete implementation
of this protocol.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3333/
(2) https://peps.python.org/pep-3333/#the-start-response-callable
(3) https://peps.python.org/pep-3333/
(4) https://peps.python.org/pep-3333/#input-and-error-streams
(5) https://peps.python.org/pep-3333/#input-and-error-streams
(6)
https://peps.python.org/pep-3333/#optional-platform-specific-file-handling
File: python.info, Node: Examples<25>, Prev: wsgiref types – WSGI types for static type checking, Up: wsgiref — WSGI Utilities and Reference Implementation
5.22.2.7 Examples
.................
This is a working “Hello World” WSGI application, where the
'start_response' callable should follow the *note StartResponse: 3969.
protocol:
"""
Every WSGI application must have an application object - a callable
object that accepts two arguments. For that purpose, we're going to
use a function (note that you're not limited to a function, you can
use a class for example). The first argument passed to the function
is a dictionary containing CGI-style environment variables and the
second variable is the callable object.
"""
from wsgiref.simple_server import make_server
def hello_world_app(environ, start_response):
status = "200 OK" # HTTP Status
headers = [("Content-type", "text/plain; charset=utf-8")] # HTTP Headers
start_response(status, headers)
# The returned object is going to be printed
return [b"Hello World"]
with make_server("", 8000, hello_world_app) as httpd:
print("Serving on port 8000...")
# Serve until process is killed
httpd.serve_forever()
Example of a WSGI application serving the current directory, accept
optional directory and port number (default: 8000) on the command line:
"""
Small wsgiref based web server. Takes a path to serve from and an
optional port number (defaults to 8000), then tries to serve files.
MIME types are guessed from the file names, 404 errors are raised
if the file is not found.
"""
import mimetypes
import os
import sys
from wsgiref import simple_server, util
def app(environ, respond):
# Get the file name and MIME type
fn = os.path.join(path, environ["PATH_INFO"][1:])
if "." not in fn.split(os.path.sep)[-1]:
fn = os.path.join(fn, "index.html")
mime_type = mimetypes.guess_file_type(fn)[0]
# Return 200 OK if file exists, otherwise 404 Not Found
if os.path.exists(fn):
respond("200 OK", [("Content-Type", mime_type)])
return util.FileWrapper(open(fn, "rb"))
else:
respond("404 Not Found", [("Content-Type", "text/plain")])
return [b"not found"]
if __name__ == "__main__":
# Get the path and port from command-line arguments
path = sys.argv[1] if len(sys.argv) > 1 else os.getcwd()
port = int(sys.argv[2]) if len(sys.argv) > 2 else 8000
# Make and start the server until control-c
httpd = simple_server.make_server("", port, app)
print(f"Serving {path} on port {port}, control-C to stop")
try:
httpd.serve_forever()
except KeyboardInterrupt:
print("Shutting down.")
httpd.server_close()
File: python.info, Node: urllib — URL handling modules, Next: urllib request — Extensible library for opening URLs, Prev: wsgiref — WSGI Utilities and Reference Implementation, Up: Internet Protocols and Support
5.22.3 ‘urllib’ — URL handling modules
--------------------------------------
'Source code:' Lib/urllib/(1)
__________________________________________________________________
‘urllib’ is a package that collects several modules for working with
URLs:
* *note urllib.request: 10b. for opening and reading URLs
* *note urllib.error: 109. containing the exceptions raised by *note
urllib.request: 10b.
* *note urllib.parse: 10a. for parsing URLs
* *note urllib.robotparser: 10d. for parsing ‘robots.txt’ files
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/urllib/
File: python.info, Node: urllib request — Extensible library for opening URLs, Next: urllib response — Response classes used by urllib, Prev: urllib — URL handling modules, Up: Internet Protocols and Support
5.22.4 ‘urllib.request’ — Extensible library for opening URLs
-------------------------------------------------------------
'Source code:' Lib/urllib/request.py(1)
__________________________________________________________________
The *note urllib.request: 10b. module defines functions and classes
which help in opening URLs (mostly HTTP) in a complex world — basic and
digest authentication, redirections, cookies and more.
See also
........
The Requests package(2) is recommended for a higher-level HTTP client
interface.
Warning: On macOS it is unsafe to use this module in programs using
*note os.fork(): 197. because the *note getproxies(): 3999.
implementation for macOS uses a higher-level system API. Set the
environment variable ‘no_proxy’ to ‘*’ to avoid this problem (e.g.
‘os.environ["no_proxy"] = "*"’).
*note Availability: 1d54.: not WASI.
This module does not work or is not available on WebAssembly. See *note
WebAssembly platforms: 17e0. for more information.
The *note urllib.request: 10b. module defines the following functions:
-- Function: urllib.request.urlopen (url, data=None[, timeout], *,
context=None)
Open 'url', which can be either a string containing a valid,
properly encoded URL, or a *note Request: fd3. object.
'data' must be an object specifying additional data to be sent to
the server, or ‘None’ if no such data is needed. See *note
Request: fd3. for details.
urllib.request module uses HTTP/1.1 and includes ‘Connection:close’
header in its HTTP requests.
The optional 'timeout' parameter specifies a timeout in seconds for
blocking operations like the connection attempt (if not specified,
the global default timeout setting will be used). This actually
only works for HTTP, HTTPS and FTP connections.
If 'context' is specified, it must be a *note ssl.SSLContext: 296.
instance describing the various SSL options. See *note
HTTPSConnection: b41. for more details.
This function always returns an object which can work as a *note
context manager: 5d0. and has the properties 'url', 'headers', and
'status'. See *note urllib.response.addinfourl: 399a. for more
detail on these properties.
For HTTP and HTTPS URLs, this function returns a *note
http.client.HTTPResponse: 10c4. object slightly modified. In
addition to the three new methods above, the msg attribute contains
the same information as the *note reason: 399b. attribute — the
reason phrase returned by server — instead of the response headers
as it is specified in the documentation for *note HTTPResponse:
10c4.
For FTP, file, and data URLs and requests explicitly handled by
legacy *note URLopener: 308. and *note FancyURLopener: 309.
classes, this function returns a *note urllib.response.addinfourl:
399a. object.
Raises *note URLError: 399c. on protocol errors.
Note that ‘None’ may be returned if no handler handles the request
(though the default installed global *note OpenerDirector: 399d.
uses *note UnknownHandler: 399e. to ensure this never happens).
In addition, if proxy settings are detected (for example, when a
‘*_proxy’ environment variable like ‘http_proxy’ is set), *note
ProxyHandler: 19cf. is default installed and makes sure the
requests are handled through the proxy.
The legacy ‘urllib.urlopen’ function from Python 2.6 and earlier
has been discontinued; *note urllib.request.urlopen(): 295.
corresponds to the old ‘urllib2.urlopen’. Proxy handling, which
was done by passing a dictionary parameter to ‘urllib.urlopen’, can
be obtained by using *note ProxyHandler: 19cf. objects.
The default opener raises an *note auditing event: 18ba.
‘urllib.Request’ with arguments ‘fullurl’, ‘data’, ‘headers’,
‘method’ taken from the request object.
Changed in version 3.2: 'cafile' and 'capath' were added.
HTTPS virtual hosts are now supported if possible (that is, if
*note ssl.HAS_SNI: 3480. is true).
'data' can be an iterable object.
Changed in version 3.3: 'cadefault' was added.
Changed in version 3.4.3: 'context' was added.
Changed in version 3.10: HTTPS connection now send an ALPN
extension with protocol indicator ‘http/1.1’ when no 'context' is
given. Custom 'context' should set ALPN protocols with *note
set_alpn_protocols(): e5f.
Changed in version 3.13: Remove 'cafile', 'capath' and 'cadefault'
parameters: use the 'context' parameter instead.
-- Function: urllib.request.install_opener (opener)
Install an *note OpenerDirector: 399d. instance as the default
global opener. Installing an opener is only necessary if you want
urlopen to use that opener; otherwise, simply call *note
OpenerDirector.open(): fd7. instead of *note urlopen(): 295. The
code does not check for a real *note OpenerDirector: 399d, and any
class with the appropriate interface will work.
-- Function: urllib.request.build_opener ([handler, ...])
Return an *note OpenerDirector: 399d. instance, which chains the
handlers in the order given. 'handler's can be either instances of
*note BaseHandler: 39a1, or subclasses of *note BaseHandler: 39a1.
(in which case it must be possible to call the constructor without
any parameters). Instances of the following classes will be in
front of the 'handler's, unless the 'handler's contain them,
instances of them or subclasses of them: *note ProxyHandler: 19cf.
(if proxy settings are detected), *note UnknownHandler: 399e, *note
HTTPHandler: 39a2, *note HTTPDefaultErrorHandler: 39a3, *note
HTTPRedirectHandler: 39a4, *note FTPHandler: 39a5, *note
FileHandler: 39a6, *note HTTPErrorProcessor: 39a7.
If the Python installation has SSL support (i.e., if the *note ssl:
d0. module can be imported), *note HTTPSHandler: 1216. will also be
added.
A *note BaseHandler: 39a1. subclass may also change its
‘handler_order’ attribute to modify its position in the handlers
list.
-- Function: urllib.request.pathname2url (path)
Convert the given local path to a ‘file:’ URL. This function uses
*note quote(): ba3. function to encode the path. For historical
reasons, the return value omits the ‘file:’ scheme prefix. This
example shows the function being used on Windows:
>>> from urllib.request import pathname2url
>>> path = 'C:\\Program Files'
>>> 'file:' + pathname2url(path)
'file:///C:/Program%20Files'
-- Function: urllib.request.url2pathname (url)
Convert the given ‘file:’ URL to a local path. This function uses
*note unquote(): 1785. to decode the URL. For historical reasons,
the given value 'must' omit the ‘file:’ scheme prefix. This
example shows the function being used on Windows:
>>> from urllib.request import url2pathname
>>> url = 'file:///C:/Program%20Files'
>>> url2pathname(url.removeprefix('file:'))
'C:\\Program Files'
-- Function: urllib.request.getproxies ()
This helper function returns a dictionary of scheme to proxy server
URL mappings. It scans the environment for variables named
‘_proxy’, in a case insensitive approach, for all operating
systems first, and when it cannot find it, looks for proxy
information from System Configuration for macOS and Windows Systems
Registry for Windows. If both lowercase and uppercase environment
variables exist (and disagree), lowercase is preferred.
Note: If the environment variable ‘REQUEST_METHOD’ is set,
which usually indicates your script is running in a CGI
environment, the environment variable ‘HTTP_PROXY’ (uppercase
‘_PROXY’) will be ignored. This is because that variable can
be injected by a client using the “Proxy:” HTTP header. If
you need to use an HTTP proxy in a CGI environment, either use
‘ProxyHandler’ explicitly, or make sure the variable name is
in lowercase (or at least the ‘_proxy’ suffix).
The following classes are provided:
-- Class: urllib.request.Request (url, data=None, headers={},
origin_req_host=None, unverifiable=False, method=None)
This class is an abstraction of a URL request.
'url' should be a string containing a valid, properly encoded URL.
'data' must be an object specifying additional data to send to the
server, or ‘None’ if no such data is needed. Currently HTTP
requests are the only ones that use 'data'. The supported object
types include bytes, file-like objects, and iterables of bytes-like
objects. If no ‘Content-Length’ nor ‘Transfer-Encoding’ header
field has been provided, *note HTTPHandler: 39a2. will set these
headers according to the type of 'data'. ‘Content-Length’ will be
used to send bytes objects, while ‘Transfer-Encoding: chunked’ as
specified in RFC 7230(3), Section 3.3.1 will be used to send files
and other iterables.
For an HTTP POST request method, 'data' should be a buffer in the
standard ‘application/x-www-form-urlencoded’ format. The *note
urllib.parse.urlencode(): e8e. function takes a mapping or sequence
of 2-tuples and returns an ASCII string in this format. It should
be encoded to bytes before being used as the 'data' parameter.
'headers' should be a dictionary, and will be treated as if *note
add_header(): 39a8. was called with each key and value as
arguments. This is often used to “spoof” the ‘User-Agent’ header
value, which is used by a browser to identify itself – some HTTP
servers only allow requests coming from common browsers as opposed
to scripts. For example, Mozilla Firefox may identify itself as
‘"Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127
Firefox/2.0.0.11"’, while *note urllib: 108.’s default user agent
string is ‘"Python-urllib/2.6"’ (on Python 2.6). All header keys
are sent in camel case.
An appropriate ‘Content-Type’ header should be included if the
'data' argument is present. If this header has not been provided
and 'data' is not ‘None’, ‘Content-Type:
application/x-www-form-urlencoded’ will be added as a default.
The next two arguments are only of interest for correct handling of
third-party HTTP cookies:
'origin_req_host' should be the request-host of the origin
transaction, as defined by RFC 2965(4). It defaults to
‘http.cookiejar.request_host(self)’. This is the host name or IP
address of the original request that was initiated by the user.
For example, if the request is for an image in an HTML document,
this should be the request-host of the request for the page
containing the image.
'unverifiable' should indicate whether the request is unverifiable,
as defined by RFC 2965(5). It defaults to ‘False’. An
unverifiable request is one whose URL the user did not have the
option to approve. For example, if the request is for an image in
an HTML document, and the user had no option to approve the
automatic fetching of the image, this should be true.
'method' should be a string that indicates the HTTP request method
that will be used (e.g. ‘'HEAD'’). If provided, its value is
stored in the *note method: fd4. attribute and is used by *note
get_method(): 115d. The default is ‘'GET'’ if 'data' is ‘None’ or
‘'POST'’ otherwise. Subclasses may indicate a different default
method by setting the *note method: fd4. attribute in the class
itself.
Note: The request will not work as expected if the data object
is unable to deliver its content more than once (e.g. a file
or an iterable that can produce the content only once) and the
request is retried for HTTP redirects or authentication. The
'data' is sent to the HTTP server right away after the
headers. There is no support for a 100-continue expectation
in the library.
Changed in version 3.3: *note Request.method: fd4. argument is
added to the Request class.
Changed in version 3.4: Default *note Request.method: fd4. may be
indicated at the class level.
Changed in version 3.6: Do not raise an error if the
‘Content-Length’ has not been provided and 'data' is neither ‘None’
nor a bytes object. Fall back to use chunked transfer encoding
instead.
-- Class: urllib.request.OpenerDirector
The *note OpenerDirector: 399d. class opens URLs via *note
BaseHandler: 39a1.s chained together. It manages the chaining of
handlers, and recovery from errors.
-- Class: urllib.request.BaseHandler
This is the base class for all registered handlers — and handles
only the simple mechanics of registration.
-- Class: urllib.request.HTTPDefaultErrorHandler
A class which defines a default handler for HTTP error responses;
all responses are turned into *note HTTPError: fd9. exceptions.
-- Class: urllib.request.HTTPRedirectHandler
A class to handle redirections.
-- Class: urllib.request.HTTPCookieProcessor (cookiejar=None)
A class to handle HTTP Cookies.
-- Class: urllib.request.ProxyHandler (proxies=None)
Cause requests to go through a proxy. If 'proxies' is given, it
must be a dictionary mapping protocol names to URLs of proxies.
The default is to read the list of proxies from the environment
variables ‘_proxy’. If no proxy environment variables
are set, then in a Windows environment proxy settings are obtained
from the registry’s Internet Settings section, and in a macOS
environment proxy information is retrieved from the System
Configuration Framework.
To disable autodetected proxy pass an empty dictionary.
The ‘no_proxy’ environment variable can be used to specify hosts
which shouldn’t be reached via proxy; if set, it should be a
comma-separated list of hostname suffixes, optionally with ‘:port’
appended, for example ‘cern.ch,ncsa.uiuc.edu,some.host:8080’.
Note: ‘HTTP_PROXY’ will be ignored if a variable
‘REQUEST_METHOD’ is set; see the documentation on *note
getproxies(): 3999.
-- Class: urllib.request.HTTPPasswordMgr
Keep a database of ‘(realm, uri) -> (user, password)’ mappings.
-- Class: urllib.request.HTTPPasswordMgrWithDefaultRealm
Keep a database of ‘(realm, uri) -> (user, password)’ mappings. A
realm of ‘None’ is considered a catch-all realm, which is searched
if no other realm fits.
-- Class: urllib.request.HTTPPasswordMgrWithPriorAuth
A variant of *note HTTPPasswordMgrWithDefaultRealm: 39ab. that also
has a database of ‘uri -> is_authenticated’ mappings. Can be used
by a BasicAuth handler to determine when to send authentication
credentials immediately instead of waiting for a ‘401’ response
first.
Added in version 3.5.
-- Class: urllib.request.AbstractBasicAuthHandler (password_mgr=None)
This is a mixin class that helps with HTTP authentication, both to
the remote host and to a proxy. 'password_mgr', if given, should
be something that is compatible with *note HTTPPasswordMgr: 39aa.;
refer to section *note HTTPPasswordMgr Objects: 39ac. for
information on the interface that must be supported. If
'passwd_mgr' also provides ‘is_authenticated’ and
‘update_authenticated’ methods (see *note
HTTPPasswordMgrWithPriorAuth Objects: 39ad.), then the handler will
use the ‘is_authenticated’ result for a given URI to determine
whether or not to send authentication credentials with the request.
If ‘is_authenticated’ returns ‘True’ for the URI, credentials are
sent. If ‘is_authenticated’ is ‘False’, credentials are not sent,
and then if a ‘401’ response is received the request is re-sent
with the authentication credentials. If authentication succeeds,
‘update_authenticated’ is called to set ‘is_authenticated’ ‘True’
for the URI, so that subsequent requests to the URI or any of its
super-URIs will automatically include the authentication
credentials.
Added in version 3.5: Added ‘is_authenticated’ support.
-- Class: urllib.request.HTTPBasicAuthHandler (password_mgr=None)
Handle authentication with the remote host. 'password_mgr', if
given, should be something that is compatible with *note
HTTPPasswordMgr: 39aa.; refer to section *note HTTPPasswordMgr
Objects: 39ac. for information on the interface that must be
supported. HTTPBasicAuthHandler will raise a *note ValueError:
204. when presented with a wrong Authentication scheme.
-- Class: urllib.request.ProxyBasicAuthHandler (password_mgr=None)
Handle authentication with the proxy. 'password_mgr', if given,
should be something that is compatible with *note HTTPPasswordMgr:
39aa.; refer to section *note HTTPPasswordMgr Objects: 39ac. for
information on the interface that must be supported.
-- Class: urllib.request.AbstractDigestAuthHandler (password_mgr=None)
This is a mixin class that helps with HTTP authentication, both to
the remote host and to a proxy. 'password_mgr', if given, should
be something that is compatible with *note HTTPPasswordMgr: 39aa.;
refer to section *note HTTPPasswordMgr Objects: 39ac. for
information on the interface that must be supported.
-- Class: urllib.request.HTTPDigestAuthHandler (password_mgr=None)
Handle authentication with the remote host. 'password_mgr', if
given, should be something that is compatible with *note
HTTPPasswordMgr: 39aa.; refer to section *note HTTPPasswordMgr
Objects: 39ac. for information on the interface that must be
supported. When both Digest Authentication Handler and Basic
Authentication Handler are both added, Digest Authentication is
always tried first. If the Digest Authentication returns a 40x
response again, it is sent to Basic Authentication handler to
Handle. This Handler method will raise a *note ValueError: 204.
when presented with an authentication scheme other than Digest or
Basic.
Changed in version 3.3: Raise *note ValueError: 204. on unsupported
Authentication Scheme.
-- Class: urllib.request.ProxyDigestAuthHandler (password_mgr=None)
Handle authentication with the proxy. 'password_mgr', if given,
should be something that is compatible with *note HTTPPasswordMgr:
39aa.; refer to section *note HTTPPasswordMgr Objects: 39ac. for
information on the interface that must be supported.
-- Class: urllib.request.HTTPHandler
A class to handle opening of HTTP URLs.
-- Class: urllib.request.HTTPSHandler (debuglevel=0, context=None,
check_hostname=None)
A class to handle opening of HTTPS URLs. 'context' and
'check_hostname' have the same meaning as in *note
http.client.HTTPSConnection: b41.
Changed in version 3.2: 'context' and 'check_hostname' were added.
-- Class: urllib.request.FileHandler
Open local files.
-- Class: urllib.request.DataHandler
Open data URLs.
Added in version 3.4.
-- Class: urllib.request.FTPHandler
Open FTP URLs.
-- Class: urllib.request.CacheFTPHandler
Open FTP URLs, keeping a cache of open FTP connections to minimize
delays.
-- Class: urllib.request.UnknownHandler
A catch-all class to handle unknown URLs.
-- Class: urllib.request.HTTPErrorProcessor
Process HTTP error responses.
* Menu:
* Request Objects::
* OpenerDirector Objects::
* BaseHandler Objects::
* HTTPRedirectHandler Objects::
* HTTPCookieProcessor Objects::
* ProxyHandler Objects::
* HTTPPasswordMgr Objects::
* HTTPPasswordMgrWithPriorAuth Objects::
* AbstractBasicAuthHandler Objects::
* HTTPBasicAuthHandler Objects::
* ProxyBasicAuthHandler Objects::
* AbstractDigestAuthHandler Objects::
* HTTPDigestAuthHandler Objects::
* ProxyDigestAuthHandler Objects::
* HTTPHandler Objects::
* HTTPSHandler Objects::
* FileHandler Objects::
* DataHandler Objects::
* FTPHandler Objects::
* CacheFTPHandler Objects::
* UnknownHandler Objects::
* HTTPErrorProcessor Objects::
* Examples: Examples<26>.
* Legacy interface::
* urllib.request Restrictions: urllib request Restrictions.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/urllib/request.py
(2) https://requests.readthedocs.io/en/master/
(3) https://datatracker.ietf.org/doc/html/rfc7230.html
(4) https://datatracker.ietf.org/doc/html/rfc2965.html
(5) https://datatracker.ietf.org/doc/html/rfc2965.html
File: python.info, Node: Request Objects, Next: OpenerDirector Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.1 Request Objects
........................
The following methods describe *note Request: fd3.’s public interface,
and so all may be overridden in subclasses. It also defines several
public attributes that can be used by clients to inspect the parsed
request.
-- Attribute: Request.full_url
The original URL passed to the constructor.
Changed in version 3.4.
Request.full_url is a property with setter, getter and a deleter.
Getting *note full_url: fd5. returns the original request URL with
the fragment, if it was present.
-- Attribute: Request.type
The URI scheme.
-- Attribute: Request.host
The URI authority, typically a host, but may also contain a port
separated by a colon.
-- Attribute: Request.origin_req_host
The original host for the request, without port.
-- Attribute: Request.selector
The URI path. If the *note Request: fd3. uses a proxy, then
selector will be the full URL that is passed to the proxy.
-- Attribute: Request.data
The entity body for the request, or ‘None’ if not specified.
Changed in version 3.4: Changing value of *note Request.data: fd6.
now deletes “Content-Length” header if it was previously set or
calculated.
-- Attribute: Request.unverifiable
boolean, indicates whether the request is unverifiable as defined
by RFC 2965(1).
-- Attribute: Request.method
The HTTP request method to use. By default its value is *note
None: 671, which means that *note get_method(): 115d. will do its
normal computation of the method to be used. Its value can be set
(thus overriding the default computation in *note get_method():
115d.) either by providing a default value by setting it at the
class level in a *note Request: fd3. subclass, or by passing a
value in to the *note Request: fd3. constructor via the 'method'
argument.
Added in version 3.3.
Changed in version 3.4: A default value can now be set in
subclasses; previously it could only be set via the constructor
argument.
-- Method: Request.get_method ()
Return a string indicating the HTTP request method. If *note
Request.method: fd4. is not ‘None’, return its value, otherwise
return ‘'GET'’ if *note Request.data: fd6. is ‘None’, or ‘'POST'’
if it’s not. This is only meaningful for HTTP requests.
Changed in version 3.3: get_method now looks at the value of *note
Request.method: fd4.
-- Method: Request.add_header (key, val)
Add another header to the request. Headers are currently ignored
by all handlers except HTTP handlers, where they are added to the
list of headers sent to the server. Note that there cannot be more
than one header with the same name, and later calls will overwrite
previous calls in case the 'key' collides. Currently, this is no
loss of HTTP functionality, since all headers which have meaning
when used more than once have a (header-specific) way of gaining
the same functionality using only one header. Note that headers
added using this method are also added to redirected requests.
-- Method: Request.add_unredirected_header (key, header)
Add a header that will not be added to a redirected request.
-- Method: Request.has_header (header)
Return whether the instance has the named header (checks both
regular and unredirected).
-- Method: Request.remove_header (header)
Remove named header from the request instance (both from regular
and unredirected headers).
Added in version 3.4.
-- Method: Request.get_full_url ()
Return the URL given in the constructor.
Changed in version 3.4.
Returns *note Request.full_url: fd5.
-- Method: Request.set_proxy (host, type)
Prepare the request by connecting to a proxy server. The 'host'
and 'type' will replace those of the instance, and the instance’s
selector will be the original URL given in the constructor.
-- Method: Request.get_header (header_name, default=None)
Return the value of the given header. If the header is not
present, return the default value.
-- Method: Request.header_items ()
Return a list of tuples (header_name, header_value) of the Request
headers.
Changed in version 3.4: The request methods add_data, has_data,
get_data, get_type, get_host, get_selector, get_origin_req_host and
is_unverifiable that were deprecated since 3.3 have been removed.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2965.html
File: python.info, Node: OpenerDirector Objects, Next: BaseHandler Objects, Prev: Request Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.2 OpenerDirector Objects
...............................
*note OpenerDirector: 399d. instances have the following methods:
-- Method: OpenerDirector.add_handler (handler)
'handler' should be an instance of *note BaseHandler: 39a1. The
following methods are searched, and added to the possible chains
(note that HTTP errors are a special case). Note that, in the
following, 'protocol' should be replaced with the actual protocol
to handle, for example ‘http_response()’ would be the HTTP protocol
response handler. Also 'type' should be replaced with the actual
HTTP code, for example ‘http_error_404()’ would handle HTTP 404
errors.
* ‘_open()’ — signal that the handler knows how to
open 'protocol' URLs.
See *note BaseHandler._open(): 39c2. for more
information.
* ‘http_error_()’ — signal that the handler knows how to
handle HTTP errors with HTTP error code 'type'.
See *note BaseHandler.http_error_(): 39c3. for more
information.
* ‘_error()’ — signal that the handler knows how to
handle errors from (non-‘http’) 'protocol'.
* ‘_request()’ — signal that the handler knows how to
pre-process 'protocol' requests.
See *note BaseHandler._request(): 39c4. for more
information.
* ‘_response()’ — signal that the handler knows how to
post-process 'protocol' responses.
See *note BaseHandler._response(): 39c5. for more
information.
-- Method: OpenerDirector.open (url, data=None[, timeout])
Open the given 'url' (which can be a request object or a string),
optionally passing the given 'data'. Arguments, return values and
exceptions raised are the same as those of *note urlopen(): 295.
(which simply calls the *note open(): 517. method on the currently
installed global *note OpenerDirector: 399d.). The optional
'timeout' parameter specifies a timeout in seconds for blocking
operations like the connection attempt (if not specified, the
global default timeout setting will be used). The timeout feature
actually works only for HTTP, HTTPS and FTP connections.
-- Method: OpenerDirector.error (proto, *args)
Handle an error of the given protocol. This will call the
registered error handlers for the given protocol with the given
arguments (which are protocol specific). The HTTP protocol is a
special case which uses the HTTP response code to determine the
specific error handler; refer to the ‘http_error_()’ methods
of the handler classes.
Return values and exceptions raised are the same as those of *note
urlopen(): 295.
OpenerDirector objects open URLs in three stages:
The order in which these methods are called within each stage is
determined by sorting the handler instances.
1. Every handler with a method named like ‘_request()’ has
that method called to pre-process the request.
2. Handlers with a method named like ‘_open()’ are called to
handle the request. This stage ends when a handler either returns
a non-*note None: 671. value (ie. a response), or raises an
exception (usually *note URLError: 399c.). Exceptions are allowed
to propagate.
In fact, the above algorithm is first tried for methods named *note
default_open(): 39c7. If all such methods return *note None: 671,
the algorithm is repeated for methods named like
‘_open()’. If all such methods return *note None: 671,
the algorithm is repeated for methods named *note unknown_open():
39c8.
Note that the implementation of these methods may involve calls of
the parent *note OpenerDirector: 399d. instance’s *note open():
fd7. and *note error(): 39c6. methods.
3. Every handler with a method named like ‘_response()’ has
that method called to post-process the response.
File: python.info, Node: BaseHandler Objects, Next: HTTPRedirectHandler Objects, Prev: OpenerDirector Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.3 BaseHandler Objects
............................
*note BaseHandler: 39a1. objects provide a couple of methods that are
directly useful, and others that are meant to be used by derived
classes. These are intended for direct use:
-- Method: BaseHandler.add_parent (director)
Add a director as parent.
-- Method: BaseHandler.close ()
Remove any parents.
The following attribute and methods should only be used by classes
derived from *note BaseHandler: 39a1.
Note: The convention has been adopted that subclasses defining
‘_request()’ or ‘_response()’ methods are named
‘*Processor’; all others are named ‘*Handler’.
-- Attribute: BaseHandler.parent
A valid *note OpenerDirector: 399d, which can be used to open using
a different protocol, or handle errors.
-- Method: BaseHandler.default_open (req)
This method is 'not' defined in *note BaseHandler: 39a1, but
subclasses should define it if they want to catch all URLs.
This method, if implemented, will be called by the parent *note
OpenerDirector: 399d. It should return a file-like object as
described in the return value of the *note open(): fd7. method of
*note OpenerDirector: 399d, or ‘None’. It should raise *note
URLError: 399c, unless a truly exceptional thing happens (for
example, *note MemoryError: 1576. should not be mapped to *note
URLError: 399c.).
This method will be called before any protocol-specific open
method.
-- Method: BaseHandler._open(req)
This method is 'not' defined in *note BaseHandler: 39a1, but
subclasses should define it if they want to handle URLs with the
given protocol.
This method, if defined, will be called by the parent *note
OpenerDirector: 399d. Return values should be the same as for
*note default_open(): 39c7.
-- Method: BaseHandler.unknown_open (req)
This method is 'not' defined in *note BaseHandler: 39a1, but
subclasses should define it if they want to catch all URLs with no
specific registered handler to open it.
This method, if implemented, will be called by the *note parent:
39cd. *note OpenerDirector: 399d. Return values should be the same
as for *note default_open(): 39c7.
-- Method: BaseHandler.http_error_default (req, fp, code, msg, hdrs)
This method is 'not' defined in *note BaseHandler: 39a1, but
subclasses should override it if they intend to provide a catch-all
for otherwise unhandled HTTP errors. It will be called
automatically by the *note OpenerDirector: 399d. getting the error,
and should not normally be called in other circumstances.
*note OpenerDirector: 399d. will call this method with five
positional arguments:
1. a *note Request: fd3. object,
2. a file-like object with the HTTP error body,
3. the three-digit code of the error, as a string,
4. the user-visible explanation of the code, as a string, and
5. the headers of the error, as a mapping object.
Return values and exceptions raised should be the same as those of
*note urlopen(): 295.
-- Method: BaseHandler.http_error_(req, fp, code, msg, hdrs)
'nnn' should be a three-digit HTTP error code. This method is also
not defined in *note BaseHandler: 39a1, but will be called, if it
exists, on an instance of a subclass, when an HTTP error with code
'nnn' occurs.
Subclasses should override this method to handle specific HTTP
errors.
Arguments, return values and exceptions raised should be the same
as for *note http_error_default(): 39ce.
-- Method: BaseHandler._request(req)
This method is 'not' defined in *note BaseHandler: 39a1, but
subclasses should define it if they want to pre-process requests of
the given protocol.
This method, if defined, will be called by the parent *note
OpenerDirector: 399d. 'req' will be a *note Request: fd3. object.
The return value should be a *note Request: fd3. object.
-- Method: BaseHandler._response(req, response)
This method is 'not' defined in *note BaseHandler: 39a1, but
subclasses should define it if they want to post-process responses
of the given protocol.
This method, if defined, will be called by the parent *note
OpenerDirector: 399d. 'req' will be a *note Request: fd3. object.
'response' will be an object implementing the same interface as the
return value of *note urlopen(): 295. The return value should
implement the same interface as the return value of *note
urlopen(): 295.
File: python.info, Node: HTTPRedirectHandler Objects, Next: HTTPCookieProcessor Objects, Prev: BaseHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.4 HTTPRedirectHandler Objects
....................................
Note: Some HTTP redirections require action from this module’s
client code. If this is the case, *note HTTPError: fd9. is raised.
See RFC 2616(1) for details of the precise meanings of the various
redirection codes.
An *note HTTPError: fd9. exception raised as a security
consideration if the HTTPRedirectHandler is presented with a
redirected URL which is not an HTTP, HTTPS or FTP URL.
-- Method: HTTPRedirectHandler.redirect_request (req, fp, code, msg,
hdrs, newurl)
Return a *note Request: fd3. or ‘None’ in response to a redirect.
This is called by the default implementations of the
‘http_error_30*()’ methods when a redirection is received from the
server. If a redirection should take place, return a new *note
Request: fd3. to allow ‘http_error_30*()’ to perform the redirect
to 'newurl'. Otherwise, raise *note HTTPError: fd9. if no other
handler should try to handle this URL, or return ‘None’ if you
can’t but another handler might.
Note: The default implementation of this method does not
strictly follow RFC 2616(2), which says that 301 and 302
responses to ‘POST’ requests must not be automatically
redirected without confirmation by the user. In reality,
browsers do allow automatic redirection of these responses,
changing the POST to a ‘GET’, and the default implementation
reproduces this behavior.
-- Method: HTTPRedirectHandler.http_error_301 (req, fp, code, msg,
hdrs)
Redirect to the ‘Location:’ or ‘URI:’ URL. This method is called by
the parent *note OpenerDirector: 399d. when getting an HTTP ‘moved
permanently’ response.
-- Method: HTTPRedirectHandler.http_error_302 (req, fp, code, msg,
hdrs)
The same as *note http_error_301(): 39d2, but called for the
‘found’ response.
-- Method: HTTPRedirectHandler.http_error_303 (req, fp, code, msg,
hdrs)
The same as *note http_error_301(): 39d2, but called for the ‘see
other’ response.
-- Method: HTTPRedirectHandler.http_error_307 (req, fp, code, msg,
hdrs)
The same as *note http_error_301(): 39d2, but called for the
‘temporary redirect’ response. It does not allow changing the
request method from ‘POST’ to ‘GET’.
-- Method: HTTPRedirectHandler.http_error_308 (req, fp, code, msg,
hdrs)
The same as *note http_error_301(): 39d2, but called for the
‘permanent redirect’ response. It does not allow changing the
request method from ‘POST’ to ‘GET’.
Added in version 3.11.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2616.html
(2) https://datatracker.ietf.org/doc/html/rfc2616.html
File: python.info, Node: HTTPCookieProcessor Objects, Next: ProxyHandler Objects, Prev: HTTPRedirectHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.5 HTTPCookieProcessor Objects
....................................
*note HTTPCookieProcessor: 39a9. instances have one attribute:
-- Attribute: HTTPCookieProcessor.cookiejar
The *note http.cookiejar.CookieJar: 39da. in which cookies are
stored.
File: python.info, Node: ProxyHandler Objects, Next: HTTPPasswordMgr Objects, Prev: HTTPCookieProcessor Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.6 ProxyHandler Objects
.............................
-- Method: ProxyHandler._open(request)
The *note ProxyHandler: 19cf. will have a method
‘_open()’ for every 'protocol' which has a proxy in the
'proxies' dictionary given in the constructor. The method will
modify requests to go through the proxy, by calling
‘request.set_proxy()’, and call the next handler in the chain to
actually execute the protocol.
File: python.info, Node: HTTPPasswordMgr Objects, Next: HTTPPasswordMgrWithPriorAuth Objects, Prev: ProxyHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.7 HTTPPasswordMgr Objects
................................
These methods are available on *note HTTPPasswordMgr: 39aa. and *note
HTTPPasswordMgrWithDefaultRealm: 39ab. objects.
-- Method: HTTPPasswordMgr.add_password (realm, uri, user, passwd)
'uri' can be either a single URI, or a sequence of URIs. 'realm',
'user' and 'passwd' must be strings. This causes ‘(user, passwd)’
to be used as authentication tokens when authentication for 'realm'
and a super-URI of any of the given URIs is given.
-- Method: HTTPPasswordMgr.find_user_password (realm, authuri)
Get user/password for given realm and URI, if any. This method
will return ‘(None, None)’ if there is no matching user/password.
For *note HTTPPasswordMgrWithDefaultRealm: 39ab. objects, the realm
‘None’ will be searched if the given 'realm' has no matching
user/password.
File: python.info, Node: HTTPPasswordMgrWithPriorAuth Objects, Next: AbstractBasicAuthHandler Objects, Prev: HTTPPasswordMgr Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.8 HTTPPasswordMgrWithPriorAuth Objects
.............................................
This password manager extends *note HTTPPasswordMgrWithDefaultRealm:
39ab. to support tracking URIs for which authentication credentials
should always be sent.
-- Method: HTTPPasswordMgrWithPriorAuth.add_password (realm, uri, user,
passwd, is_authenticated=False)
'realm', 'uri', 'user', 'passwd' are as for *note
HTTPPasswordMgr.add_password(): 39de. 'is_authenticated' sets the
initial value of the ‘is_authenticated’ flag for the given URI or
list of URIs. If 'is_authenticated' is specified as ‘True’,
'realm' is ignored.
-- Method: HTTPPasswordMgrWithPriorAuth.find_user_password (realm,
authuri)
Same as for *note HTTPPasswordMgrWithDefaultRealm: 39ab. objects
-- Method: HTTPPasswordMgrWithPriorAuth.update_authenticated (self,
uri, is_authenticated=False)
Update the ‘is_authenticated’ flag for the given 'uri' or list of
URIs.
-- Method: HTTPPasswordMgrWithPriorAuth.is_authenticated (self,
authuri)
Returns the current state of the ‘is_authenticated’ flag for the
given URI.
File: python.info, Node: AbstractBasicAuthHandler Objects, Next: HTTPBasicAuthHandler Objects, Prev: HTTPPasswordMgrWithPriorAuth Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.9 AbstractBasicAuthHandler Objects
.........................................
-- Method: AbstractBasicAuthHandler.http_error_auth_reqed (authreq,
host, req, headers)
Handle an authentication request by getting a user/password pair,
and re-trying the request. 'authreq' should be the name of the
header where the information about the realm is included in the
request, 'host' specifies the URL and path to authenticate for,
'req' should be the (failed) *note Request: fd3. object, and
'headers' should be the error headers.
'host' is either an authority (e.g. ‘"python.org"’) or a URL
containing an authority component (e.g. ‘"http://python.org/"’).
In either case, the authority must not contain a userinfo component
(so, ‘"python.org"’ and ‘"python.org:80"’ are fine,
‘"joe:password@python.org"’ is not).
File: python.info, Node: HTTPBasicAuthHandler Objects, Next: ProxyBasicAuthHandler Objects, Prev: AbstractBasicAuthHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.10 HTTPBasicAuthHandler Objects
......................................
-- Method: HTTPBasicAuthHandler.http_error_401 (req, fp, code, msg,
hdrs)
Retry the request with authentication information, if available.
File: python.info, Node: ProxyBasicAuthHandler Objects, Next: AbstractDigestAuthHandler Objects, Prev: HTTPBasicAuthHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.11 ProxyBasicAuthHandler Objects
.......................................
-- Method: ProxyBasicAuthHandler.http_error_407 (req, fp, code, msg,
hdrs)
Retry the request with authentication information, if available.
File: python.info, Node: AbstractDigestAuthHandler Objects, Next: HTTPDigestAuthHandler Objects, Prev: ProxyBasicAuthHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.12 AbstractDigestAuthHandler Objects
...........................................
-- Method: AbstractDigestAuthHandler.http_error_auth_reqed (authreq,
host, req, headers)
'authreq' should be the name of the header where the information
about the realm is included in the request, 'host' should be the
host to authenticate to, 'req' should be the (failed) *note
Request: fd3. object, and 'headers' should be the error headers.
File: python.info, Node: HTTPDigestAuthHandler Objects, Next: ProxyDigestAuthHandler Objects, Prev: AbstractDigestAuthHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.13 HTTPDigestAuthHandler Objects
.......................................
-- Method: HTTPDigestAuthHandler.http_error_401 (req, fp, code, msg,
hdrs)
Retry the request with authentication information, if available.
File: python.info, Node: ProxyDigestAuthHandler Objects, Next: HTTPHandler Objects, Prev: HTTPDigestAuthHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.14 ProxyDigestAuthHandler Objects
........................................
-- Method: ProxyDigestAuthHandler.http_error_407 (req, fp, code, msg,
hdrs)
Retry the request with authentication information, if available.
File: python.info, Node: HTTPHandler Objects, Next: HTTPSHandler Objects, Prev: ProxyDigestAuthHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.15 HTTPHandler Objects
.............................
-- Method: HTTPHandler.http_open (req)
Send an HTTP request, which can be either GET or POST, depending on
‘req.data’.
File: python.info, Node: HTTPSHandler Objects, Next: FileHandler Objects, Prev: HTTPHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.16 HTTPSHandler Objects
..............................
-- Method: HTTPSHandler.https_open (req)
Send an HTTPS request, which can be either GET or POST, depending
on ‘req.data’.
File: python.info, Node: FileHandler Objects, Next: DataHandler Objects, Prev: HTTPSHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.17 FileHandler Objects
.............................
-- Method: FileHandler.file_open (req)
Open the file locally, if there is no host name, or the host name
is ‘'localhost'’.
Changed in version 3.2: This method is applicable only for local
hostnames. When a remote hostname is given, a *note URLError:
399c. is raised.
File: python.info, Node: DataHandler Objects, Next: FTPHandler Objects, Prev: FileHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.18 DataHandler Objects
.............................
-- Method: DataHandler.data_open (req)
Read a data URL. This kind of URL contains the content encoded in
the URL itself. The data URL syntax is specified in RFC 2397(1).
This implementation ignores white spaces in base64 encoded data
URLs so the URL may be wrapped in whatever source file it comes
from. But even though some browsers don’t mind about a missing
padding at the end of a base64 encoded data URL, this
implementation will raise a *note ValueError: 204. in that case.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2397.html
File: python.info, Node: FTPHandler Objects, Next: CacheFTPHandler Objects, Prev: DataHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.19 FTPHandler Objects
............................
-- Method: FTPHandler.ftp_open (req)
Open the FTP file indicated by 'req'. The login is always done
with empty username and password.
File: python.info, Node: CacheFTPHandler Objects, Next: UnknownHandler Objects, Prev: FTPHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.20 CacheFTPHandler Objects
.................................
*note CacheFTPHandler: 1728. objects are *note FTPHandler: 39a5. objects
with the following additional methods:
-- Method: CacheFTPHandler.setTimeout (t)
Set timeout of connections to 't' seconds.
-- Method: CacheFTPHandler.setMaxConns (m)
Set maximum number of cached connections to 'm'.
File: python.info, Node: UnknownHandler Objects, Next: HTTPErrorProcessor Objects, Prev: CacheFTPHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.21 UnknownHandler Objects
................................
-- Method: UnknownHandler.unknown_open ()
Raise a *note URLError: 399c. exception.
File: python.info, Node: HTTPErrorProcessor Objects, Next: Examples<26>, Prev: UnknownHandler Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.22 HTTPErrorProcessor Objects
....................................
-- Method: HTTPErrorProcessor.http_response (request, response)
Process HTTP error responses.
For 200 error codes, the response object is returned immediately.
For non-200 error codes, this simply passes the job on to the
‘http_error_()’ handler methods, via *note
OpenerDirector.error(): 39c6. Eventually, *note
HTTPDefaultErrorHandler: 39a3. will raise an *note HTTPError: fd9.
if no other handler handles the error.
-- Method: HTTPErrorProcessor.https_response (request, response)
Process HTTPS error responses.
The behavior is same as *note http_response(): 3a0d.
File: python.info, Node: Examples<26>, Next: Legacy interface, Prev: HTTPErrorProcessor Objects, Up: urllib request — Extensible library for opening URLs
5.22.4.23 Examples
..................
In addition to the examples below, more examples are given in *note
HOWTO Fetch Internet Resources Using The urllib Package: 3a11.
This example gets the python.org main page and displays the first 300
bytes of it:
>>> import urllib.request
>>> with urllib.request.urlopen('http://www.python.org/') as f:
... print(f.read(300))
...
b'\n\n\n
>> import urllib.request
>>> f = urllib.request.urlopen('http://www.python.org/')
>>> try:
... print(f.read(100).decode('utf-8'))
... finally:
... f.close()
...
| UnixStreamServer |
+-----------+ +------------------+
|
v
+-----------+ +--------------------+
| UDPServer |------->| UnixDatagramServer |
+-----------+ +--------------------+
Note that *note UnixDatagramServer: 3b3b. derives from *note UDPServer:
3b39, not from *note UnixStreamServer: 3b3a. — the only difference
between an IP and a Unix server is the address family.
-- Class: socketserver.ForkingMixIn
-- Class: socketserver.ThreadingMixIn
Forking and threading versions of each type of server can be
created using these mix-in classes. For instance, *note
ThreadingUDPServer: 3b40. is created as follows:
class ThreadingUDPServer(ThreadingMixIn, UDPServer):
pass
The mix-in class comes first, since it overrides a method defined
in *note UDPServer: 3b39. Setting the various attributes also
changes the behavior of the underlying server mechanism.
*note ForkingMixIn: b77. and the Forking classes mentioned below
are only available on POSIX platforms that support *note fork():
197.
-- Attribute: block_on_close
*note ForkingMixIn.server_close: 3b3e. waits until all child
processes complete, except if *note block_on_close: bfd.
attribute is ‘False’.
*note ThreadingMixIn.server_close: 3b3e. waits until all
non-daemon threads complete, except if *note block_on_close:
bfd. attribute is ‘False’.
-- Attribute: daemon_threads
For *note ThreadingMixIn: b78. use daemonic threads by setting
*note ThreadingMixIn.daemon_threads: 3b41. to ‘True’ to not
wait until threads complete.
Changed in version 3.7: *note ForkingMixIn.server_close: 3b3e. and
*note ThreadingMixIn.server_close: 3b3e. now waits until all child
processes and non-daemonic threads complete. Add a new *note
ForkingMixIn.block_on_close: bfd. class attribute to opt-in for the
pre-3.7 behaviour.
-- Class: socketserver.ForkingTCPServer
-- Class: socketserver.ForkingUDPServer
-- Class: socketserver.ThreadingTCPServer
-- Class: socketserver.ThreadingUDPServer
-- Class: socketserver.ForkingUnixStreamServer
-- Class: socketserver.ForkingUnixDatagramServer
-- Class: socketserver.ThreadingUnixStreamServer
-- Class: socketserver.ThreadingUnixDatagramServer
These classes are pre-defined using the mix-in classes.
Added in version 3.12: The ‘ForkingUnixStreamServer’ and
‘ForkingUnixDatagramServer’ classes were added.
To implement a service, you must derive a class from *note
BaseRequestHandler: 3b3c. and redefine its *note handle(): 3b3d. method.
You can then run various versions of the service by combining one of the
server classes with your request handler class. The request handler
class must be different for datagram or stream services. This can be
hidden by using the handler subclasses *note StreamRequestHandler: cda.
or *note DatagramRequestHandler: 3b49.
Of course, you still have to use your head! For instance, it makes no
sense to use a forking server if the service contains state in memory
that can be modified by different requests, since the modifications in
the child process would never reach the initial state kept in the parent
process and passed to each child. In this case, you can use a threading
server, but you will probably have to use locks to protect the integrity
of the shared data.
On the other hand, if you are building an HTTP server where all data is
stored externally (for instance, in the file system), a synchronous
class will essentially render the service “deaf” while one request is
being handled – which may be for a very long time if a client is slow to
receive all the data it has requested. Here a threading or forking
server is appropriate.
In some cases, it may be appropriate to process part of a request
synchronously, but to finish processing in a forked child depending on
the request data. This can be implemented by using a synchronous server
and doing an explicit fork in the request handler class *note handle():
3b3d. method.
Another approach to handling multiple simultaneous requests in an
environment that supports neither threads nor *note fork(): 197. (or
where these are too expensive or inappropriate for the service) is to
maintain an explicit table of partially finished requests and to use
*note selectors: c2. to decide which request to work on next (or whether
to handle a new incoming request). This is particularly important for
stream services where each client can potentially be connected for a
long time (if threads or subprocesses cannot be used).
File: python.info, Node: Server Objects<2>, Next: Request Handler Objects, Prev: Server Creation Notes, Up: socketserver — A framework for network servers
5.22.16.2 Server Objects
........................
-- Class: socketserver.BaseServer (server_address, RequestHandlerClass)
This is the superclass of all Server objects in the module. It
defines the interface, given below, but does not implement most of
the methods, which is done in subclasses. The two parameters are
stored in the respective *note server_address: 3b4b. and *note
RequestHandlerClass: 3b4c. attributes.
-- Method: fileno ()
Return an integer file descriptor for the socket on which the
server is listening. This function is most commonly passed to
*note selectors: c2, to allow monitoring multiple servers in
the same process.
-- Method: handle_request ()
Process a single request. This function calls the following
methods in order: *note get_request(): 3b4e, *note
verify_request(): 3b4f, and *note process_request(): 3b50. If
the user-provided *note handle(): 3b3d. method of the handler
class raises an exception, the server’s *note handle_error():
d41. method will be called. If no request is received within
*note timeout: 1313. seconds, *note handle_timeout(): 1314.
will be called and *note handle_request(): 1315. will return.
-- Method: serve_forever (poll_interval=0.5)
Handle requests until an explicit *note shutdown(): 1a34.
request. Poll for shutdown every 'poll_interval' seconds.
Ignores the *note timeout: 1313. attribute. It also calls
*note service_actions(): 1132, which may be used by a subclass
or mixin to provide actions specific to a given service. For
example, the *note ForkingMixIn: b77. class uses *note
service_actions(): 1132. to clean up zombie child processes.
Changed in version 3.3: Added ‘service_actions’ call to the
‘serve_forever’ method.
-- Method: service_actions ()
This is called in the *note serve_forever(): 1133. loop. This
method can be overridden by subclasses or mixin classes to
perform actions specific to a given service, such as cleanup
actions.
Added in version 3.3.
-- Method: shutdown ()
Tell the *note serve_forever(): 1133. loop to stop and wait
until it does. *note shutdown(): 1a34. must be called while
*note serve_forever(): 1133. is running in a different thread
otherwise it will deadlock.
-- Method: server_close ()
Clean up the server. May be overridden.
-- Attribute: address_family
The family of protocols to which the server’s socket belongs.
Common examples are *note socket.AF_INET: 181c, *note
socket.AF_INET6: 181d, and *note socket.AF_UNIX: 181e.
Subclass the TCP or UDP server classes in this module with
class attribute ‘address_family = AF_INET6’ set if you want
IPv6 server classes.
-- Attribute: RequestHandlerClass
The user-provided request handler class; an instance of this
class is created for each request.
-- Attribute: server_address
The address on which the server is listening. The format of
addresses varies depending on the protocol family; see the
documentation for the *note socket: cc. module for details.
For internet protocols, this is a tuple containing a string
giving the address, and an integer port number: ‘('127.0.0.1',
80)’, for example.
-- Attribute: socket
The socket object on which the server will listen for incoming
requests.
The server classes support the following class variables:
-- Attribute: allow_reuse_address
Whether the server will allow the reuse of an address. This
defaults to *note False: b37, and can be set in subclasses to
change the policy.
-- Attribute: request_queue_size
The size of the request queue. If it takes a long time to
process a single request, any requests that arrive while the
server is busy are placed into a queue, up to *note
request_queue_size: 3b54. requests. Once the queue is full,
further requests from clients will get a “Connection denied”
error. The default value is usually 5, but this can be
overridden by subclasses.
-- Attribute: socket_type
The type of socket used by the server; *note
socket.SOCK_STREAM: 12e6. and *note socket.SOCK_DGRAM: 12e5.
are two common values.
-- Attribute: timeout
Timeout duration, measured in seconds, or *note None: 671. if
no timeout is desired. If *note handle_request(): 1315.
receives no incoming requests within the timeout period, the
*note handle_timeout(): 1314. method is called.
There are various server methods that can be overridden by
subclasses of base server classes like *note TCPServer: 1312.;
these methods aren’t useful to external users of the server object.
-- Method: finish_request (request, client_address)
Actually processes the request by instantiating *note
RequestHandlerClass: 3b4c. and calling its *note handle():
3b3d. method.
-- Method: get_request ()
Must accept a request from the socket, and return a 2-tuple
containing the 'new' socket object to be used to communicate
with the client, and the client’s address.
-- Method: handle_error (request, client_address)
This function is called if the *note handle(): 3b3d. method of
a *note RequestHandlerClass: 3b4c. instance raises an
exception. The default action is to print the traceback to
standard error and continue handling further requests.
Changed in version 3.6: Now only called for exceptions derived
from the *note Exception: 9d9. class.
-- Method: handle_timeout ()
This function is called when the *note timeout: 1313.
attribute has been set to a value other than *note None: 671.
and the timeout period has passed with no requests being
received. The default action for forking servers is to
collect the status of any child processes that have exited,
while in threading servers this method does nothing.
-- Method: process_request (request, client_address)
Calls *note finish_request(): 3b56. to create an instance of
the *note RequestHandlerClass: 3b4c. If desired, this
function can create a new process or thread to handle the
request; the *note ForkingMixIn: b77. and *note
ThreadingMixIn: b78. classes do this.
-- Method: server_activate ()
Called by the server’s constructor to activate the server.
The default behavior for a TCP server just invokes *note
listen(): e59. on the server’s socket. May be overridden.
-- Method: server_bind ()
Called by the server’s constructor to bind the socket to the
desired address. May be overridden.
-- Method: verify_request (request, client_address)
Must return a Boolean value; if the value is *note True: c0d,
the request will be processed, and if it’s *note False: b37,
the request will be denied. This function can be overridden
to implement access controls for a server. The default
implementation always returns *note True: c0d.
Changed in version 3.6: Support for the *note context manager: 5d0.
protocol was added. Exiting the context manager is equivalent to
calling *note server_close(): 3b3e.
File: python.info, Node: Request Handler Objects, Next: Examples<28>, Prev: Server Objects<2>, Up: socketserver — A framework for network servers
5.22.16.3 Request Handler Objects
.................................
-- Class: socketserver.BaseRequestHandler
This is the superclass of all request handler objects. It defines
the interface, given below. A concrete request handler subclass
must define a new *note handle(): 3b3d. method, and can override
any of the other methods. A new instance of the subclass is
created for each request.
-- Method: setup ()
Called before the *note handle(): 3b3d. method to perform any
initialization actions required. The default implementation
does nothing.
-- Method: handle ()
This function must do all the work required to service a
request. The default implementation does nothing. Several
instance attributes are available to it; the request is
available as *note request: 3b59.; the client address as *note
client_address: 3b5a.; and the server instance as *note
server: 3b5b, in case it needs access to per-server
information.
The type of *note request: 3b59. is different for datagram or
stream services. For stream services, *note request: 3b59. is
a socket object; for datagram services, *note request: 3b59.
is a pair of string and socket.
-- Method: finish ()
Called after the *note handle(): 3b3d. method to perform any
clean-up actions required. The default implementation does
nothing. If *note setup(): 3b58. raises an exception, this
function will not be called.
-- Attribute: request
The 'new' *note socket.socket: da0. object to be used to
communicate with the client.
-- Attribute: client_address
Client address returned by *note BaseServer.get_request():
3b4e.
-- Attribute: server
*note BaseServer: 1131. object used for handling the request.
-- Class: socketserver.StreamRequestHandler
-- Class: socketserver.DatagramRequestHandler
These *note BaseRequestHandler: 3b3c. subclasses override the *note
setup(): 3b58. and *note finish(): 3b5c. methods, and provide *note
rfile: 3b5d. and *note wfile: 3b5e. attributes.
-- Attribute: rfile
A file object from which receives the request is read.
Support the *note io.BufferedIOBase: 690. readable interface.
-- Attribute: wfile
A file object to which the reply is written. Support the
*note io.BufferedIOBase: 690. writable interface
Changed in version 3.6: *note wfile: 3b5e. also supports the *note
io.BufferedIOBase: 690. writable interface.
File: python.info, Node: Examples<28>, Prev: Request Handler Objects, Up: socketserver — A framework for network servers
5.22.16.4 Examples
..................
* Menu:
* socketserver.TCPServer Example: socketserver TCPServer Example.
* socketserver.UDPServer Example: socketserver UDPServer Example.
* Asynchronous Mixins::
File: python.info, Node: socketserver TCPServer Example, Next: socketserver UDPServer Example, Up: Examples<28>
5.22.16.5 ‘socketserver.TCPServer’ Example
..........................................
This is the server side:
import socketserver
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The request handler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
pieces = [b'']
total = 0
while b'\n' not in pieces[-1] and total < 10_000:
pieces.append(self.request.recv(2000))
total += len(pieces[-1])
self.data = b''.join(pieces)
print(f"Received from {self.client_address[0]}:")
print(self.data.decode("utf-8"))
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
# after we return, the socket will be closed.
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()
An alternative request handler class that makes use of streams
(file-like objects that simplify communication by providing the standard
file interface):
class MyTCPHandler(socketserver.StreamRequestHandler):
def handle(self):
# self.rfile is a file-like object created by the handler.
# We can now use e.g. readline() instead of raw recv() calls.
# We limit ourselves to 10000 bytes to avoid abuse by the sender.
self.data = self.rfile.readline(10000).rstrip()
print(f"{self.client_address[0]} wrote:")
print(self.data.decode("utf-8"))
# Likewise, self.wfile is a file-like object used to write back
# to the client
self.wfile.write(self.data.upper())
The difference is that the ‘readline()’ call in the second handler will
call ‘recv()’ multiple times until it encounters a newline character,
while the first handler had to use a ‘recv()’ loop to accumulate data
until a newline itself. If it had just used a single ‘recv()’ without
the loop it would just have returned what has been received so far from
the client. TCP is stream based: data arrives in the order it was sent,
but there no correlation between client ‘send()’ or ‘sendall()’ calls
and the number of ‘recv()’ calls on the server required to receive it.
This is the client side:
import socket
import sys
HOST, PORT = "localhost", 9999
data = " ".join(sys.argv[1:])
# Create a socket (SOCK_STREAM means a TCP socket)
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
# Connect to server and send data
sock.connect((HOST, PORT))
sock.sendall(bytes(data, "utf-8"))
sock.sendall(b"\n")
# Receive data from the server and shut down
received = str(sock.recv(1024), "utf-8")
print("Sent: ", data)
print("Received:", received)
The output of the example should look something like this:
Server:
$ python TCPServer.py
127.0.0.1 wrote:
b'hello world with TCP'
127.0.0.1 wrote:
b'python is nice'
Client:
$ python TCPClient.py hello world with TCP
Sent: hello world with TCP
Received: HELLO WORLD WITH TCP
$ python TCPClient.py python is nice
Sent: python is nice
Received: PYTHON IS NICE
File: python.info, Node: socketserver UDPServer Example, Next: Asynchronous Mixins, Prev: socketserver TCPServer Example, Up: Examples<28>
5.22.16.6 ‘socketserver.UDPServer’ Example
..........................................
This is the server side:
import socketserver
class MyUDPHandler(socketserver.BaseRequestHandler):
"""
This class works similar to the TCP handler class, except that
self.request consists of a pair of data and client socket, and since
there is no connection the client address must be given explicitly
when sending data back via sendto().
"""
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
print(f"{self.client_address[0]} wrote:")
print(data)
socket.sendto(data.upper(), self.client_address)
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
with socketserver.UDPServer((HOST, PORT), MyUDPHandler) as server:
server.serve_forever()
This is the client side:
import socket
import sys
HOST, PORT = "localhost", 9999
data = " ".join(sys.argv[1:])
# SOCK_DGRAM is the socket type to use for UDP sockets
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# As you can see, there is no connect() call; UDP has no connections.
# Instead, data is directly sent to the recipient via sendto().
sock.sendto(bytes(data + "\n", "utf-8"), (HOST, PORT))
received = str(sock.recv(1024), "utf-8")
print("Sent: ", data)
print("Received:", received)
The output of the example should look exactly like for the TCP server
example.
File: python.info, Node: Asynchronous Mixins, Prev: socketserver UDPServer Example, Up: Examples<28>
5.22.16.7 Asynchronous Mixins
.............................
To build asynchronous handlers, use the *note ThreadingMixIn: b78. and
*note ForkingMixIn: b77. classes.
An example for the *note ThreadingMixIn: b78. class:
import socket
import threading
import socketserver
class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler):
def handle(self):
data = str(self.request.recv(1024), 'ascii')
cur_thread = threading.current_thread()
response = bytes("{}: {}".format(cur_thread.name, data), 'ascii')
self.request.sendall(response)
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def client(ip, port, message):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect((ip, port))
sock.sendall(bytes(message, 'ascii'))
response = str(sock.recv(1024), 'ascii')
print("Received: {}".format(response))
if __name__ == "__main__":
# Port 0 means to select an arbitrary unused port
HOST, PORT = "localhost", 0
server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
with server:
ip, port = server.server_address
# Start a thread with the server -- that thread will then start one
# more thread for each request
server_thread = threading.Thread(target=server.serve_forever)
# Exit the server thread when the main thread terminates
server_thread.daemon = True
server_thread.start()
print("Server loop running in thread:", server_thread.name)
client(ip, port, "Hello World 1")
client(ip, port, "Hello World 2")
client(ip, port, "Hello World 3")
server.shutdown()
The output of the example should look something like this:
$ python ThreadedTCPServer.py
Server loop running in thread: Thread-1
Received: Thread-2: Hello World 1
Received: Thread-3: Hello World 2
Received: Thread-4: Hello World 3
The *note ForkingMixIn: b77. class is used in the same way, except that
the server will spawn a new process for each request. Available only on
POSIX platforms that support *note fork(): 197.
File: python.info, Node: http server — HTTP servers, Next: http cookies — HTTP state management, Prev: socketserver — A framework for network servers, Up: Internet Protocols and Support
5.22.17 ‘http.server’ — HTTP servers
------------------------------------
'Source code:' Lib/http/server.py(1)
__________________________________________________________________
This module defines classes for implementing HTTP servers.
Warning: *note http.server: 72. is not recommended for production.
It only implements *note basic security checks: 3b65.
*note Availability: 1d54.: not WASI.
This module does not work or is not available on WebAssembly. See *note
WebAssembly platforms: 17e0. for more information.
One class, *note HTTPServer: 396b, is a *note socketserver.TCPServer:
1312. subclass. It creates and listens at the HTTP socket, dispatching
the requests to a handler. Code to create and run the server looks like
this:
def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
server_address = ('', 8000)
httpd = server_class(server_address, handler_class)
httpd.serve_forever()
-- Class: http.server.HTTPServer (server_address, RequestHandlerClass)
This class builds on the *note TCPServer: 1312. class by storing
the server address as instance variables named ‘server_name’ and
‘server_port’. The server is accessible by the handler, typically
through the handler’s ‘server’ instance variable.
-- Class: http.server.ThreadingHTTPServer (server_address,
RequestHandlerClass)
This class is identical to HTTPServer but uses threads to handle
requests by using the *note ThreadingMixIn: b78. This is useful to
handle web browsers pre-opening sockets, on which *note HTTPServer:
396b. would wait indefinitely.
Added in version 3.7.
The *note HTTPServer: 396b. and *note ThreadingHTTPServer: b44. must be
given a 'RequestHandlerClass' on instantiation, of which this module
provides three different variants:
-- Class: http.server.BaseHTTPRequestHandler (request, client_address,
server)
This class is used to handle the HTTP requests that arrive at the
server. By itself, it cannot respond to any actual HTTP requests;
it must be subclassed to handle each request method (e.g. GET or
POST). *note BaseHTTPRequestHandler: 10c1. provides a number of
class and instance variables, and methods for use by subclasses.
The handler will parse the request and the headers, then call a
method specific to the request type. The method name is
constructed from the request. For example, for the request method
‘SPAM’, the ‘do_SPAM()’ method will be called with no arguments.
All of the relevant information is stored in instance variables of
the handler. Subclasses should not need to override or extend the
‘__init__()’ method.
*note BaseHTTPRequestHandler: 10c1. has the following instance
variables:
-- Attribute: client_address
Contains a tuple of the form ‘(host, port)’ referring to the
client’s address.
-- Attribute: server
Contains the server instance.
-- Attribute: close_connection
Boolean that should be set before *note handle_one_request():
3b69. returns, indicating if another request may be expected,
or if the connection should be shut down.
-- Attribute: requestline
Contains the string representation of the HTTP request line.
The terminating CRLF is stripped. This attribute should be
set by *note handle_one_request(): 3b69. If no valid request
line was processed, it should be set to the empty string.
-- Attribute: command
Contains the command (request type). For example, ‘'GET'’.
-- Attribute: path
Contains the request path. If query component of the URL is
present, then ‘path’ includes the query. Using the
terminology of RFC 3986(2), ‘path’ here includes ‘hier-part’
and the ‘query’.
-- Attribute: request_version
Contains the version string from the request. For example,
‘'HTTP/1.0'’.
-- Attribute: headers
Holds an instance of the class specified by the *note
MessageClass: 3b6e. class variable. This instance parses and
manages the headers in the HTTP request. The *note
parse_headers(): 1a48. function from *note http.client: 6f. is
used to parse the headers and it requires that the HTTP
request provide a valid RFC 2822(3) style header.
-- Attribute: rfile
An *note io.BufferedIOBase: 690. input stream, ready to read
from the start of the optional input data.
-- Attribute: wfile
Contains the output stream for writing a response back to the
client. Proper adherence to the HTTP protocol must be used
when writing to this stream in order to achieve successful
interoperation with HTTP clients.
Changed in version 3.6: This is an *note io.BufferedIOBase:
690. stream.
*note BaseHTTPRequestHandler: 10c1. has the following attributes:
-- Attribute: server_version
Specifies the server software version. You may want to
override this. The format is multiple whitespace-separated
strings, where each string is of the form name[/version]. For
example, ‘'BaseHTTP/0.2'’.
-- Attribute: sys_version
Contains the Python system version, in a form usable by the
*note version_string: 3b73. method and the *note
server_version: 3b71. class variable. For example,
‘'Python/1.4'’.
-- Attribute: error_message_format
Specifies a format string that should be used by *note
send_error(): f4b. method for building an error response to
the client. The string is filled by default with variables
from *note responses: 3a3a. based on the status code that
passed to *note send_error(): f4b.
-- Attribute: error_content_type
Specifies the Content-Type HTTP header of error responses sent
to the client. The default value is ‘'text/html'’.
-- Attribute: protocol_version
Specifies the HTTP version to which the server is conformant.
It is sent in responses to let the client know the server’s
communication capabilities for future requests. If set to
‘'HTTP/1.1'’, the server will permit HTTP persistent
connections; however, your server 'must' then include an
accurate ‘Content-Length’ header (using *note send_header():
3b77.) in all of its responses to clients. For backwards
compatibility, the setting defaults to ‘'HTTP/1.0'’.
-- Attribute: MessageClass
Specifies an *note email.message.Message: 27e.-like class to
parse HTTP headers. Typically, this is not overridden, and it
defaults to *note http.client.HTTPMessage: 3a51.
-- Attribute: responses
This attribute contains a mapping of error code integers to
two-element tuples containing a short and long message. For
example, ‘{code: (shortmessage, longmessage)}’. The
'shortmessage' is usually used as the 'message' key in an
error response, and 'longmessage' as the 'explain' key. It is
used by *note send_response_only(): 3b78. and *note
send_error(): f4b. methods.
A *note BaseHTTPRequestHandler: 10c1. instance has the following
methods:
-- Method: handle ()
Calls *note handle_one_request(): 3b69. once (or, if
persistent connections are enabled, multiple times) to handle
incoming HTTP requests. You should never need to override it;
instead, implement appropriate ‘do_*()’ methods.
-- Method: handle_one_request ()
This method will parse and dispatch the request to the
appropriate ‘do_*()’ method. You should never need to
override it.
-- Method: handle_expect_100 ()
When an HTTP/1.1 conformant server receives an ‘Expect:
100-continue’ request header it responds back with a ‘100
Continue’ followed by ‘200 OK’ headers. This method can be
overridden to raise an error if the server does not want the
client to continue. For e.g. server can choose to send ‘417
Expectation Failed’ as a response header and ‘return False’.
Added in version 3.2.
-- Method: send_error (code, message=None, explain=None)
Sends and logs a complete error reply to the client. The
numeric 'code' specifies the HTTP error code, with 'message'
as an optional, short, human readable description of the
error. The 'explain' argument can be used to provide more
detailed information about the error; it will be formatted
using the *note error_message_format: 3b74. attribute and
emitted, after a complete set of headers, as the response
body. The *note responses: 3a3a. attribute holds the default
values for 'message' and 'explain' that will be used if no
value is provided; for unknown codes the default value for
both is the string ‘???’. The body will be empty if the
method is HEAD or the response code is one of the following:
‘1XX’, ‘204 No Content’, ‘205 Reset Content’, ‘304 Not
Modified’.
Changed in version 3.4: The error response includes a
Content-Length header. Added the 'explain' argument.
-- Method: send_response (code, message=None)
Adds a response header to the headers buffer and logs the
accepted request. The HTTP response line is written to the
internal buffer, followed by 'Server' and 'Date' headers. The
values for these two headers are picked up from the *note
version_string(): 3b73. and *note date_time_string(): 3b7c.
methods, respectively. If the server does not intend to send
any other headers using the *note send_header(): 3b77. method,
then *note send_response(): 3b7b. should be followed by an
*note end_headers(): 10c2. call.
Changed in version 3.3: Headers are stored to an internal
buffer and *note end_headers(): 10c2. needs to be called
explicitly.
-- Method: send_header (keyword, value)
Adds the HTTP header to an internal buffer which will be
written to the output stream when either *note end_headers():
10c2. or *note flush_headers(): 10c3. is invoked. 'keyword'
should specify the header keyword, with 'value' specifying its
value. Note that, after the send_header calls are done, *note
end_headers(): 10c2. MUST BE called in order to complete the
operation.
Changed in version 3.2: Headers are stored in an internal
buffer.
-- Method: send_response_only (code, message=None)
Sends the response header only, used for the purposes when
‘100 Continue’ response is sent by the server to the client.
The headers not buffered and sent directly the output
stream.If the 'message' is not specified, the HTTP message
corresponding the response 'code' is sent.
Added in version 3.2.
-- Method: end_headers ()
Adds a blank line (indicating the end of the HTTP headers in
the response) to the headers buffer and calls *note
flush_headers(): 10c3.
Changed in version 3.2: The buffered headers are written to
the output stream.
-- Method: flush_headers ()
Finally send the headers to the output stream and flush the
internal headers buffer.
Added in version 3.3.
-- Method: log_request (code='-', size='-')
Logs an accepted (successful) request. 'code' should specify
the numeric HTTP code associated with the response. If a size
of the response is available, then it should be passed as the
'size' parameter.
-- Method: log_error (...)
Logs an error when a request cannot be fulfilled. By default,
it passes the message to *note log_message(): 3b7f, so it
takes the same arguments ('format' and additional values).
-- Method: log_message (format, ...)
Logs an arbitrary message to ‘sys.stderr’. This is typically
overridden to create custom error logging mechanisms. The
'format' argument is a standard printf-style format string,
where the additional arguments to *note log_message(): 3b7f.
are applied as inputs to the formatting. The client ip
address and current date and time are prefixed to every
message logged.
-- Method: version_string ()
Returns the server software’s version string. This is a
combination of the *note server_version: 3b71. and *note
sys_version: 3b72. attributes.
-- Method: date_time_string (timestamp=None)
Returns the date and time given by 'timestamp' (which must be
‘None’ or in the format returned by *note time.time(): 256.),
formatted for a message header. If 'timestamp' is omitted, it
uses the current date and time.
The result looks like ‘'Sun, 06 Nov 1994 08:49:37 GMT'’.
-- Method: log_date_time_string ()
Returns the current date and time, formatted for logging.
-- Method: address_string ()
Returns the client address.
Changed in version 3.3: Previously, a name lookup was
performed. To avoid name resolution delays, it now always
returns the IP address.
-- Class: http.server.SimpleHTTPRequestHandler (request,
client_address, server, directory=None)
This class serves files from the directory 'directory' and below,
or the current directory if 'directory' is not provided, directly
mapping the directory structure to HTTP requests.
Changed in version 3.7: Added the 'directory' parameter.
Changed in version 3.9: The 'directory' parameter accepts a *note
path-like object: 2a2.
A lot of the work, such as parsing the request, is done by the base
class *note BaseHTTPRequestHandler: 10c1. This class implements
the *note do_GET(): 3b82. and *note do_HEAD(): 3b83. functions.
The following are defined as class-level attributes of *note
SimpleHTTPRequestHandler: b43.:
-- Attribute: server_version
This will be ‘"SimpleHTTP/" + __version__’, where
‘__version__’ is defined at the module level.
-- Attribute: extensions_map
A dictionary mapping suffixes into MIME types, contains custom
overrides for the default system mappings. The mapping is
used case-insensitively, and so should contain only
lower-cased keys.
Changed in version 3.9: This dictionary is no longer filled
with the default system mappings, but only contains overrides.
The *note SimpleHTTPRequestHandler: b43. class defines the
following methods:
-- Method: do_HEAD ()
This method serves the ‘'HEAD'’ request type: it sends the
headers it would send for the equivalent ‘GET’ request. See
the *note do_GET(): 3b82. method for a more complete
explanation of the possible headers.
-- Method: do_GET ()
The request is mapped to a local file by interpreting the
request as a path relative to the current working directory.
If the request was mapped to a directory, the directory is
checked for a file named ‘index.html’ or ‘index.htm’ (in that
order). If found, the file’s contents are returned; otherwise
a directory listing is generated by calling the
‘list_directory()’ method. This method uses *note
os.listdir(): 10ee. to scan the directory, and returns a ‘404’
error response if the *note listdir(): 10ee. fails.
If the request was mapped to a file, it is opened. Any *note
OSError: 413. exception in opening the requested file is
mapped to a ‘404’, ‘'File not found'’ error. If there was an
‘'If-Modified-Since'’ header in the request, and the file was
not modified after this time, a ‘304’, ‘'Not Modified'’
response is sent. Otherwise, the content type is guessed by
calling the ‘guess_type()’ method, which in turn uses the
'extensions_map' variable, and the file contents are returned.
A ‘'Content-type:'’ header with the guessed content type is
output, followed by a ‘'Content-Length:'’ header with the
file’s size and a ‘'Last-Modified:'’ header with the file’s
modification time.
Then follows a blank line signifying the end of the headers,
and then the contents of the file are output.
For example usage, see the implementation of the ‘test’
function in Lib/http/server.py(4).
Changed in version 3.7: Support of the ‘'If-Modified-Since'’
header.
The *note SimpleHTTPRequestHandler: b43. class can be used in the
following manner in order to create a very basic webserver serving files
relative to the current directory:
import http.server
import socketserver
PORT = 8000
Handler = http.server.SimpleHTTPRequestHandler
with socketserver.TCPServer(("", PORT), Handler) as httpd:
print("serving at port", PORT)
httpd.serve_forever()
*note SimpleHTTPRequestHandler: b43. can also be subclassed to enhance
behavior, such as using different index file names by overriding the
class attribute ‘index_pages’.
-- Class: http.server.CGIHTTPRequestHandler (request, client_address,
server)
This class is used to serve either files or output of CGI scripts
from the current directory and below. Note that mapping HTTP
hierarchic structure to local directory structure is exactly as in
*note SimpleHTTPRequestHandler: b43.
Note: CGI scripts run by the *note CGIHTTPRequestHandler: 2a3.
class cannot execute redirects (HTTP code 302), because code
200 (script output follows) is sent prior to execution of the
CGI script. This pre-empts the status code.
The class will however, run the CGI script, instead of serving it
as a file, if it guesses it to be a CGI script. Only
directory-based CGI are used — the other common server
configuration is to treat special extensions as denoting CGI
scripts.
The ‘do_GET()’ and ‘do_HEAD()’ functions are modified to run CGI
scripts and serve the output, instead of serving files, if the
request leads to somewhere below the ‘cgi_directories’ path.
The *note CGIHTTPRequestHandler: 2a3. defines the following data
member:
-- Attribute: cgi_directories
This defaults to ‘['/cgi-bin', '/htbin']’ and describes
directories to treat as containing CGI scripts.
The *note CGIHTTPRequestHandler: 2a3. defines the following method:
-- Method: do_POST ()
This method serves the ‘'POST'’ request type, only allowed for
CGI scripts. Error 501, “Can only POST to CGI scripts”, is
output when trying to POST to a non-CGI url.
Note that CGI scripts will be run with UID of user nobody, for
security reasons. Problems with the CGI script will be translated
to error 403.
Deprecated since version 3.13, will be removed in version 3.15:
*note CGIHTTPRequestHandler: 2a3. is being removed in 3.15. CGI
has not been considered a good way to do things for well over a
decade. This code has been unmaintained for a while now and sees
very little practical use. Retaining it could lead to further
*note security considerations: 3b65.
* Menu:
* Command-line interface: Command-line interface<2>.
* Security considerations: Security considerations<3>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/http/server.py
(2) https://datatracker.ietf.org/doc/html/rfc3986.html
(3) https://datatracker.ietf.org/doc/html/rfc2822.html
(4) https://github.com/python/cpython/tree/3.13/Lib/http/server.py
File: python.info, Node: Command-line interface<2>, Next: Security considerations<3>, Up: http server — HTTP servers
5.22.17.1 Command-line interface
................................
*note http.server: 72. can also be invoked directly using the *note -m:
5dd. switch of the interpreter. The following example illustrates how
to serve files relative to the current directory:
python -m http.server [OPTIONS] [port]
The following options are accepted:
-- Option: port
The server listens to port 8000 by default. The default can be
overridden by passing the desired port number as an argument:
python -m http.server 9000
-- Option: -b, --bind
Specifies a specific address to which it should bind. Both IPv4
and IPv6 addresses are supported. By default, the server binds
itself to all interfaces. For example, the following command
causes the server to bind to localhost only:
python -m http.server --bind 127.0.0.1
Added in version 3.4.
Changed in version 3.8: Support IPv6 in the ‘--bind’ option.
-- Option: -d, --directory
Specifies a directory to which it should serve the files. By
default, the server uses the current directory. For example, the
following command uses a specific directory:
python -m http.server --directory /tmp/
Added in version 3.7.
-- Option: -p, --protocol
Specifies the HTTP version to which the server is conformant. By
default, the server is conformant to HTTP/1.0. For example, the
following command runs an HTTP/1.1 conformant server:
python -m http.server --protocol HTTP/1.1
Added in version 3.11.
-- Option: --cgi
*note CGIHTTPRequestHandler: 2a3. can be enabled in the command
line by passing the ‘--cgi’ option:
python -m http.server --cgi
Deprecated since version 3.13, will be removed in version 3.15:
*note http.server: 72. command line ‘--cgi’ support is being
removed because *note CGIHTTPRequestHandler: 2a3. is being removed.
Warning: *note CGIHTTPRequestHandler: 2a3. and the ‘--cgi’
command-line option are not intended for use by untrusted clients
and may be vulnerable to exploitation. Always use within a secure
environment.
File: python.info, Node: Security considerations<3>, Prev: Command-line interface<2>, Up: http server — HTTP servers
5.22.17.2 Security considerations
.................................
*note SimpleHTTPRequestHandler: b43. will follow symbolic links when
handling requests, this makes it possible for files outside of the
specified directory to be served.
Earlier versions of Python did not scrub control characters from the log
messages emitted to stderr from ‘python -m http.server’ or the default
*note BaseHTTPRequestHandler: 10c1. ‘.log_message’ implementation. This
could allow remote clients connecting to your server to send nefarious
control codes to your terminal.
Changed in version 3.12: Control characters are scrubbed in stderr logs.
File: python.info, Node: http cookies — HTTP state management, Next: http cookiejar — Cookie handling for HTTP clients, Prev: http server — HTTP servers, Up: Internet Protocols and Support
5.22.18 ‘http.cookies’ — HTTP state management
----------------------------------------------
'Source code:' Lib/http/cookies.py(1)
__________________________________________________________________
The *note http.cookies: 71. module defines classes for abstracting the
concept of cookies, an HTTP state management mechanism. It supports
both simple string-only cookies, and provides an abstraction for having
any serializable data-type as cookie value.
The module formerly strictly applied the parsing rules described in the
RFC 2109(2) and RFC 2068(3) specifications. It has since been
discovered that MSIE 3.0x didn’t follow the character rules outlined in
those specs; many current-day browsers and servers have also relaxed
parsing rules when it comes to cookie handling. As a result, this
module now uses parsing rules that are a bit less strict than they once
were.
The character set, *note string.ascii_letters: 12b6, *note
string.digits: 22c2. and ‘!#$%&'*+-.^_`|~:’ denote the set of valid
characters allowed by this module in a cookie name (as *note key: c01.).
Changed in version 3.3: Allowed ‘:’ as a valid cookie name character.
Note: On encountering an invalid cookie, *note CookieError: 3b94.
is raised, so if your cookie data comes from a browser you should
always prepare for invalid data and catch *note CookieError: 3b94.
on parsing.
-- Exception: http.cookies.CookieError
Exception failing because of RFC 2109(4) invalidity: incorrect
attributes, incorrect ‘Set-Cookie’ header, etc.
-- Class: http.cookies.BaseCookie ([input])
This class is a dictionary-like object whose keys are strings and
whose values are *note Morsel: c04. instances. Note that upon
setting a key to a value, the value is first converted to a *note
Morsel: c04. containing the key and the value.
If 'input' is given, it is passed to the *note load(): 3b96.
method.
-- Class: http.cookies.SimpleCookie ([input])
This class derives from *note BaseCookie: 3b95. and overrides *note
value_decode(): 3b97. and *note value_encode(): 3b98.
‘SimpleCookie’ supports strings as cookie values. When setting the
value, ‘SimpleCookie’ calls the builtin *note str(): 447. to
convert the value to a string. Values received from HTTP are kept
as strings.
See also
........
Module *note http.cookiejar: 70.
HTTP cookie handling for web 'clients'. The *note http.cookiejar:
70. and *note http.cookies: 71. modules do not depend on each
other.
RFC 2109(5) - HTTP State Management Mechanism
This is the state management specification implemented by this
module.
* Menu:
* Cookie Objects::
* Morsel Objects::
* Example: Example<12>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/http/cookies.py
(2) https://datatracker.ietf.org/doc/html/rfc2109.html
(3) https://datatracker.ietf.org/doc/html/rfc2068.html
(4) https://datatracker.ietf.org/doc/html/rfc2109.html
(5) https://datatracker.ietf.org/doc/html/rfc2109.html
File: python.info, Node: Cookie Objects, Next: Morsel Objects, Up: http cookies — HTTP state management
5.22.18.1 Cookie Objects
........................
-- Method: BaseCookie.value_decode (val)
Return a tuple ‘(real_value, coded_value)’ from a string
representation. ‘real_value’ can be any type. This method does no
decoding in *note BaseCookie: 3b95. — it exists so it can be
overridden.
-- Method: BaseCookie.value_encode (val)
Return a tuple ‘(real_value, coded_value)’. 'val' can be any type,
but ‘coded_value’ will always be converted to a string. This
method does no encoding in *note BaseCookie: 3b95. — it exists so
it can be overridden.
In general, it should be the case that *note value_encode(): 3b98.
and *note value_decode(): 3b97. are inverses on the range of
'value_decode'.
-- Method: BaseCookie.output (attrs=None, header='Set-Cookie:',
sep='\r\n')
Return a string representation suitable to be sent as HTTP headers.
'attrs' and 'header' are sent to each *note Morsel: c04.’s *note
output(): 3b9c. method. 'sep' is used to join the headers
together, and is by default the combination ‘'\r\n'’ (CRLF).
-- Method: BaseCookie.js_output (attrs=None)
Return an embeddable JavaScript snippet, which, if run on a browser
which supports JavaScript, will act the same as if the HTTP headers
was sent.
The meaning for 'attrs' is the same as in *note output(): 3b9b.
-- Method: BaseCookie.load (rawdata)
If 'rawdata' is a string, parse it as an ‘HTTP_COOKIE’ and add the
values found there as *note Morsel: c04.s. If it is a dictionary,
it is equivalent to:
for k, v in rawdata.items():
cookie[k] = v
File: python.info, Node: Morsel Objects, Next: Example<12>, Prev: Cookie Objects, Up: http cookies — HTTP state management
5.22.18.2 Morsel Objects
........................
-- Class: http.cookies.Morsel
Abstract a key/value pair, which has some RFC 2109(1) attributes.
Morsels are dictionary-like objects, whose set of keys is constant
— the valid RFC 2109(2) attributes, which are:
-- Attribute: expires
-- Attribute: path
-- Attribute: comment
-- Attribute: domain
-- Attribute: max-age
-- Attribute: secure
-- Attribute: version
-- Attribute: httponly
-- Attribute: samesite
The attribute *note httponly: 139c. specifies that the cookie is
only transferred in HTTP requests, and is not accessible through
JavaScript. This is intended to mitigate some forms of cross-site
scripting.
The attribute *note samesite: 3ba6. controls when the browser sends
the cookie with cross-site requests. This helps to mitigate CSRF
attacks. Valid values are “Strict” (only sent with same-site
requests), “Lax” (sent with same-site requests and top-level
navigations), and “None” (sent with same-site and cross-site
requests). When using “None”, the “secure” attribute must also be
set, as required by modern browsers.
The keys are case-insensitive and their default value is ‘''’.
Changed in version 3.5: ‘__eq__()’ now takes *note key: c01. and
*note value: c02. into account.
Changed in version 3.7: Attributes *note key: c01, *note value:
c02. and *note coded_value: c03. are read-only. Use *note set():
c05. for setting them.
Changed in version 3.8: Added support for the *note samesite: 3ba6.
attribute.
-- Attribute: Morsel.value
The value of the cookie.
-- Attribute: Morsel.coded_value
The encoded value of the cookie — this is what should be sent.
-- Attribute: Morsel.key
The name of the cookie.
-- Method: Morsel.set (key, value, coded_value)
Set the 'key', 'value' and 'coded_value' attributes.
-- Method: Morsel.isReservedKey (K)
Whether 'K' is a member of the set of keys of a *note Morsel: c04.
-- Method: Morsel.output (attrs=None, header='Set-Cookie:')
Return a string representation of the Morsel, suitable to be sent
as an HTTP header. By default, all the attributes are included,
unless 'attrs' is given, in which case it should be a list of
attributes to use. 'header' is by default ‘"Set-Cookie:"’.
-- Method: Morsel.js_output (attrs=None)
Return an embeddable JavaScript snippet, which, if run on a browser
which supports JavaScript, will act the same as if the HTTP header
was sent.
The meaning for 'attrs' is the same as in *note output(): 3b9c.
-- Method: Morsel.OutputString (attrs=None)
Return a string representing the Morsel, without any surrounding
HTTP or JavaScript.
The meaning for 'attrs' is the same as in *note output(): 3b9c.
-- Method: Morsel.update (values)
Update the values in the Morsel dictionary with the values in the
dictionary 'values'. Raise an error if any of the keys in the
'values' dict is not a valid RFC 2109(3) attribute.
Changed in version 3.5: an error is raised for invalid keys.
-- Method: Morsel.copy (value)
Return a shallow copy of the Morsel object.
Changed in version 3.5: return a Morsel object instead of a dict.
-- Method: Morsel.setdefault (key, value=None)
Raise an error if key is not a valid RFC 2109(4) attribute,
otherwise behave the same as *note dict.setdefault(): 1072.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2109.html
(2) https://datatracker.ietf.org/doc/html/rfc2109.html
(3) https://datatracker.ietf.org/doc/html/rfc2109.html
(4) https://datatracker.ietf.org/doc/html/rfc2109.html
File: python.info, Node: Example<12>, Prev: Morsel Objects, Up: http cookies — HTTP state management
5.22.18.3 Example
.................
The following example demonstrates how to use the *note http.cookies:
71. module.
>>> from http import cookies
>>> C = cookies.SimpleCookie()
>>> C["fig"] = "newton"
>>> C["sugar"] = "wafer"
>>> print(C) # generate HTTP headers
Set-Cookie: fig=newton
Set-Cookie: sugar=wafer
>>> print(C.output()) # same thing
Set-Cookie: fig=newton
Set-Cookie: sugar=wafer
>>> C = cookies.SimpleCookie()
>>> C["rocky"] = "road"
>>> C["rocky"]["path"] = "/cookie"
>>> print(C.output(header="Cookie:"))
Cookie: rocky=road; Path=/cookie
>>> print(C.output(attrs=[], header="Cookie:"))
Cookie: rocky=road
>>> C = cookies.SimpleCookie()
>>> C.load("chips=ahoy; vienna=finger") # load from a string (HTTP header)
>>> print(C)
Set-Cookie: chips=ahoy
Set-Cookie: vienna=finger
>>> C = cookies.SimpleCookie()
>>> C.load('keebler="E=everybody; L=\\"Loves\\"; fudge=\\012;";')
>>> print(C)
Set-Cookie: keebler="E=everybody; L=\"Loves\"; fudge=\012;"
>>> C = cookies.SimpleCookie()
>>> C["oreo"] = "doublestuff"
>>> C["oreo"]["path"] = "/"
>>> print(C)
Set-Cookie: oreo=doublestuff; Path=/
>>> C = cookies.SimpleCookie()
>>> C["twix"] = "none for you"
>>> C["twix"].value
'none for you'
>>> C = cookies.SimpleCookie()
>>> C["number"] = 7 # equivalent to C["number"] = str(7)
>>> C["string"] = "seven"
>>> C["number"].value
'7'
>>> C["string"].value
'seven'
>>> print(C)
Set-Cookie: number=7
Set-Cookie: string=seven
File: python.info, Node: http cookiejar — Cookie handling for HTTP clients, Next: xmlrpc — XMLRPC server and client modules, Prev: http cookies — HTTP state management, Up: Internet Protocols and Support
5.22.19 ‘http.cookiejar’ — Cookie handling for HTTP clients
-----------------------------------------------------------
'Source code:' Lib/http/cookiejar.py(1)
__________________________________________________________________
The *note http.cookiejar: 70. module defines classes for automatic
handling of HTTP cookies. It is useful for accessing web sites that
require small pieces of data – 'cookies' – to be set on the client
machine by an HTTP response from a web server, and then returned to the
server in later HTTP requests.
Both the regular Netscape cookie protocol and the protocol defined by
RFC 2965(2) are handled. RFC 2965 handling is switched off by default.
RFC 2109(3) cookies are parsed as Netscape cookies and subsequently
treated either as Netscape or RFC 2965 cookies according to the ‘policy’
in effect. Note that the great majority of cookies on the internet are
Netscape cookies. *note http.cookiejar: 70. attempts to follow the
de-facto Netscape cookie protocol (which differs substantially from that
set out in the original Netscape specification), including taking note
of the ‘max-age’ and ‘port’ cookie-attributes introduced with RFC 2965.
Note: The various named parameters found in ‘Set-Cookie’ and
‘Set-Cookie2’ headers (eg. ‘domain’ and ‘expires’) are
conventionally referred to as 'attributes'. To distinguish them
from Python attributes, the documentation for this module uses the
term 'cookie-attribute' instead.
The module defines the following exception:
-- Exception: http.cookiejar.LoadError
Instances of *note FileCookieJar: 3bb0. raise this exception on
failure to load cookies from a file. *note LoadError: 3baf. is a
subclass of *note OSError: 413.
Changed in version 3.3: *note LoadError: 3baf. used to be a subtype
of *note IOError: 104b, which is now an alias of *note OSError:
413.
The following classes are provided:
-- Class: http.cookiejar.CookieJar (policy=None)
'policy' is an object implementing the *note CookiePolicy: 3bb1.
interface.
The *note CookieJar: 39da. class stores HTTP cookies. It extracts
cookies from HTTP requests, and returns them in HTTP responses.
*note CookieJar: 39da. instances automatically expire contained
cookies when necessary. Subclasses are also responsible for
storing and retrieving cookies from a file or database.
-- Class: http.cookiejar.FileCookieJar (filename=None, delayload=None,
policy=None)
'policy' is an object implementing the *note CookiePolicy: 3bb1.
interface. For the other arguments, see the documentation for the
corresponding attributes.
A *note CookieJar: 39da. which can load cookies from, and perhaps
save cookies to, a file on disk. Cookies are 'NOT' loaded from the
named file until either the *note load(): 3bb2. or *note revert():
3bb3. method is called. Subclasses of this class are documented in
section *note FileCookieJar subclasses and co-operation with web
browsers: 3bb4.
This should not be initialized directly – use its subclasses below
instead.
Changed in version 3.8: The filename parameter supports a *note
path-like object: 2a2.
-- Class: http.cookiejar.CookiePolicy
This class is responsible for deciding whether each cookie should
be accepted from / returned to the server.
-- Class: http.cookiejar.DefaultCookiePolicy (blocked_domains=None,
allowed_domains=None, netscape=True, rfc2965=False,
rfc2109_as_netscape=None, hide_cookie2=False,
strict_domain=False, strict_rfc2965_unverifiable=True,
strict_ns_unverifiable=False,
strict_ns_domain=DefaultCookiePolicy.DomainLiberal,
strict_ns_set_initial_dollar=False, strict_ns_set_path=False,
secure_protocols=('https', 'wss'))
Constructor arguments should be passed as keyword arguments only.
'blocked_domains' is a sequence of domain names that we never
accept cookies from, nor return cookies to. 'allowed_domains' if
not *note None: 671, this is a sequence of the only domains for
which we accept and return cookies. 'secure_protocols' is a
sequence of protocols for which secure cookies can be added to. By
default 'https' and 'wss' (secure websocket) are considered secure
protocols. For all other arguments, see the documentation for
*note CookiePolicy: 3bb1. and *note DefaultCookiePolicy: 1a10.
objects.
*note DefaultCookiePolicy: 1a10. implements the standard accept /
reject rules for Netscape and RFC 2965(4) cookies. By default, RFC
2109(5) cookies (ie. cookies received in a ‘Set-Cookie’ header
with a version cookie-attribute of 1) are treated according to the
RFC 2965 rules. However, if RFC 2965 handling is turned off or
*note rfc2109_as_netscape: 3bb5. is ‘True’, RFC 2109 cookies are
‘downgraded’ by the *note CookieJar: 39da. instance to Netscape
cookies, by setting the ‘version’ attribute of the *note Cookie:
3bb6. instance to 0. *note DefaultCookiePolicy: 1a10. also
provides some parameters to allow some fine-tuning of policy.
-- Class: http.cookiejar.Cookie
This class represents Netscape, RFC 2109(6) and RFC 2965(7)
cookies. It is not expected that users of *note http.cookiejar:
70. construct their own *note Cookie: 3bb6. instances. Instead, if
necessary, call ‘make_cookies()’ on a *note CookieJar: 39da.
instance.
See also
........
Module *note urllib.request: 10b.
URL opening with automatic cookie handling.
Module *note http.cookies: 71.
HTTP cookie classes, principally useful for server-side code. The
*note http.cookiejar: 70. and *note http.cookies: 71. modules do
not depend on each other.
‘https://curl.se/rfc/cookie_spec.html’
The specification of the original Netscape cookie protocol. Though
this is still the dominant protocol, the ‘Netscape cookie protocol’
implemented by all the major browsers (and *note http.cookiejar:
70.) only bears a passing resemblance to the one sketched out in
‘cookie_spec.html’.
RFC 2109(8) - HTTP State Management Mechanism
Obsoleted by RFC 2965(9). Uses ‘Set-Cookie’ with version=1.
RFC 2965(10) - HTTP State Management Mechanism
The Netscape protocol with the bugs fixed. Uses ‘Set-Cookie2’ in
place of ‘Set-Cookie’. Not widely used.
‘https://kristol.org/cookie/errata.html’
Unfinished errata to RFC 2965(11).
RFC 2964(12) - Use of HTTP State Management
* Menu:
* CookieJar and FileCookieJar Objects::
* FileCookieJar subclasses and co-operation with web browsers::
* CookiePolicy Objects::
* DefaultCookiePolicy Objects::
* Cookie Objects: Cookie Objects<2>.
* Examples: Examples<29>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/http/cookiejar.py
(2) https://datatracker.ietf.org/doc/html/rfc2965.html
(3) https://datatracker.ietf.org/doc/html/rfc2109.html
(4) https://datatracker.ietf.org/doc/html/rfc2965.html
(5) https://datatracker.ietf.org/doc/html/rfc2109.html
(6) https://datatracker.ietf.org/doc/html/rfc2109.html
(7) https://datatracker.ietf.org/doc/html/rfc2965.html
(8) https://datatracker.ietf.org/doc/html/rfc2109.html
(9) https://datatracker.ietf.org/doc/html/rfc2965.html
(10) https://datatracker.ietf.org/doc/html/rfc2965.html
(11) https://datatracker.ietf.org/doc/html/rfc2965.html
(12) https://datatracker.ietf.org/doc/html/rfc2964.html
File: python.info, Node: CookieJar and FileCookieJar Objects, Next: FileCookieJar subclasses and co-operation with web browsers, Up: http cookiejar — Cookie handling for HTTP clients
5.22.19.1 CookieJar and FileCookieJar Objects
.............................................
*note CookieJar: 39da. objects support the *note iterator: 1ac. protocol
for iterating over contained *note Cookie: 3bb6. objects.
*note CookieJar: 39da. has the following methods:
-- Method: CookieJar.add_cookie_header (request)
Add correct ‘Cookie’ header to 'request'.
If policy allows (ie. the ‘rfc2965’ and ‘hide_cookie2’ attributes
of the *note CookieJar: 39da.’s *note CookiePolicy: 3bb1. instance
are true and false respectively), the ‘Cookie2’ header is also
added when appropriate.
The 'request' object (usually a *note urllib.request.Request: fd3.
instance) must support the methods ‘get_full_url()’,
‘has_header()’, ‘get_header()’, ‘header_items()’,
‘add_unredirected_header()’ and the attributes ‘host’, ‘type’,
‘unverifiable’ and ‘origin_req_host’ as documented by *note
urllib.request: 10b.
Changed in version 3.3: 'request' object needs ‘origin_req_host’
attribute. Dependency on a deprecated method
‘get_origin_req_host()’ has been removed.
-- Method: CookieJar.extract_cookies (response, request)
Extract cookies from HTTP 'response' and store them in the *note
CookieJar: 39da, where allowed by policy.
The *note CookieJar: 39da. will look for allowable ‘Set-Cookie’ and
‘Set-Cookie2’ headers in the 'response' argument, and store cookies
as appropriate (subject to the *note CookiePolicy.set_ok(): 3bbb.
method’s approval).
The 'response' object (usually the result of a call to *note
urllib.request.urlopen(): 295, or similar) should support an
‘info()’ method, which returns an *note email.message.Message: 27e.
instance.
The 'request' object (usually a *note urllib.request.Request: fd3.
instance) must support the method ‘get_full_url()’ and the
attributes ‘host’, ‘unverifiable’ and ‘origin_req_host’, as
documented by *note urllib.request: 10b. The request is used to
set default values for cookie-attributes as well as for checking
that the cookie is allowed to be set.
Changed in version 3.3: 'request' object needs ‘origin_req_host’
attribute. Dependency on a deprecated method
‘get_origin_req_host()’ has been removed.
-- Method: CookieJar.set_policy (policy)
Set the *note CookiePolicy: 3bb1. instance to be used.
-- Method: CookieJar.make_cookies (response, request)
Return sequence of *note Cookie: 3bb6. objects extracted from
'response' object.
See the documentation for *note extract_cookies(): 3bba. for the
interfaces required of the 'response' and 'request' arguments.
-- Method: CookieJar.set_cookie_if_ok (cookie, request)
Set a *note Cookie: 3bb6. if policy says it’s OK to do so.
-- Method: CookieJar.set_cookie (cookie)
Set a *note Cookie: 3bb6, without checking with policy to see
whether or not it should be set.
-- Method: CookieJar.clear ([domain[, path[, name]]])
Clear some cookies.
If invoked without arguments, clear all cookies. If given a single
argument, only cookies belonging to that 'domain' will be removed.
If given two arguments, cookies belonging to the specified 'domain'
and URL 'path' are removed. If given three arguments, then the
cookie with the specified 'domain', 'path' and 'name' is removed.
Raises *note KeyError: 33f. if no matching cookie exists.
-- Method: CookieJar.clear_session_cookies ()
Discard all session cookies.
Discards all contained cookies that have a true ‘discard’ attribute
(usually because they had either no ‘max-age’ or ‘expires’
cookie-attribute, or an explicit ‘discard’ cookie-attribute). For
interactive browsers, the end of a session usually corresponds to
closing the browser window.
Note that the ‘save()’ method won’t save session cookies anyway,
unless you ask otherwise by passing a true 'ignore_discard'
argument.
*note FileCookieJar: 3bb0. implements the following additional methods:
-- Method: FileCookieJar.save (filename=None, ignore_discard=False,
ignore_expires=False)
Save cookies to a file.
This base class raises *note NotImplementedError: 22a. Subclasses
may leave this method unimplemented.
'filename' is the name of file in which to save cookies. If
'filename' is not specified, ‘self.filename’ is used (whose default
is the value passed to the constructor, if any); if ‘self.filename’
is *note None: 671, *note ValueError: 204. is raised.
'ignore_discard': save even cookies set to be discarded.
'ignore_expires': save even cookies that have expired
The file is overwritten if it already exists, thus wiping all the
cookies it contains. Saved cookies can be restored later using the
*note load(): 3bb2. or *note revert(): 3bb3. methods.
-- Method: FileCookieJar.load (filename=None, ignore_discard=False,
ignore_expires=False)
Load cookies from a file.
Old cookies are kept unless overwritten by newly loaded ones.
Arguments are as for *note save(): 3bc2.
The named file must be in the format understood by the class, or
*note LoadError: 3baf. will be raised. Also, *note OSError: 413.
may be raised, for example if the file does not exist.
Changed in version 3.3: *note IOError: 104b. used to be raised, it
is now an alias of *note OSError: 413.
-- Method: FileCookieJar.revert (filename=None, ignore_discard=False,
ignore_expires=False)
Clear all cookies and reload cookies from a saved file.
*note revert(): 3bb3. can raise the same exceptions as *note
load(): 3bb2. If there is a failure, the object’s state will not
be altered.
*note FileCookieJar: 3bb0. instances have the following public
attributes:
-- Attribute: FileCookieJar.filename
Filename of default file in which to keep cookies. This attribute
may be assigned to.
-- Attribute: FileCookieJar.delayload
If true, load cookies lazily from disk. This attribute should not
be assigned to. This is only a hint, since this only affects
performance, not behaviour (unless the cookies on disk are
changing). A *note CookieJar: 39da. object may ignore it. None of
the *note FileCookieJar: 3bb0. classes included in the standard
library lazily loads cookies.
File: python.info, Node: FileCookieJar subclasses and co-operation with web browsers, Next: CookiePolicy Objects, Prev: CookieJar and FileCookieJar Objects, Up: http cookiejar — Cookie handling for HTTP clients
5.22.19.2 FileCookieJar subclasses and co-operation with web browsers
.....................................................................
The following *note CookieJar: 39da. subclasses are provided for reading
and writing.
-- Class: http.cookiejar.MozillaCookieJar (filename=None,
delayload=None, policy=None)
A *note FileCookieJar: 3bb0. that can load from and save cookies to
disk in the Mozilla ‘cookies.txt’ file format (which is also used
by curl and the Lynx and Netscape browsers).
Note: This loses information about RFC 2965(1) cookies, and
also about newer or non-standard cookie-attributes such as
‘port’.
Warning: Back up your cookies before saving if you have
cookies whose loss / corruption would be inconvenient (there
are some subtleties which may lead to slight changes in the
file over a load / save round-trip).
Also note that cookies saved while Mozilla is running will get
clobbered by Mozilla.
-- Class: http.cookiejar.LWPCookieJar (filename=None, delayload=None,
policy=None)
A *note FileCookieJar: 3bb0. that can load from and save cookies to
disk in format compatible with the libwww-perl library’s
‘Set-Cookie3’ file format. This is convenient if you want to store
cookies in a human-readable file.
Changed in version 3.8: The filename parameter supports a *note
path-like object: 2a2.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2965.html
File: python.info, Node: CookiePolicy Objects, Next: DefaultCookiePolicy Objects, Prev: FileCookieJar subclasses and co-operation with web browsers, Up: http cookiejar — Cookie handling for HTTP clients
5.22.19.3 CookiePolicy Objects
..............................
Objects implementing the *note CookiePolicy: 3bb1. interface have the
following methods:
-- Method: CookiePolicy.set_ok (cookie, request)
Return boolean value indicating whether cookie should be accepted
from server.
'cookie' is a *note Cookie: 3bb6. instance. 'request' is an object
implementing the interface defined by the documentation for *note
CookieJar.extract_cookies(): 3bba.
-- Method: CookiePolicy.return_ok (cookie, request)
Return boolean value indicating whether cookie should be returned
to server.
'cookie' is a *note Cookie: 3bb6. instance. 'request' is an object
implementing the interface defined by the documentation for *note
CookieJar.add_cookie_header(): 3bb9.
-- Method: CookiePolicy.domain_return_ok (domain, request)
Return ‘False’ if cookies should not be returned, given cookie
domain.
This method is an optimization. It removes the need for checking
every cookie with a particular domain (which might involve reading
many files). Returning true from *note domain_return_ok(): 3bcb.
and *note path_return_ok(): 3bcc. leaves all the work to *note
return_ok(): 3bca.
If *note domain_return_ok(): 3bcb. returns true for the cookie
domain, *note path_return_ok(): 3bcc. is called for the cookie
path. Otherwise, *note path_return_ok(): 3bcc. and *note
return_ok(): 3bca. are never called for that cookie domain. If
*note path_return_ok(): 3bcc. returns true, *note return_ok():
3bca. is called with the *note Cookie: 3bb6. object itself for a
full check. Otherwise, *note return_ok(): 3bca. is never called
for that cookie path.
Note that *note domain_return_ok(): 3bcb. is called for every
'cookie' domain, not just for the 'request' domain. For example,
the function might be called with both ‘".example.com"’ and
‘"www.example.com"’ if the request domain is ‘"www.example.com"’.
The same goes for *note path_return_ok(): 3bcc.
The 'request' argument is as documented for *note return_ok():
3bca.
-- Method: CookiePolicy.path_return_ok (path, request)
Return ‘False’ if cookies should not be returned, given cookie
path.
See the documentation for *note domain_return_ok(): 3bcb.
In addition to implementing the methods above, implementations of the
*note CookiePolicy: 3bb1. interface must also supply the following
attributes, indicating which protocols should be used, and how. All of
these attributes may be assigned to.
-- Attribute: CookiePolicy.netscape
Implement Netscape protocol.
-- Attribute: CookiePolicy.rfc2965
Implement RFC 2965(1) protocol.
-- Attribute: CookiePolicy.hide_cookie2
Don’t add ‘Cookie2’ header to requests (the presence of this header
indicates to the server that we understand RFC 2965(2) cookies).
The most useful way to define a *note CookiePolicy: 3bb1. class is by
subclassing from *note DefaultCookiePolicy: 1a10. and overriding some or
all of the methods above. *note CookiePolicy: 3bb1. itself may be used
as a ‘null policy’ to allow setting and receiving any and all cookies
(this is unlikely to be useful).
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2965.html
(2) https://datatracker.ietf.org/doc/html/rfc2965.html
File: python.info, Node: DefaultCookiePolicy Objects, Next: Cookie Objects<2>, Prev: CookiePolicy Objects, Up: http cookiejar — Cookie handling for HTTP clients
5.22.19.4 DefaultCookiePolicy Objects
.....................................
Implements the standard rules for accepting and returning cookies.
Both RFC 2965(1) and Netscape cookies are covered. RFC 2965 handling is
switched off by default.
The easiest way to provide your own policy is to override this class and
call its methods in your overridden implementations before adding your
own additional checks:
import http.cookiejar
class MyCookiePolicy(http.cookiejar.DefaultCookiePolicy):
def set_ok(self, cookie, request):
if not http.cookiejar.DefaultCookiePolicy.set_ok(self, cookie, request):
return False
if i_dont_want_to_store_this_cookie(cookie):
return False
return True
In addition to the features required to implement the *note
CookiePolicy: 3bb1. interface, this class allows you to block and allow
domains from setting and receiving cookies. There are also some
strictness switches that allow you to tighten up the rather loose
Netscape protocol rules a little bit (at the cost of blocking some
benign cookies).
A domain blocklist and allowlist is provided (both off by default).
Only domains not in the blocklist and present in the allowlist (if the
allowlist is active) participate in cookie setting and returning. Use
the 'blocked_domains' constructor argument, and ‘blocked_domains()’ and
‘set_blocked_domains()’ methods (and the corresponding argument and
methods for 'allowed_domains'). If you set an allowlist, you can turn
it off again by setting it to *note None: 671.
Domains in block or allow lists that do not start with a dot must equal
the cookie domain to be matched. For example, ‘"example.com"’ matches a
blocklist entry of ‘"example.com"’, but ‘"www.example.com"’ does not.
Domains that do start with a dot are matched by more specific domains
too. For example, both ‘"www.example.com"’ and
‘"www.coyote.example.com"’ match ‘".example.com"’ (but ‘"example.com"’
itself does not). IP addresses are an exception, and must match
exactly. For example, if blocked_domains contains ‘"192.168.1.2"’ and
‘".168.1.2"’, 192.168.1.2 is blocked, but 193.168.1.2 is not.
*note DefaultCookiePolicy: 1a10. implements the following additional
methods:
-- Method: DefaultCookiePolicy.blocked_domains ()
Return the sequence of blocked domains (as a tuple).
-- Method: DefaultCookiePolicy.set_blocked_domains (blocked_domains)
Set the sequence of blocked domains.
-- Method: DefaultCookiePolicy.is_blocked (domain)
Return ‘True’ if 'domain' is on the blocklist for setting or
receiving cookies.
-- Method: DefaultCookiePolicy.allowed_domains ()
Return *note None: 671, or the sequence of allowed domains (as a
tuple).
-- Method: DefaultCookiePolicy.set_allowed_domains (allowed_domains)
Set the sequence of allowed domains, or *note None: 671.
-- Method: DefaultCookiePolicy.is_not_allowed (domain)
Return ‘True’ if 'domain' is not on the allowlist for setting or
receiving cookies.
*note DefaultCookiePolicy: 1a10. instances have the following
attributes, which are all initialised from the constructor arguments of
the same name, and which may all be assigned to.
-- Attribute: DefaultCookiePolicy.rfc2109_as_netscape
If true, request that the *note CookieJar: 39da. instance downgrade
RFC 2109(2) cookies (ie. cookies received in a ‘Set-Cookie’ header
with a version cookie-attribute of 1) to Netscape cookies by
setting the version attribute of the *note Cookie: 3bb6. instance
to 0. The default value is *note None: 671, in which case RFC 2109
cookies are downgraded if and only if RFC 2965(3) handling is
turned off. Therefore, RFC 2109 cookies are downgraded by default.
General strictness switches:
-- Attribute: DefaultCookiePolicy.strict_domain
Don’t allow sites to set two-component domains with country-code
top-level domains like ‘.co.uk’, ‘.gov.uk’, ‘.co.nz’.etc. This is
far from perfect and isn’t guaranteed to work!
RFC 2965(4) protocol strictness switches:
-- Attribute: DefaultCookiePolicy.strict_rfc2965_unverifiable
Follow RFC 2965(5) rules on unverifiable transactions (usually, an
unverifiable transaction is one resulting from a redirect or a
request for an image hosted on another site). If this is false,
cookies are 'never' blocked on the basis of verifiability
Netscape protocol strictness switches:
-- Attribute: DefaultCookiePolicy.strict_ns_unverifiable
Apply RFC 2965(6) rules on unverifiable transactions even to
Netscape cookies.
-- Attribute: DefaultCookiePolicy.strict_ns_domain
Flags indicating how strict to be with domain-matching rules for
Netscape cookies. See below for acceptable values.
-- Attribute: DefaultCookiePolicy.strict_ns_set_initial_dollar
Ignore cookies in Set-Cookie: headers that have names starting with
‘'$'’.
-- Attribute: DefaultCookiePolicy.strict_ns_set_path
Don’t allow setting cookies whose path doesn’t path-match request
URI.
*note strict_ns_domain: 3bdb. is a collection of flags. Its value is
constructed by or-ing together (for example,
‘DomainStrictNoDots|DomainStrictNonDomain’ means both flags are set).
-- Attribute: DefaultCookiePolicy.DomainStrictNoDots
When setting cookies, the ‘host prefix’ must not contain a dot (eg.
‘www.foo.bar.com’ can’t set a cookie for ‘.bar.com’, because
‘www.foo’ contains a dot).
-- Attribute: DefaultCookiePolicy.DomainStrictNonDomain
Cookies that did not explicitly specify a ‘domain’ cookie-attribute
can only be returned to a domain equal to the domain that set the
cookie (eg. ‘spam.example.com’ won’t be returned cookies from
‘example.com’ that had no ‘domain’ cookie-attribute).
-- Attribute: DefaultCookiePolicy.DomainRFC2965Match
When setting cookies, require a full RFC 2965(7) domain-match.
The following attributes are provided for convenience, and are the most
useful combinations of the above flags:
-- Attribute: DefaultCookiePolicy.DomainLiberal
Equivalent to 0 (ie. all of the above Netscape domain strictness
flags switched off).
-- Attribute: DefaultCookiePolicy.DomainStrict
Equivalent to ‘DomainStrictNoDots|DomainStrictNonDomain’.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2965.html
(2) https://datatracker.ietf.org/doc/html/rfc2109.html
(3) https://datatracker.ietf.org/doc/html/rfc2965.html
(4) https://datatracker.ietf.org/doc/html/rfc2965.html
(5) https://datatracker.ietf.org/doc/html/rfc2965.html
(6) https://datatracker.ietf.org/doc/html/rfc2965.html
(7) https://datatracker.ietf.org/doc/html/rfc2965.html
File: python.info, Node: Cookie Objects<2>, Next: Examples<29>, Prev: DefaultCookiePolicy Objects, Up: http cookiejar — Cookie handling for HTTP clients
5.22.19.5 Cookie Objects
........................
*note Cookie: 3bb6. instances have Python attributes roughly
corresponding to the standard cookie-attributes specified in the various
cookie standards. The correspondence is not one-to-one, because there
are complicated rules for assigning default values, because the
‘max-age’ and ‘expires’ cookie-attributes contain equivalent
information, and because RFC 2109(1) cookies may be ‘downgraded’ by
*note http.cookiejar: 70. from version 1 to version 0 (Netscape)
cookies.
Assignment to these attributes should not be necessary other than in
rare circumstances in a *note CookiePolicy: 3bb1. method. The class
does not enforce internal consistency, so you should know what you’re
doing if you do that.
-- Attribute: Cookie.version
Integer or *note None: 671. Netscape cookies have *note version:
3be4. 0. RFC 2965(2) and RFC 2109(3) cookies have a ‘version’
cookie-attribute of 1. However, note that *note http.cookiejar:
70. may ‘downgrade’ RFC 2109 cookies to Netscape cookies, in which
case *note version: 3be4. is 0.
-- Attribute: Cookie.name
Cookie name (a string).
-- Attribute: Cookie.value
Cookie value (a string), or *note None: 671.
-- Attribute: Cookie.port
String representing a port or a set of ports (eg. ‘80’, or
‘80,8080’), or *note None: 671.
-- Attribute: Cookie.domain
Cookie domain (a string).
-- Attribute: Cookie.path
Cookie path (a string, eg. ‘'/acme/rocket_launchers'’).
-- Attribute: Cookie.secure
‘True’ if cookie should only be returned over a secure connection.
-- Attribute: Cookie.expires
Integer expiry date in seconds since epoch, or *note None: 671.
See also the *note is_expired(): 3bec. method.
-- Attribute: Cookie.discard
‘True’ if this is a session cookie.
-- Attribute: Cookie.comment
String comment from the server explaining the function of this
cookie, or *note None: 671.
-- Attribute: Cookie.comment_url
URL linking to a comment from the server explaining the function of
this cookie, or *note None: 671.
-- Attribute: Cookie.rfc2109
‘True’ if this cookie was received as an RFC 2109(4) cookie (ie.
the cookie arrived in a ‘Set-Cookie’ header, and the value of the
Version cookie-attribute in that header was 1). This attribute is
provided because *note http.cookiejar: 70. may ‘downgrade’ RFC 2109
cookies to Netscape cookies, in which case *note version: 3be4. is
0.
-- Attribute: Cookie.port_specified
‘True’ if a port or set of ports was explicitly specified by the
server (in the ‘Set-Cookie’ / ‘Set-Cookie2’ header).
-- Attribute: Cookie.domain_specified
‘True’ if a domain was explicitly specified by the server.
-- Attribute: Cookie.domain_initial_dot
‘True’ if the domain explicitly specified by the server began with
a dot (‘'.'’).
Cookies may have additional non-standard cookie-attributes. These may
be accessed using the following methods:
-- Method: Cookie.has_nonstandard_attr (name)
Return ‘True’ if cookie has the named cookie-attribute.
-- Method: Cookie.get_nonstandard_attr (name, default=None)
If cookie has the named cookie-attribute, return its value.
Otherwise, return 'default'.
-- Method: Cookie.set_nonstandard_attr (name, value)
Set the value of the named cookie-attribute.
The *note Cookie: 3bb6. class also defines the following method:
-- Method: Cookie.is_expired (now=None)
‘True’ if cookie has passed the time at which the server requested
it should expire. If 'now' is given (in seconds since the epoch),
return whether the cookie has expired at the specified time.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2109.html
(2) https://datatracker.ietf.org/doc/html/rfc2965.html
(3) https://datatracker.ietf.org/doc/html/rfc2109.html
(4) https://datatracker.ietf.org/doc/html/rfc2109.html
File: python.info, Node: Examples<29>, Prev: Cookie Objects<2>, Up: http cookiejar — Cookie handling for HTTP clients
5.22.19.6 Examples
..................
The first example shows the most common usage of *note http.cookiejar:
70.:
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
This example illustrates how to open a URL using your Netscape, Mozilla,
or Lynx cookies (assumes Unix/Netscape convention for location of the
cookies file):
import os, http.cookiejar, urllib.request
cj = http.cookiejar.MozillaCookieJar()
cj.load(os.path.join(os.path.expanduser("~"), ".netscape", "cookies.txt"))
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
The next example illustrates the use of *note DefaultCookiePolicy: 1a10.
Turn on RFC 2965(1) cookies, be more strict about domains when setting
and returning Netscape cookies, and block some domains from setting
cookies or having them returned:
import urllib.request
from http.cookiejar import CookieJar, DefaultCookiePolicy
policy = DefaultCookiePolicy(
rfc2965=True, strict_ns_domain=Policy.DomainStrict,
blocked_domains=["ads.net", ".ads.net"])
cj = CookieJar(policy)
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2965.html
File: python.info, Node: xmlrpc — XMLRPC server and client modules, Next: xmlrpc client — XML-RPC client access, Prev: http cookiejar — Cookie handling for HTTP clients, Up: Internet Protocols and Support
5.22.20 ‘xmlrpc’ — XMLRPC server and client modules
---------------------------------------------------
XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP
as a transport. With it, a client can call methods with parameters on a
remote server (the server is named by a URI) and get back structured
data.
‘xmlrpc’ is a package that collects server and client modules
implementing XML-RPC. The modules are:
* *note xmlrpc.client: 12e.
* *note xmlrpc.server: 12f.
File: python.info, Node: xmlrpc client — XML-RPC client access, Next: xmlrpc server — Basic XML-RPC servers, Prev: xmlrpc — XMLRPC server and client modules, Up: Internet Protocols and Support
5.22.21 ‘xmlrpc.client’ — XML-RPC client access
-----------------------------------------------
'Source code:' Lib/xmlrpc/client.py(1)
__________________________________________________________________
XML-RPC is a Remote Procedure Call method that uses XML passed via
HTTP(S) as a transport. With it, a client can call methods with
parameters on a remote server (the server is named by a URI) and get
back structured data. This module supports writing XML-RPC client code;
it handles all the details of translating between conformable Python
objects and XML on the wire.
Warning: The *note xmlrpc.client: 12e. module is not secure against
maliciously constructed data. If you need to parse untrusted or
unauthenticated data, see *note XML security: 3774.
Changed in version 3.5: For HTTPS URIs, *note xmlrpc.client: 12e. now
performs all the necessary certificate and hostname checks by default.
*note Availability: 1d54.: not WASI.
This module does not work or is not available on WebAssembly. See *note
WebAssembly platforms: 17e0. for more information.
-- Class: xmlrpc.client.ServerProxy (uri, transport=None,
encoding=None, verbose=False, allow_none=False,
use_datetime=False, use_builtin_types=False, *, headers=(),
context=None)
A *note ServerProxy: a56. instance is an object that manages
communication with a remote XML-RPC server. The required first
argument is a URI (Uniform Resource Indicator), and will normally
be the URL of the server. The optional second argument is a
transport factory instance; by default it is an internal
‘SafeTransport’ instance for https: URLs and an internal HTTP
‘Transport’ instance otherwise. The optional third argument is an
encoding, by default UTF-8. The optional fourth argument is a
debugging flag.
The following parameters govern the use of the returned proxy
instance. If 'allow_none' is true, the Python constant ‘None’ will
be translated into XML; the default behaviour is for ‘None’ to
raise a *note TypeError: 534. This is a commonly used extension to
the XML-RPC specification, but isn’t supported by all clients and
servers; see http://ontosys.com/xml-rpc/extensions.php(2) for a
description. The 'use_builtin_types' flag can be used to cause
date/time values to be presented as *note datetime.datetime: 1cc.
objects and binary data to be presented as *note bytes: 1c2.
objects; this flag is false by default. *note datetime.datetime:
1cc, *note bytes: 1c2. and *note bytearray: 53a. objects may be
passed to calls. The 'headers' parameter is an optional sequence
of HTTP headers to send with each request, expressed as a sequence
of 2-tuples representing the header name and value. (e.g.
‘[('Header-Name', 'value')]’). If an HTTPS URL is provided,
'context' may be *note ssl.SSLContext: 296. and configures the SSL
settings of the underlying HTTPS connection. The obsolete
'use_datetime' flag is similar to 'use_builtin_types' but it
applies only to date/time values.
Changed in version 3.3: The 'use_builtin_types' flag was added.
Changed in version 3.8: The 'headers' parameter was added.
Both the HTTP and HTTPS transports support the URL syntax extension
for HTTP Basic Authentication: ‘http://user:pass@host:port/path’.
The ‘user:pass’ portion will be base64-encoded as an HTTP
‘Authorization’ header, and sent to the remote server as part of
the connection process when invoking an XML-RPC method. You only
need to use this if the remote server requires a Basic
Authentication user and password.
The returned instance is a proxy object with methods that can be
used to invoke corresponding RPC calls on the remote server. If
the remote server supports the introspection API, the proxy can
also be used to query the remote server for the methods it supports
(service discovery) and fetch other server-associated metadata.
Types that are conformable (e.g. that can be marshalled through
XML), include the following (and except where noted, they are
unmarshalled as the same Python type):
XML-RPC type Python type
---------------------------------------------------------------------------------------
‘boolean’ *note bool: 463.
‘int’, ‘i1’, ‘i2’, ‘i4’, *note int: 259. in range from -2147483648 to 2147483647.
‘i8’ or ‘biginteger’ Values get the ‘’ tag.
‘double’ or ‘float’ *note float: 2f1. Values get the ‘’ tag.
‘string’ *note str: 447.
‘array’ *note list: 60d. or *note tuple: 36b. containing
conformable elements. Arrays are returned as
*note lists: 60d.
‘struct’ *note dict: 258. Keys must be strings, values may be any
conformable type. Objects of user-defined classes can be
passed in; only their *note __dict__: 558. attribute is
transmitted.
‘dateTime.iso8601’ *note DateTime: 13af. or *note datetime.datetime: 1cc.
Returned type depends on values of 'use_builtin_types'
and 'use_datetime' flags.
‘base64’ *note Binary: 3bfc, *note bytes: 1c2. or
*note bytearray: 53a. Returned type depends on the value
of the 'use_builtin_types' flag.
‘nil’ The ‘None’ constant. Passing is allowed only if
'allow_none' is true.
‘bigdecimal’ *note decimal.Decimal: 29f. Returned type only.
This is the full set of data types supported by XML-RPC. Method
calls may also raise a special *note Fault: 3bfd. instance, used to
signal XML-RPC server errors, or *note ProtocolError: 3bfe. used to
signal an error in the HTTP/HTTPS transport layer. Both *note
Fault: 3bfd. and *note ProtocolError: 3bfe. derive from a base
class called ‘Error’. Note that the xmlrpc client module currently
does not marshal instances of subclasses of built-in types.
When passing strings, characters special to XML such as ‘<’, ‘>’,
and ‘&’ will be automatically escaped. However, it’s the caller’s
responsibility to ensure that the string is free of characters that
aren’t allowed in XML, such as the control characters with ASCII
values between 0 and 31 (except, of course, tab, newline and
carriage return); failing to do this will result in an XML-RPC
request that isn’t well-formed XML. If you have to pass arbitrary
bytes via XML-RPC, use *note bytes: 1c2. or *note bytearray: 53a.
classes or the *note Binary: 3bfc. wrapper class described below.
‘Server’ is retained as an alias for *note ServerProxy: a56. for
backwards compatibility. New code should use *note ServerProxy:
a56.
Changed in version 3.5: Added the 'context' argument.
Changed in version 3.6: Added support of type tags with prefixes
(e.g. ‘ex:nil’). Added support of unmarshalling additional types
used by Apache XML-RPC implementation for numerics: ‘i1’, ‘i2’,
‘i8’, ‘biginteger’, ‘float’ and ‘bigdecimal’. See
‘https://ws.apache.org/xmlrpc/types.html’ for a description.
See also
........
XML-RPC HOWTO(3)
A good description of XML-RPC operation and client software in
several languages. Contains pretty much everything an XML-RPC
client developer needs to know.
XML-RPC Introspection(4)
Describes the XML-RPC protocol extension for introspection.
XML-RPC Specification(5)
The official specification.
* Menu:
* ServerProxy Objects::
* DateTime Objects::
* Binary Objects::
* Fault Objects::
* ProtocolError Objects::
* MultiCall Objects::
* Convenience Functions::
* Example of Client Usage::
* Example of Client and Server Usage::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/xmlrpc/client.py
(2)
https://web.archive.org/web/20130120074804/http://ontosys.com/xml-rpc/extensions.php
(3) https://tldp.org/HOWTO/XML-RPC-HOWTO/index.html
(4) https://xmlrpc-c.sourceforge.io/introspection.html
(5) http://xmlrpc.scripting.com/spec.html
File: python.info, Node: ServerProxy Objects, Next: DateTime Objects, Up: xmlrpc client — XML-RPC client access
5.22.21.1 ServerProxy Objects
.............................
A *note ServerProxy: a56. instance has a method corresponding to each
remote procedure call accepted by the XML-RPC server. Calling the
method performs an RPC, dispatched by both name and argument signature
(e.g. the same method name can be overloaded with multiple argument
signatures). The RPC finishes by returning a value, which may be either
returned data in a conformant type or a *note Fault: 3bfd. or *note
ProtocolError: 3bfe. object indicating an error.
Servers that support the XML introspection API support some common
methods grouped under the reserved ‘system’ attribute:
-- Method: ServerProxy.system.listMethods ()
This method returns a list of strings, one for each (non-system)
method supported by the XML-RPC server.
-- Method: ServerProxy.system.methodSignature (name)
This method takes one parameter, the name of a method implemented
by the XML-RPC server. It returns an array of possible signatures
for this method. A signature is an array of types. The first of
these types is the return type of the method, the rest are
parameters.
Because multiple signatures (ie. overloading) is permitted, this
method returns a list of signatures rather than a singleton.
Signatures themselves are restricted to the top level parameters
expected by a method. For instance if a method expects one array
of structs as a parameter, and it returns a string, its signature
is simply “string, array”. If it expects three integers and
returns a string, its signature is “string, int, int, int”.
If no signature is defined for the method, a non-array value is
returned. In Python this means that the type of the returned value
will be something other than list.
-- Method: ServerProxy.system.methodHelp (name)
This method takes one parameter, the name of a method implemented
by the XML-RPC server. It returns a documentation string
describing the use of that method. If no such string is available,
an empty string is returned. The documentation string may contain
HTML markup.
Changed in version 3.5: Instances of *note ServerProxy: a56. support the
*note context manager: 5d0. protocol for closing the underlying
transport.
A working example follows. The server code:
from xmlrpc.server import SimpleXMLRPCServer
def is_even(n):
return n % 2 == 0
server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(is_even, "is_even")
server.serve_forever()
The client code for the preceding server:
import xmlrpc.client
with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy:
print("3 is even: %s" % str(proxy.is_even(3)))
print("100 is even: %s" % str(proxy.is_even(100)))
File: python.info, Node: DateTime Objects, Next: Binary Objects, Prev: ServerProxy Objects, Up: xmlrpc client — XML-RPC client access
5.22.21.2 DateTime Objects
..........................
-- Class: xmlrpc.client.DateTime
This class may be initialized with seconds since the epoch, a time
tuple, an ISO 8601 time/date string, or a *note datetime.datetime:
1cc. instance. It has the following methods, supported mainly for
internal use by the marshalling/unmarshalling code:
-- Method: decode (string)
Accept a string as the instance’s new time value.
-- Method: encode (out)
Write the XML-RPC encoding of this *note DateTime: 13af. item
to the 'out' stream object.
It also supports certain of Python’s built-in operators through
*note rich comparison: 1292. and *note __repr__(): 618. methods.
A working example follows. The server code:
import datetime
from xmlrpc.server import SimpleXMLRPCServer
import xmlrpc.client
def today():
today = datetime.datetime.today()
return xmlrpc.client.DateTime(today)
server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(today, "today")
server.serve_forever()
The client code for the preceding server:
import xmlrpc.client
import datetime
proxy = xmlrpc.client.ServerProxy("http://localhost:8000/")
today = proxy.today()
# convert the ISO8601 string to a datetime object
converted = datetime.datetime.strptime(today.value, "%Y%m%dT%H:%M:%S")
print("Today: %s" % converted.strftime("%d.%m.%Y, %H:%M"))
File: python.info, Node: Binary Objects, Next: Fault Objects, Prev: DateTime Objects, Up: xmlrpc client — XML-RPC client access
5.22.21.3 Binary Objects
........................
-- Class: xmlrpc.client.Binary
This class may be initialized from bytes data (which may include
NULs). The primary access to the content of a *note Binary: 3bfc.
object is provided by an attribute:
-- Attribute: data
The binary data encapsulated by the *note Binary: 3bfc.
instance. The data is provided as a *note bytes: 1c2. object.
*note Binary: 3bfc. objects have the following methods, supported
mainly for internal use by the marshalling/unmarshalling code:
-- Method: decode (bytes)
Accept a base64 *note bytes: 1c2. object and decode it as the
instance’s new data.
-- Method: encode (out)
Write the XML-RPC base 64 encoding of this binary item to the
'out' stream object.
The encoded data will have newlines every 76 characters as per
RFC 2045 section 6.8(1), which was the de facto standard
base64 specification when the XML-RPC spec was written.
It also supports certain of Python’s built-in operators through
*note __eq__(): afa. and *note __ne__(): 14e2. methods.
Example usage of the binary objects. We’re going to transfer an image
over XMLRPC:
from xmlrpc.server import SimpleXMLRPCServer
import xmlrpc.client
def python_logo():
with open("python_logo.jpg", "rb") as handle:
return xmlrpc.client.Binary(handle.read())
server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(python_logo, 'python_logo')
server.serve_forever()
The client gets the image and saves it to a file:
import xmlrpc.client
proxy = xmlrpc.client.ServerProxy("http://localhost:8000/")
with open("fetched_python_logo.jpg", "wb") as handle:
handle.write(proxy.python_logo().data)
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc2045.html#section-6.8
File: python.info, Node: Fault Objects, Next: ProtocolError Objects, Prev: Binary Objects, Up: xmlrpc client — XML-RPC client access
5.22.21.4 Fault Objects
.......................
-- Class: xmlrpc.client.Fault
A *note Fault: 3bfd. object encapsulates the content of an XML-RPC
fault tag. Fault objects have the following attributes:
-- Attribute: faultCode
An int indicating the fault type.
-- Attribute: faultString
A string containing a diagnostic message associated with the
fault.
In the following example we’re going to intentionally cause a *note
Fault: 3bfd. by returning a complex type object. The server code:
from xmlrpc.server import SimpleXMLRPCServer
# A marshalling error is going to occur because we're returning a
# complex number
def add(x, y):
return x+y+0j
server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_function(add, 'add')
server.serve_forever()
The client code for the preceding server:
import xmlrpc.client
proxy = xmlrpc.client.ServerProxy("http://localhost:8000/")
try:
proxy.add(2, 5)
except xmlrpc.client.Fault as err:
print("A fault occurred")
print("Fault code: %d" % err.faultCode)
print("Fault string: %s" % err.faultString)
File: python.info, Node: ProtocolError Objects, Next: MultiCall Objects, Prev: Fault Objects, Up: xmlrpc client — XML-RPC client access
5.22.21.5 ProtocolError Objects
...............................
-- Class: xmlrpc.client.ProtocolError
A *note ProtocolError: 3bfe. object describes a protocol error in
the underlying transport layer (such as a 404 ‘not found’ error if
the server named by the URI does not exist). It has the following
attributes:
-- Attribute: url
The URI or URL that triggered the error.
-- Attribute: errcode
The error code.
-- Attribute: errmsg
The error message or diagnostic string.
-- Attribute: headers
A dict containing the headers of the HTTP/HTTPS request that
triggered the error.
In the following example we’re going to intentionally cause a *note
ProtocolError: 3bfe. by providing an invalid URI:
import xmlrpc.client
# create a ServerProxy with a URI that doesn't respond to XMLRPC requests
proxy = xmlrpc.client.ServerProxy("http://google.com/")
try:
proxy.some_method()
except xmlrpc.client.ProtocolError as err:
print("A protocol error occurred")
print("URL: %s" % err.url)
print("HTTP/HTTPS headers: %s" % err.headers)
print("Error code: %d" % err.errcode)
print("Error message: %s" % err.errmsg)
File: python.info, Node: MultiCall Objects, Next: Convenience Functions, Prev: ProtocolError Objects, Up: xmlrpc client — XML-RPC client access
5.22.21.6 MultiCall Objects
...........................
The *note MultiCall: 3c18. object provides a way to encapsulate multiple
calls to a remote server into a single request (1).
-- Class: xmlrpc.client.MultiCall (server)
Create an object used to boxcar method calls. 'server' is the
eventual target of the call. Calls can be made to the result
object, but they will immediately return ‘None’, and only store the
call name and parameters in the *note MultiCall: 3c18. object.
Calling the object itself causes all stored calls to be transmitted
as a single ‘system.multicall’ request. The result of this call is
a *note generator: 105a.; iterating over this generator yields the
individual results.
A usage example of this class follows. The server code:
from xmlrpc.server import SimpleXMLRPCServer
def add(x, y):
return x + y
def subtract(x, y):
return x - y
def multiply(x, y):
return x * y
def divide(x, y):
return x // y
# A simple server with simple arithmetic functions
server = SimpleXMLRPCServer(("localhost", 8000))
print("Listening on port 8000...")
server.register_multicall_functions()
server.register_function(add, 'add')
server.register_function(subtract, 'subtract')
server.register_function(multiply, 'multiply')
server.register_function(divide, 'divide')
server.serve_forever()
The client code for the preceding server:
import xmlrpc.client
proxy = xmlrpc.client.ServerProxy("http://localhost:8000/")
multicall = xmlrpc.client.MultiCall(proxy)
multicall.add(7, 3)
multicall.subtract(7, 3)
multicall.multiply(7, 3)
multicall.divide(7, 3)
result = multicall()
print("7+3=%d, 7-3=%d, 7*3=%d, 7//3=%d" % tuple(result))
---------- Footnotes ----------
(1) This approach has been first presented in a discussion on
xmlrpc.com
(https://web.archive.org/web/20060624230303/http://www.xmlrpc.com/discuss/msgReader$1208?mode=topic).
File: python.info, Node: Convenience Functions, Next: Example of Client Usage, Prev: MultiCall Objects, Up: xmlrpc client — XML-RPC client access
5.22.21.7 Convenience Functions
...............................
-- Function: xmlrpc.client.dumps (params, methodname=None,
methodresponse=None, encoding=None, allow_none=False)
Convert 'params' into an XML-RPC request. or into a response if
'methodresponse' is true. 'params' can be either a tuple of
arguments or an instance of the *note Fault: 3bfd. exception class.
If 'methodresponse' is true, only a single value can be returned,
meaning that 'params' must be of length 1. 'encoding', if
supplied, is the encoding to use in the generated XML; the default
is UTF-8. Python’s *note None: 671. value cannot be used in
standard XML-RPC; to allow using it via an extension, provide a
true value for 'allow_none'.
-- Function: xmlrpc.client.loads (data, use_datetime=False,
use_builtin_types=False)
Convert an XML-RPC request or response into Python objects, a
‘(params, methodname)’. 'params' is a tuple of argument;
'methodname' is a string, or ‘None’ if no method name is present in
the packet. If the XML-RPC packet represents a fault condition,
this function will raise a *note Fault: 3bfd. exception. The
'use_builtin_types' flag can be used to cause date/time values to
be presented as *note datetime.datetime: 1cc. objects and binary
data to be presented as *note bytes: 1c2. objects; this flag is
false by default.
The obsolete 'use_datetime' flag is similar to 'use_builtin_types'
but it applies only to date/time values.
Changed in version 3.3: The 'use_builtin_types' flag was added.
File: python.info, Node: Example of Client Usage, Next: Example of Client and Server Usage, Prev: Convenience Functions, Up: xmlrpc client — XML-RPC client access
5.22.21.8 Example of Client Usage
.................................
# simple test program (from the XML-RPC specification)
from xmlrpc.client import ServerProxy, Error
# server = ServerProxy("http://localhost:8000") # local server
with ServerProxy("http://betty.userland.com") as proxy:
print(proxy)
try:
print(proxy.examples.getStateName(41))
except Error as v:
print("ERROR", v)
To access an XML-RPC server through a HTTP proxy, you need to define a
custom transport. The following example shows how:
import http.client
import xmlrpc.client
class ProxiedTransport(xmlrpc.client.Transport):
def set_proxy(self, host, port=None, headers=None):
self.proxy = host, port
self.proxy_headers = headers
def make_connection(self, host):
connection = http.client.HTTPConnection(*self.proxy)
connection.set_tunnel(host, headers=self.proxy_headers)
self._connection = host, connection
return connection
transport = ProxiedTransport()
transport.set_proxy('proxy-server', 8080)
server = xmlrpc.client.ServerProxy('http://betty.userland.com', transport=transport)
print(server.examples.getStateName(41))
File: python.info, Node: Example of Client and Server Usage, Prev: Example of Client Usage, Up: xmlrpc client — XML-RPC client access
5.22.21.9 Example of Client and Server Usage
............................................
See *note SimpleXMLRPCServer Example: 3c1e.
File: python.info, Node: xmlrpc server — Basic XML-RPC servers, Next: ipaddress — IPv4/IPv6 manipulation library, Prev: xmlrpc client — XML-RPC client access, Up: Internet Protocols and Support
5.22.22 ‘xmlrpc.server’ — Basic XML-RPC servers
-----------------------------------------------
'Source code:' Lib/xmlrpc/server.py(1)
__________________________________________________________________
The *note xmlrpc.server: 12f. module provides a basic server framework
for XML-RPC servers written in Python. Servers can either be free
standing, using *note SimpleXMLRPCServer: 3c21, or embedded in a CGI
environment, using *note CGIXMLRPCRequestHandler: 3c22.
Warning: The *note xmlrpc.server: 12f. module is not secure against
maliciously constructed data. If you need to parse untrusted or
unauthenticated data, see *note XML security: 3774.
*note Availability: 1d54.: not WASI.
This module does not work or is not available on WebAssembly. See *note
WebAssembly platforms: 17e0. for more information.
-- Class: xmlrpc.server.SimpleXMLRPCServer (addr,
requestHandler=SimpleXMLRPCRequestHandler, logRequests=True,
allow_none=False, encoding=None, bind_and_activate=True,
use_builtin_types=False)
Create a new server instance. This class provides methods for
registration of functions that can be called by the XML-RPC
protocol. The 'requestHandler' parameter should be a factory for
request handler instances; it defaults to *note
SimpleXMLRPCRequestHandler: 131a. The 'addr' and 'requestHandler'
parameters are passed to the *note socketserver.TCPServer: 1312.
constructor. If 'logRequests' is true (the default), requests will
be logged; setting this parameter to false will turn off logging.
The 'allow_none' and 'encoding' parameters are passed on to *note
xmlrpc.client: 12e. and control the XML-RPC responses that will be
returned from the server. The 'bind_and_activate' parameter
controls whether ‘server_bind()’ and ‘server_activate()’ are called
immediately by the constructor; it defaults to true. Setting it to
false allows code to manipulate the 'allow_reuse_address' class
variable before the address is bound. The 'use_builtin_types'
parameter is passed to the *note loads(): 1400. function and
controls which types are processed when date/times values or binary
data are received; it defaults to false.
Changed in version 3.3: The 'use_builtin_types' flag was added.
-- Class: xmlrpc.server.CGIXMLRPCRequestHandler (allow_none=False,
encoding=None, use_builtin_types=False)
Create a new instance to handle XML-RPC requests in a CGI
environment. The 'allow_none' and 'encoding' parameters are passed
on to *note xmlrpc.client: 12e. and control the XML-RPC responses
that will be returned from the server. The 'use_builtin_types'
parameter is passed to the *note loads(): 1400. function and
controls which types are processed when date/times values or binary
data are received; it defaults to false.
Changed in version 3.3: The 'use_builtin_types' flag was added.
-- Class: xmlrpc.server.SimpleXMLRPCRequestHandler
Create a new request handler instance. This request handler
supports ‘POST’ requests and modifies logging so that the
'logRequests' parameter to the *note SimpleXMLRPCServer: 3c21.
constructor parameter is honored.
* Menu:
* SimpleXMLRPCServer Objects::
* CGIXMLRPCRequestHandler::
* Documenting XMLRPC server::
* DocXMLRPCServer Objects::
* DocCGIXMLRPCRequestHandler::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/xmlrpc/server.py
File: python.info, Node: SimpleXMLRPCServer Objects, Next: CGIXMLRPCRequestHandler, Up: xmlrpc server — Basic XML-RPC servers
5.22.22.1 SimpleXMLRPCServer Objects
....................................
The *note SimpleXMLRPCServer: 3c21. class is based on *note
socketserver.TCPServer: 1312. and provides a means of creating simple,
stand alone XML-RPC servers.
-- Method: SimpleXMLRPCServer.register_function (function=None,
name=None)
Register a function that can respond to XML-RPC requests. If
'name' is given, it will be the method name associated with
'function', otherwise *note function.__name__: 12c7. will be used.
'name' is a string, and may contain characters not legal in Python
identifiers, including the period character.
This method can also be used as a decorator. When used as a
decorator, 'name' can only be given as a keyword argument to
register 'function' under 'name'. If no 'name' is given, *note
function.__name__: 12c7. will be used.
Changed in version 3.7: *note register_function(): 3c25. can be
used as a decorator.
-- Method: SimpleXMLRPCServer.register_instance (instance,
allow_dotted_names=False)
Register an object which is used to expose method names which have
not been registered using *note register_function(): 3c25. If
'instance' contains a ‘_dispatch()’ method, it is called with the
requested method name and the parameters from the request. Its API
is ‘def _dispatch(self, method, params)’ (note that 'params' does
not represent a variable argument list). If it calls an underlying
function to perform its task, that function is called as
‘func(*params)’, expanding the parameter list. The return value
from ‘_dispatch()’ is returned to the client as the result. If
'instance' does not have a ‘_dispatch()’ method, it is searched for
an attribute matching the name of the requested method.
If the optional 'allow_dotted_names' argument is true and the
instance does not have a ‘_dispatch()’ method, then if the
requested method name contains periods, each component of the
method name is searched for individually, with the effect that a
simple hierarchical search is performed. The value found from this
search is then called with the parameters from the request, and the
return value is passed back to the client.
Warning: Enabling the 'allow_dotted_names' option allows
intruders to access your module’s global variables and may
allow intruders to execute arbitrary code on your machine.
Only use this option on a secure, closed network.
-- Method: SimpleXMLRPCServer.register_introspection_functions ()
Registers the XML-RPC introspection functions ‘system.listMethods’,
‘system.methodHelp’ and ‘system.methodSignature’.
-- Method: SimpleXMLRPCServer.register_multicall_functions ()
Registers the XML-RPC multicall function system.multicall.
-- Attribute: SimpleXMLRPCRequestHandler.rpc_paths
An attribute value that must be a tuple listing valid path portions
of the URL for receiving XML-RPC requests. Requests posted to
other paths will result in a 404 “no such page” HTTP error. If
this tuple is empty, all paths will be considered valid. The
default value is ‘('/', '/RPC2')’.
* Menu:
* SimpleXMLRPCServer Example::
File: python.info, Node: SimpleXMLRPCServer Example, Up: SimpleXMLRPCServer Objects
5.22.22.2 SimpleXMLRPCServer Example
....................................
Server code:
from xmlrpc.server import SimpleXMLRPCServer
from xmlrpc.server import SimpleXMLRPCRequestHandler
# Restrict to a particular path.
class RequestHandler(SimpleXMLRPCRequestHandler):
rpc_paths = ('/RPC2',)
# Create server
with SimpleXMLRPCServer(('localhost', 8000),
requestHandler=RequestHandler) as server:
server.register_introspection_functions()
# Register pow() function; this will use the value of
# pow.__name__ as the name, which is just 'pow'.
server.register_function(pow)
# Register a function under a different name
def adder_function(x, y):
return x + y
server.register_function(adder_function, 'add')
# Register an instance; all the methods of the instance are
# published as XML-RPC methods (in this case, just 'mul').
class MyFuncs:
def mul(self, x, y):
return x * y
server.register_instance(MyFuncs())
# Run the server's main loop
server.serve_forever()
The following client code will call the methods made available by the
preceding server:
import xmlrpc.client
s = xmlrpc.client.ServerProxy('http://localhost:8000')
print(s.pow(2,3)) # Returns 2**3 = 8
print(s.add(2,3)) # Returns 5
print(s.mul(5,2)) # Returns 5*2 = 10
# Print list of available methods
print(s.system.listMethods())
‘register_function()’ can also be used as a decorator. The previous
server example can register functions in a decorator way:
from xmlrpc.server import SimpleXMLRPCServer
from xmlrpc.server import SimpleXMLRPCRequestHandler
class RequestHandler(SimpleXMLRPCRequestHandler):
rpc_paths = ('/RPC2',)
with SimpleXMLRPCServer(('localhost', 8000),
requestHandler=RequestHandler) as server:
server.register_introspection_functions()
# Register pow() function; this will use the value of
# pow.__name__ as the name, which is just 'pow'.
server.register_function(pow)
# Register a function under a different name, using
# register_function as a decorator. *name* can only be given
# as a keyword argument.
@server.register_function(name='add')
def adder_function(x, y):
return x + y
# Register a function under function.__name__.
@server.register_function
def mul(x, y):
return x * y
server.serve_forever()
The following example included in the ‘Lib/xmlrpc/server.py’ module
shows a server allowing dotted names and registering a multicall
function.
Warning: Enabling the 'allow_dotted_names' option allows intruders
to access your module’s global variables and may allow intruders to
execute arbitrary code on your machine. Only use this example only
within a secure, closed network.
import datetime
class ExampleService:
def getData(self):
return '42'
class currentTime:
@staticmethod
def getCurrentTime():
return datetime.datetime.now()
with SimpleXMLRPCServer(("localhost", 8000)) as server:
server.register_function(pow)
server.register_function(lambda x,y: x+y, 'add')
server.register_instance(ExampleService(), allow_dotted_names=True)
server.register_multicall_functions()
print('Serving XML-RPC on localhost port 8000')
try:
server.serve_forever()
except KeyboardInterrupt:
print("\nKeyboard interrupt received, exiting.")
sys.exit(0)
This ExampleService demo can be invoked from the command line:
python -m xmlrpc.server
The client that interacts with the above server is included in
‘Lib/xmlrpc/client.py’:
server = ServerProxy("http://localhost:8000")
try:
print(server.currentTime.getCurrentTime())
except Error as v:
print("ERROR", v)
multi = MultiCall(server)
multi.getData()
multi.pow(2,9)
multi.add(1,2)
try:
for response in multi():
print(response)
except Error as v:
print("ERROR", v)
This client which interacts with the demo XMLRPC server can be invoked
as:
python -m xmlrpc.client
File: python.info, Node: CGIXMLRPCRequestHandler, Next: Documenting XMLRPC server, Prev: SimpleXMLRPCServer Objects, Up: xmlrpc server — Basic XML-RPC servers
5.22.22.3 CGIXMLRPCRequestHandler
.................................
The *note CGIXMLRPCRequestHandler: 3c22. class can be used to handle
XML-RPC requests sent to Python CGI scripts.
-- Method: CGIXMLRPCRequestHandler.register_function (function=None,
name=None)
Register a function that can respond to XML-RPC requests. If
'name' is given, it will be the method name associated with
'function', otherwise *note function.__name__: 12c7. will be used.
'name' is a string, and may contain characters not legal in Python
identifiers, including the period character.
This method can also be used as a decorator. When used as a
decorator, 'name' can only be given as a keyword argument to
register 'function' under 'name'. If no 'name' is given, *note
function.__name__: 12c7. will be used.
Changed in version 3.7: *note register_function(): 3c2c. can be
used as a decorator.
-- Method: CGIXMLRPCRequestHandler.register_instance (instance)
Register an object which is used to expose method names which have
not been registered using *note register_function(): 3c2c. If
instance contains a ‘_dispatch()’ method, it is called with the
requested method name and the parameters from the request; the
return value is returned to the client as the result. If instance
does not have a ‘_dispatch()’ method, it is searched for an
attribute matching the name of the requested method; if the
requested method name contains periods, each component of the
method name is searched for individually, with the effect that a
simple hierarchical search is performed. The value found from this
search is then called with the parameters from the request, and the
return value is passed back to the client.
-- Method: CGIXMLRPCRequestHandler.register_introspection_functions ()
Register the XML-RPC introspection functions ‘system.listMethods’,
‘system.methodHelp’ and ‘system.methodSignature’.
-- Method: CGIXMLRPCRequestHandler.register_multicall_functions ()
Register the XML-RPC multicall function ‘system.multicall’.
-- Method: CGIXMLRPCRequestHandler.handle_request (request_text=None)
Handle an XML-RPC request. If 'request_text' is given, it should
be the POST data provided by the HTTP server, otherwise the
contents of stdin will be used.
Example:
class MyFuncs:
def mul(self, x, y):
return x * y
handler = CGIXMLRPCRequestHandler()
handler.register_function(pow)
handler.register_function(lambda x,y: x+y, 'add')
handler.register_introspection_functions()
handler.register_instance(MyFuncs())
handler.handle_request()
File: python.info, Node: Documenting XMLRPC server, Next: DocXMLRPCServer Objects, Prev: CGIXMLRPCRequestHandler, Up: xmlrpc server — Basic XML-RPC servers
5.22.22.4 Documenting XMLRPC server
...................................
These classes extend the above classes to serve HTML documentation in
response to HTTP GET requests. Servers can either be free standing,
using *note DocXMLRPCServer: 19bd, or embedded in a CGI environment,
using *note DocCGIXMLRPCRequestHandler: 3c32.
-- Class: xmlrpc.server.DocXMLRPCServer (addr,
requestHandler=DocXMLRPCRequestHandler, logRequests=True,
allow_none=False, encoding=None, bind_and_activate=True,
use_builtin_types=True)
Create a new server instance. All parameters have the same meaning
as for *note SimpleXMLRPCServer: 3c21.; 'requestHandler' defaults
to *note DocXMLRPCRequestHandler: 3c33.
Changed in version 3.3: The 'use_builtin_types' flag was added.
-- Class: xmlrpc.server.DocCGIXMLRPCRequestHandler
Create a new instance to handle XML-RPC requests in a CGI
environment.
-- Class: xmlrpc.server.DocXMLRPCRequestHandler
Create a new request handler instance. This request handler
supports XML-RPC POST requests, documentation GET requests, and
modifies logging so that the 'logRequests' parameter to the *note
DocXMLRPCServer: 19bd. constructor parameter is honored.
File: python.info, Node: DocXMLRPCServer Objects, Next: DocCGIXMLRPCRequestHandler, Prev: Documenting XMLRPC server, Up: xmlrpc server — Basic XML-RPC servers
5.22.22.5 DocXMLRPCServer Objects
.................................
The *note DocXMLRPCServer: 19bd. class is derived from *note
SimpleXMLRPCServer: 3c21. and provides a means of creating
self-documenting, stand alone XML-RPC servers. HTTP POST requests are
handled as XML-RPC method calls. HTTP GET requests are handled by
generating pydoc-style HTML documentation. This allows a server to
provide its own web-based documentation.
-- Method: DocXMLRPCServer.set_server_title (server_title)
Set the title used in the generated HTML documentation. This title
will be used inside the HTML “title” element.
-- Method: DocXMLRPCServer.set_server_name (server_name)
Set the name used in the generated HTML documentation. This name
will appear at the top of the generated documentation inside a “h1”
element.
-- Method: DocXMLRPCServer.set_server_documentation
(server_documentation)
Set the description used in the generated HTML documentation. This
description will appear as a paragraph, below the server name, in
the documentation.
File: python.info, Node: DocCGIXMLRPCRequestHandler, Prev: DocXMLRPCServer Objects, Up: xmlrpc server — Basic XML-RPC servers
5.22.22.6 DocCGIXMLRPCRequestHandler
....................................
The *note DocCGIXMLRPCRequestHandler: 3c32. class is derived from *note
CGIXMLRPCRequestHandler: 3c22. and provides a means of creating
self-documenting, XML-RPC CGI scripts. HTTP POST requests are handled
as XML-RPC method calls. HTTP GET requests are handled by generating
pydoc-style HTML documentation. This allows a server to provide its own
web-based documentation.
-- Method: DocCGIXMLRPCRequestHandler.set_server_title (server_title)
Set the title used in the generated HTML documentation. This title
will be used inside the HTML “title” element.
-- Method: DocCGIXMLRPCRequestHandler.set_server_name (server_name)
Set the name used in the generated HTML documentation. This name
will appear at the top of the generated documentation inside a “h1”
element.
-- Method: DocCGIXMLRPCRequestHandler.set_server_documentation
(server_documentation)
Set the description used in the generated HTML documentation. This
description will appear as a paragraph, below the server name, in
the documentation.
File: python.info, Node: ipaddress — IPv4/IPv6 manipulation library, Prev: xmlrpc server — Basic XML-RPC servers, Up: Internet Protocols and Support
5.22.23 ‘ipaddress’ — IPv4/IPv6 manipulation library
----------------------------------------------------
'Source code:' Lib/ipaddress.py(1)
__________________________________________________________________
*note ipaddress: 80. provides the capabilities to create, manipulate and
operate on IPv4 and IPv6 addresses and networks.
The functions and classes in this module make it straightforward to
handle various tasks related to IP addresses, including checking whether
or not two hosts are on the same subnet, iterating over all hosts in a
particular subnet, checking whether or not a string represents a valid
IP address or network definition, and so on.
This is the full module API reference—for an overview and introduction,
see *note An introduction to the ipaddress module: 3c3f.
Added in version 3.3.
* Menu:
* Convenience factory functions::
* IP Addresses::
* IP Network definitions::
* Interface objects::
* Other Module Level Functions::
* Custom Exceptions::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/ipaddress.py
File: python.info, Node: Convenience factory functions, Next: IP Addresses, Up: ipaddress — IPv4/IPv6 manipulation library
5.22.23.1 Convenience factory functions
.......................................
The *note ipaddress: 80. module provides factory functions to
conveniently create IP addresses, networks and interfaces:
-- Function: ipaddress.ip_address (address)
Return an *note IPv4Address: 1fe. or *note IPv6Address: 1ff. object
depending on the IP address passed as argument. Either IPv4 or
IPv6 addresses may be supplied; integers less than ‘2**32’ will be
considered to be IPv4 by default. A *note ValueError: 204. is
raised if 'address' does not represent a valid IPv4 or IPv6
address.
>>> ipaddress.ip_address('192.168.0.1')
IPv4Address('192.168.0.1')
>>> ipaddress.ip_address('2001:db8::')
IPv6Address('2001:db8::')
-- Function: ipaddress.ip_network (address, strict=True)
Return an *note IPv4Network: 200. or *note IPv6Network: 201. object
depending on the IP address passed as argument. 'address' is a
string or integer representing the IP network. Either IPv4 or IPv6
networks may be supplied; integers less than ‘2**32’ will be
considered to be IPv4 by default. 'strict' is passed to *note
IPv4Network: 200. or *note IPv6Network: 201. constructor. A *note
ValueError: 204. is raised if 'address' does not represent a valid
IPv4 or IPv6 address, or if the network has host bits set.
>>> ipaddress.ip_network('192.168.0.0/28')
IPv4Network('192.168.0.0/28')
-- Function: ipaddress.ip_interface (address)
Return an *note IPv4Interface: 180d. or *note IPv6Interface: 180e.
object depending on the IP address passed as argument. 'address'
is a string or integer representing the IP address. Either IPv4 or
IPv6 addresses may be supplied; integers less than ‘2**32’ will be
considered to be IPv4 by default. A *note ValueError: 204. is
raised if 'address' does not represent a valid IPv4 or IPv6
address.
One downside of these convenience functions is that the need to handle
both IPv4 and IPv6 formats means that error messages provide minimal
information on the precise error, as the functions don’t know whether
the IPv4 or IPv6 format was intended. More detailed error reporting can
be obtained by calling the appropriate version specific class
constructors directly.
File: python.info, Node: IP Addresses, Next: IP Network definitions, Prev: Convenience factory functions, Up: ipaddress — IPv4/IPv6 manipulation library
5.22.23.2 IP Addresses
......................
* Menu:
* Address objects::
* Conversion to Strings and Integers::
* Operators: Operators<3>.
File: python.info, Node: Address objects, Next: Conversion to Strings and Integers, Up: IP Addresses
5.22.23.3 Address objects
.........................
The *note IPv4Address: 1fe. and *note IPv6Address: 1ff. objects share a
lot of common attributes. Some attributes that are only meaningful for
IPv6 addresses are also implemented by *note IPv4Address: 1fe. objects,
in order to make it easier to write code that handles both IP versions
correctly. Address objects are *note hashable: 60c, so they can be used
as keys in dictionaries.
-- Class: ipaddress.IPv4Address (address)
Construct an IPv4 address. An *note AddressValueError: 3c46. is
raised if 'address' is not a valid IPv4 address.
The following constitutes a valid IPv4 address:
1. A string in decimal-dot notation, consisting of four decimal
integers in the inclusive range 0–255, separated by dots (e.g.
‘192.168.0.1’). Each integer represents an octet (byte) in
the address. Leading zeroes are not tolerated to prevent
confusion with octal notation.
2. An integer that fits into 32 bits.
3. An integer packed into a *note bytes: 1c2. object of length 4
(most significant octet first).
>>> ipaddress.IPv4Address('192.168.0.1')
IPv4Address('192.168.0.1')
>>> ipaddress.IPv4Address(3232235521)
IPv4Address('192.168.0.1')
>>> ipaddress.IPv4Address(b'\xC0\xA8\x00\x01')
IPv4Address('192.168.0.1')
Changed in version 3.8: Leading zeros are tolerated, even in
ambiguous cases that look like octal notation.
Changed in version 3.9.5: Leading zeros are no longer tolerated and
are treated as an error. IPv4 address strings are now parsed as
strict as glibc *note inet_pton(): 962.
-- Attribute: version
The appropriate version number: ‘4’ for IPv4, ‘6’ for IPv6.
-- Attribute: max_prefixlen
The total number of bits in the address representation for
this version: ‘32’ for IPv4, ‘128’ for IPv6.
The prefix defines the number of leading bits in an address
that are compared to determine whether or not an address is
part of a network.
-- Attribute: compressed
-- Attribute: exploded
The string representation in dotted decimal notation. Leading
zeroes are never included in the representation.
As IPv4 does not define a shorthand notation for addresses
with octets set to zero, these two attributes are always the
same as ‘str(addr)’ for IPv4 addresses. Exposing these
attributes makes it easier to write display code that can
handle both IPv4 and IPv6 addresses.
-- Attribute: packed
The binary representation of this address - a *note bytes:
1c2. object of the appropriate length (most significant octet
first). This is 4 bytes for IPv4 and 16 bytes for IPv6.
-- Attribute: reverse_pointer
The name of the reverse DNS PTR record for the IP address,
e.g.:
>>> ipaddress.ip_address("127.0.0.1").reverse_pointer
'1.0.0.127.in-addr.arpa'
>>> ipaddress.ip_address("2001:db8::1").reverse_pointer
'1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa'
This is the name that could be used for performing a PTR
lookup, not the resolved hostname itself.
Added in version 3.5.
-- Attribute: is_multicast
‘True’ if the address is reserved for multicast use. See RFC
3171(1) (for IPv4) or RFC 2373(2) (for IPv6).
-- Attribute: is_private
‘True’ if the address is defined as not globally reachable by
iana-ipv4-special-registry(3) (for IPv4) or
iana-ipv6-special-registry(4) (for IPv6) with the following
exceptions:
* ‘is_private’ is ‘False’ for the shared address space
(‘100.64.0.0/10’)
* For IPv4-mapped IPv6-addresses the ‘is_private’ value is
determined by the semantics of the underlying IPv4
addresses and the following condition holds (see *note
IPv6Address.ipv4_mapped: 3c4e.):
address.is_private == address.ipv4_mapped.is_private
‘is_private’ has value opposite to *note is_global: f5a,
except for the shared address space (‘100.64.0.0/10’ range)
where they are both ‘False’.
Changed in version 3.13: Fixed some false positives and false
negatives.
* ‘192.0.0.0/24’ is considered private with the exception
of ‘192.0.0.9/32’ and ‘192.0.0.10/32’ (previously: only
the ‘192.0.0.0/29’ sub-range was considered private).
* ‘64:ff9b:1::/48’ is considered private.
* ‘2002::/16’ is considered private.
* There are exceptions within ‘2001::/23’ (otherwise
considered private): ‘2001:1::1/128’, ‘2001:1::2/128’,
‘2001:3::/32’, ‘2001:4:112::/48’, ‘2001:20::/28’,
‘2001:30::/28’. The exceptions are not considered
private.
-- Attribute: is_global
‘True’ if the address is defined as globally reachable by
iana-ipv4-special-registry(5) (for IPv4) or
iana-ipv6-special-registry(6) (for IPv6) with the following
exception:
For IPv4-mapped IPv6-addresses the ‘is_private’ value is
determined by the semantics of the underlying IPv4 addresses
and the following condition holds (see *note
IPv6Address.ipv4_mapped: 3c4e.):
address.is_global == address.ipv4_mapped.is_global
‘is_global’ has value opposite to *note is_private: 1649,
except for the shared address space (‘100.64.0.0/10’ range)
where they are both ‘False’.
Added in version 3.4.
Changed in version 3.13: Fixed some false positives and false
negatives, see *note is_private: 1649. for details.
-- Attribute: is_unspecified
‘True’ if the address is unspecified. See RFC 5735(7) (for
IPv4) or RFC 2373(8) (for IPv6).
-- Attribute: is_reserved
‘True’ if the address is noted as reserved by the IETF. For
IPv4, this is only ‘240.0.0.0/4’, the ‘Reserved’ address
block. For IPv6, this is all addresses allocated(9) as
‘Reserved by IETF’ for future use.
Note: For IPv4, ‘is_reserved’ is not related to the
address block value of the ‘Reserved-by-Protocol’ column
in iana-ipv4-special-registry(10).
Caution: For IPv6, ‘fec0::/10’ a former Site-Local scoped
address prefix is currently excluded from that list (see
*note is_site_local: 3c51. & RFC 3879(11)).
-- Attribute: is_loopback
‘True’ if this is a loopback address. See RFC 3330(12) (for
IPv4) or RFC 2373(13) (for IPv6).
-- Attribute: is_link_local
‘True’ if the address is reserved for link-local usage. See
RFC 3927(14).
-- Attribute: ipv6_mapped
*note IPv4Address: 1fe. object representing the IPv4-mapped
IPv6 address. See RFC 4291(15).
Added in version 3.13.
-- Method: IPv4Address.__format__ (fmt)
Returns a string representation of the IP address, controlled by an
explicit format string. 'fmt' can be one of the following: ‘'s'’,
the default option, equivalent to *note str(): 447, ‘'b'’ for a
zero-padded binary string, ‘'X'’ or ‘'x'’ for an uppercase or
lowercase hexadecimal representation, or ‘'n'’, which is equivalent
to ‘'b'’ for IPv4 addresses and ‘'x'’ for IPv6. For binary and
hexadecimal representations, the form specifier ‘'#'’ and the
grouping option ‘'_'’ are available. ‘__format__’ is used by
‘format’, ‘str.format’ and f-strings.
>>> format(ipaddress.IPv4Address('192.168.0.1'))
'192.168.0.1'
>>> '{:#b}'.format(ipaddress.IPv4Address('192.168.0.1'))
'0b11000000101010000000000000000001'
>>> f'{ipaddress.IPv6Address("2001:db8::1000"):s}'
'2001:db8::1000'
>>> format(ipaddress.IPv6Address('2001:db8::1000'), '_X')
'2001_0DB8_0000_0000_0000_0000_0000_1000'
>>> '{:#_n}'.format(ipaddress.IPv6Address('2001:db8::1000'))
'0x2001_0db8_0000_0000_0000_0000_0000_1000'
Added in version 3.9.
-- Class: ipaddress.IPv6Address (address)
Construct an IPv6 address. An *note AddressValueError: 3c46. is
raised if 'address' is not a valid IPv6 address.
The following constitutes a valid IPv6 address:
1. A string consisting of eight groups of four hexadecimal
digits, each group representing 16 bits. The groups are
separated by colons. This describes an 'exploded' (longhand)
notation. The string can also be 'compressed' (shorthand
notation) by various means. See RFC 4291(16) for details.
For example, ‘"0000:0000:0000:0000:0000:0abc:0007:0def"’ can
be compressed to ‘"::abc:7:def"’.
Optionally, the string may also have a scope zone ID,
expressed with a suffix ‘%scope_id’. If present, the scope ID
must be non-empty, and may not contain ‘%’. See RFC 4007(17)
for details. For example, ‘fe80::1234%1’ might identify
address ‘fe80::1234’ on the first link of the node.
2. An integer that fits into 128 bits.
3. An integer packed into a *note bytes: 1c2. object of length
16, big-endian.
>>> ipaddress.IPv6Address('2001:db8::1000')
IPv6Address('2001:db8::1000')
>>> ipaddress.IPv6Address('ff02::5678%1')
IPv6Address('ff02::5678%1')
-- Attribute: compressed
The short form of the address representation, with leading zeroes
in groups omitted and the longest sequence of groups consisting
entirely of zeroes collapsed to a single empty group.
This is also the value returned by ‘str(addr)’ for IPv6 addresses.
-- Attribute: exploded
The long form of the address representation, with all leading
zeroes and groups consisting entirely of zeroes included.
For the following attributes and methods, see the corresponding
documentation of the *note IPv4Address: 1fe. class:
-- Attribute: packed
-- Attribute: reverse_pointer
-- Attribute: version
-- Attribute: max_prefixlen
-- Attribute: is_multicast
-- Attribute: is_private
-- Attribute: is_global
Added in version 3.4.
-- Attribute: is_unspecified
-- Attribute: is_reserved
-- Attribute: is_loopback
-- Attribute: is_link_local
-- Attribute: is_site_local
‘True’ if the address is reserved for site-local usage. Note
that the site-local address space has been deprecated by RFC
3879(18). Use *note is_private: 1649. to test if this address
is in the space of unique local addresses as defined by RFC
4193(19).
-- Attribute: ipv4_mapped
For addresses that appear to be IPv4 mapped addresses
(starting with ‘::FFFF/96’), this property will report the
embedded IPv4 address. For any other address, this property
will be ‘None’.
-- Attribute: scope_id
For scoped addresses as defined by RFC 4007(20), this property
identifies the particular zone of the address’s scope that the
address belongs to, as a string. When no scope zone is
specified, this property will be ‘None’.
-- Attribute: sixtofour
For addresses that appear to be 6to4 addresses (starting with
‘2002::/16’) as defined by RFC 3056(21), this property will
report the embedded IPv4 address. For any other address, this
property will be ‘None’.
-- Attribute: teredo
For addresses that appear to be Teredo addresses (starting
with ‘2001::/32’) as defined by RFC 4380(22), this property
will report the embedded ‘(server, client)’ IP address pair.
For any other address, this property will be ‘None’.
-- Method: IPv6Address.__format__ (fmt)
Refer to the corresponding method documentation in *note
IPv4Address: 1fe.
Added in version 3.9.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc3171.html
(2) https://datatracker.ietf.org/doc/html/rfc2373.html
(3)
https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
(4)
https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml
(5)
https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
(6)
https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml
(7) https://datatracker.ietf.org/doc/html/rfc5735.html
(8) https://datatracker.ietf.org/doc/html/rfc2373.html
(9)
https://www.iana.org/assignments/ipv6-address-space/ipv6-address-space.xhtml
(10)
https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
(11) https://datatracker.ietf.org/doc/html/rfc3879.html
(12) https://datatracker.ietf.org/doc/html/rfc3330.html
(13) https://datatracker.ietf.org/doc/html/rfc2373.html
(14) https://datatracker.ietf.org/doc/html/rfc3927.html
(15) https://datatracker.ietf.org/doc/html/rfc4291.html
(16) https://datatracker.ietf.org/doc/html/rfc4291.html
(17) https://datatracker.ietf.org/doc/html/rfc4007.html
(18) https://datatracker.ietf.org/doc/html/rfc3879.html
(19) https://datatracker.ietf.org/doc/html/rfc4193.html
(20) https://datatracker.ietf.org/doc/html/rfc4007.html
(21) https://datatracker.ietf.org/doc/html/rfc3056.html
(22) https://datatracker.ietf.org/doc/html/rfc4380.html
File: python.info, Node: Conversion to Strings and Integers, Next: Operators<3>, Prev: Address objects, Up: IP Addresses
5.22.23.4 Conversion to Strings and Integers
............................................
To interoperate with networking interfaces such as the socket module,
addresses must be converted to strings or integers. This is handled
using the *note str(): 447. and *note int(): 259. builtin functions:
>>> str(ipaddress.IPv4Address('192.168.0.1'))
'192.168.0.1'
>>> int(ipaddress.IPv4Address('192.168.0.1'))
3232235521
>>> str(ipaddress.IPv6Address('::1'))
'::1'
>>> int(ipaddress.IPv6Address('::1'))
1
Note that IPv6 scoped addresses are converted to integers without scope
zone ID.
File: python.info, Node: Operators<3>, Prev: Conversion to Strings and Integers, Up: IP Addresses
5.22.23.5 Operators
...................
Address objects support some operators. Unless stated otherwise,
operators can only be applied between compatible objects (i.e. IPv4
with IPv4, IPv6 with IPv6).
* Menu:
* Comparison operators::
* Arithmetic operators::
File: python.info, Node: Comparison operators, Next: Arithmetic operators, Up: Operators<3>
5.22.23.6 Comparison operators
..............................
Address objects can be compared with the usual set of comparison
operators. Same IPv6 addresses with different scope zone IDs are not
equal. Some examples:
>>> IPv4Address('127.0.0.2') > IPv4Address('127.0.0.1')
True
>>> IPv4Address('127.0.0.2') == IPv4Address('127.0.0.1')
False
>>> IPv4Address('127.0.0.2') != IPv4Address('127.0.0.1')
True
>>> IPv6Address('fe80::1234') == IPv6Address('fe80::1234%1')
False
>>> IPv6Address('fe80::1234%1') != IPv6Address('fe80::1234%2')
True
File: python.info, Node: Arithmetic operators, Prev: Comparison operators, Up: Operators<3>
5.22.23.7 Arithmetic operators
..............................
Integers can be added to or subtracted from address objects. Some
examples:
>>> IPv4Address('127.0.0.2') + 3
IPv4Address('127.0.0.5')
>>> IPv4Address('127.0.0.2') - 3
IPv4Address('126.255.255.255')
>>> IPv4Address('255.255.255.255') + 1
Traceback (most recent call last):
File "", line 1, in
ipaddress.AddressValueError: 4294967296 (>= 2**32) is not permitted as an IPv4 address
File: python.info, Node: IP Network definitions, Next: Interface objects, Prev: IP Addresses, Up: ipaddress — IPv4/IPv6 manipulation library
5.22.23.8 IP Network definitions
................................
The *note IPv4Network: 200. and *note IPv6Network: 201. objects provide
a mechanism for defining and inspecting IP network definitions. A
network definition consists of a 'mask' and a 'network address', and as
such defines a range of IP addresses that equal the network address when
masked (binary AND) with the mask. For example, a network definition
with the mask ‘255.255.255.0’ and the network address ‘192.168.1.0’
consists of IP addresses in the inclusive range ‘192.168.1.0’ to
‘192.168.1.255’.
* Menu:
* Prefix, net mask and host mask: Prefix net mask and host mask.
* Network objects::
* Operators: Operators<4>.
File: python.info, Node: Prefix net mask and host mask, Next: Network objects, Up: IP Network definitions
5.22.23.9 Prefix, net mask and host mask
........................................
There are several equivalent ways to specify IP network masks. A
'prefix' ‘/’ is a notation that denotes how many high-order bits
are set in the network mask. A 'net mask' is an IP address with some
number of high-order bits set. Thus the prefix ‘/24’ is equivalent to
the net mask ‘255.255.255.0’ in IPv4, or ‘ffff:ff00::’ in IPv6. In
addition, a 'host mask' is the logical inverse of a 'net mask', and is
sometimes used (for example in Cisco access control lists) to denote a
network mask. The host mask equivalent to ‘/24’ in IPv4 is ‘0.0.0.255’.
File: python.info, Node: Network objects, Next: Operators<4>, Prev: Prefix net mask and host mask, Up: IP Network definitions
5.22.23.10 Network objects
..........................
All attributes implemented by address objects are implemented by network
objects as well. In addition, network objects implement additional
attributes. All of these are common between *note IPv4Network: 200. and
*note IPv6Network: 201, so to avoid duplication they are only documented
for *note IPv4Network: 200. Network objects are *note hashable: 60c, so
they can be used as keys in dictionaries.
-- Class: ipaddress.IPv4Network (address, strict=True)
Construct an IPv4 network definition. 'address' can be one of the
following:
1. A string consisting of an IP address and an optional mask,
separated by a slash (‘/’). The IP address is the network
address, and the mask can be either a single number, which
means it’s a 'prefix', or a string representation of an IPv4
address. If it’s the latter, the mask is interpreted as a
'net mask' if it starts with a non-zero field, or as a 'host
mask' if it starts with a zero field, with the single
exception of an all-zero mask which is treated as a 'net
mask'. If no mask is provided, it’s considered to be ‘/32’.
For example, the following 'address' specifications are
equivalent: ‘192.168.1.0/24’, ‘192.168.1.0/255.255.255.0’ and
‘192.168.1.0/0.0.0.255’.
2. An integer that fits into 32 bits. This is equivalent to a
single-address network, with the network address being
'address' and the mask being ‘/32’.
3. An integer packed into a *note bytes: 1c2. object of length 4,
big-endian. The interpretation is similar to an integer
'address'.
4. A two-tuple of an address description and a netmask, where the
address description is either a string, a 32-bits integer, a
4-bytes packed integer, or an existing IPv4Address object; and
the netmask is either an integer representing the prefix
length (e.g. ‘24’) or a string representing the prefix mask
(e.g. ‘255.255.255.0’).
An *note AddressValueError: 3c46. is raised if 'address' is not a
valid IPv4 address. A *note NetmaskValueError: 3c68. is raised if
the mask is not valid for an IPv4 address.
If 'strict' is ‘True’ and host bits are set in the supplied
address, then *note ValueError: 204. is raised. Otherwise, the
host bits are masked out to determine the appropriate network
address.
Unless stated otherwise, all network methods accepting other
network/address objects will raise *note TypeError: 534. if the
argument’s IP version is incompatible to ‘self’.
Changed in version 3.5: Added the two-tuple form for the 'address'
constructor parameter.
-- Attribute: version
-- Attribute: max_prefixlen
Refer to the corresponding attribute documentation in *note
IPv4Address: 1fe.
-- Attribute: is_multicast
-- Attribute: is_private
-- Attribute: is_unspecified
-- Attribute: is_reserved
-- Attribute: is_loopback
-- Attribute: is_link_local
These attributes are true for the network as a whole if they
are true for both the network address and the broadcast
address.
-- Attribute: network_address
The network address for the network. The network address and
the prefix length together uniquely define a network.
-- Attribute: broadcast_address
The broadcast address for the network. Packets sent to the
broadcast address should be received by every host on the
network.
-- Attribute: hostmask
The host mask, as an *note IPv4Address: 1fe. object.
-- Attribute: netmask
The net mask, as an *note IPv4Address: 1fe. object.
-- Attribute: with_prefixlen
-- Attribute: compressed
-- Attribute: exploded
A string representation of the network, with the mask in
prefix notation.
‘with_prefixlen’ and ‘compressed’ are always the same as
‘str(network)’. ‘exploded’ uses the exploded form the network
address.
-- Attribute: with_netmask
A string representation of the network, with the mask in net
mask notation.
-- Attribute: with_hostmask
A string representation of the network, with the mask in host
mask notation.
-- Attribute: num_addresses
The total number of addresses in the network.
-- Attribute: prefixlen
Length of the network prefix, in bits.
-- Method: hosts ()
Returns an iterator over the usable hosts in the network. The
usable hosts are all the IP addresses that belong to the
network, except the network address itself and the network
broadcast address. For networks with a mask length of 31, the
network address and network broadcast address are also
included in the result. Networks with a mask of 32 will
return a list containing the single host address.
>>> list(ip_network('192.0.2.0/29').hosts())
[IPv4Address('192.0.2.1'), IPv4Address('192.0.2.2'),
IPv4Address('192.0.2.3'), IPv4Address('192.0.2.4'),
IPv4Address('192.0.2.5'), IPv4Address('192.0.2.6')]
>>> list(ip_network('192.0.2.0/31').hosts())
[IPv4Address('192.0.2.0'), IPv4Address('192.0.2.1')]
>>> list(ip_network('192.0.2.1/32').hosts())
[IPv4Address('192.0.2.1')]
-- Method: overlaps (other)
‘True’ if this network is partly or wholly contained in
'other' or 'other' is wholly contained in this network.
-- Method: address_exclude (network)
Computes the network definitions resulting from removing the
given 'network' from this one. Returns an iterator of network
objects. Raises *note ValueError: 204. if 'network' is not
completely contained in this network.
>>> n1 = ip_network('192.0.2.0/28')
>>> n2 = ip_network('192.0.2.1/32')
>>> list(n1.address_exclude(n2))
[IPv4Network('192.0.2.8/29'), IPv4Network('192.0.2.4/30'),
IPv4Network('192.0.2.2/31'), IPv4Network('192.0.2.0/32')]
-- Method: subnets (prefixlen_diff=1, new_prefix=None)
The subnets that join to make the current network definition,
depending on the argument values. 'prefixlen_diff' is the
amount our prefix length should be increased by. 'new_prefix'
is the desired new prefix of the subnets; it must be larger
than our prefix. One and only one of 'prefixlen_diff' and
'new_prefix' must be set. Returns an iterator of network
objects.
>>> list(ip_network('192.0.2.0/24').subnets())
[IPv4Network('192.0.2.0/25'), IPv4Network('192.0.2.128/25')]
>>> list(ip_network('192.0.2.0/24').subnets(prefixlen_diff=2))
[IPv4Network('192.0.2.0/26'), IPv4Network('192.0.2.64/26'),
IPv4Network('192.0.2.128/26'), IPv4Network('192.0.2.192/26')]
>>> list(ip_network('192.0.2.0/24').subnets(new_prefix=26))
[IPv4Network('192.0.2.0/26'), IPv4Network('192.0.2.64/26'),
IPv4Network('192.0.2.128/26'), IPv4Network('192.0.2.192/26')]
>>> list(ip_network('192.0.2.0/24').subnets(new_prefix=23))
Traceback (most recent call last):
File "", line 1, in
raise ValueError('new prefix must be longer')
ValueError: new prefix must be longer
>>> list(ip_network('192.0.2.0/24').subnets(new_prefix=25))
[IPv4Network('192.0.2.0/25'), IPv4Network('192.0.2.128/25')]
-- Method: supernet (prefixlen_diff=1, new_prefix=None)
The supernet containing this network definition, depending on
the argument values. 'prefixlen_diff' is the amount our
prefix length should be decreased by. 'new_prefix' is the
desired new prefix of the supernet; it must be smaller than
our prefix. One and only one of 'prefixlen_diff' and
'new_prefix' must be set. Returns a single network object.
>>> ip_network('192.0.2.0/24').supernet()
IPv4Network('192.0.2.0/23')
>>> ip_network('192.0.2.0/24').supernet(prefixlen_diff=2)
IPv4Network('192.0.0.0/22')
>>> ip_network('192.0.2.0/24').supernet(new_prefix=20)
IPv4Network('192.0.0.0/20')
-- Method: subnet_of (other)
Return ‘True’ if this network is a subnet of 'other'.
>>> a = ip_network('192.168.1.0/24')
>>> b = ip_network('192.168.1.128/30')
>>> b.subnet_of(a)
True
Added in version 3.7.
-- Method: supernet_of (other)
Return ‘True’ if this network is a supernet of 'other'.
>>> a = ip_network('192.168.1.0/24')
>>> b = ip_network('192.168.1.128/30')
>>> a.supernet_of(b)
True
Added in version 3.7.
-- Method: compare_networks (other)
Compare this network to 'other'. In this comparison only the
network addresses are considered; host bits aren’t. Returns
either ‘-1’, ‘0’ or ‘1’.
>>> ip_network('192.0.2.1/32').compare_networks(ip_network('192.0.2.2/32'))
-1
>>> ip_network('192.0.2.1/32').compare_networks(ip_network('192.0.2.0/32'))
1
>>> ip_network('192.0.2.1/32').compare_networks(ip_network('192.0.2.1/32'))
0
Deprecated since version 3.7: It uses the same ordering and
comparison algorithm as “<”, “==”, and “>”
-- Class: ipaddress.IPv6Network (address, strict=True)
Construct an IPv6 network definition. 'address' can be one of the
following:
1. A string consisting of an IP address and an optional prefix
length, separated by a slash (‘/’). The IP address is the
network address, and the prefix length must be a single
number, the 'prefix'. If no prefix length is provided, it’s
considered to be ‘/128’.
Note that currently expanded netmasks are not supported. That
means ‘2001:db00::0/24’ is a valid argument while
‘2001:db00::0/ffff:ff00::’ is not.
2. An integer that fits into 128 bits. This is equivalent to a
single-address network, with the network address being
'address' and the mask being ‘/128’.
3. An integer packed into a *note bytes: 1c2. object of length
16, big-endian. The interpretation is similar to an integer
'address'.
4. A two-tuple of an address description and a netmask, where the
address description is either a string, a 128-bits integer, a
16-bytes packed integer, or an existing IPv6Address object;
and the netmask is an integer representing the prefix length.
An *note AddressValueError: 3c46. is raised if 'address' is not a
valid IPv6 address. A *note NetmaskValueError: 3c68. is raised if
the mask is not valid for an IPv6 address.
If 'strict' is ‘True’ and host bits are set in the supplied
address, then *note ValueError: 204. is raised. Otherwise, the
host bits are masked out to determine the appropriate network
address.
Changed in version 3.5: Added the two-tuple form for the 'address'
constructor parameter.
-- Attribute: version
-- Attribute: max_prefixlen
-- Attribute: is_multicast
-- Attribute: is_private
-- Attribute: is_unspecified
-- Attribute: is_reserved
-- Attribute: is_loopback
-- Attribute: is_link_local
-- Attribute: network_address
-- Attribute: broadcast_address
-- Attribute: hostmask
-- Attribute: netmask
-- Attribute: with_prefixlen
-- Attribute: compressed
-- Attribute: exploded
-- Attribute: with_netmask
-- Attribute: with_hostmask
-- Attribute: num_addresses
-- Attribute: prefixlen
-- Method: hosts ()
Returns an iterator over the usable hosts in the network. The
usable hosts are all the IP addresses that belong to the
network, except the Subnet-Router anycast address. For
networks with a mask length of 127, the Subnet-Router anycast
address is also included in the result. Networks with a mask
of 128 will return a list containing the single host address.
-- Method: overlaps (other)
-- Method: address_exclude (network)
-- Method: subnets (prefixlen_diff=1, new_prefix=None)
-- Method: supernet (prefixlen_diff=1, new_prefix=None)
-- Method: subnet_of (other)
-- Method: supernet_of (other)
-- Method: compare_networks (other)
Refer to the corresponding attribute documentation in *note
IPv4Network: 200.
-- Attribute: is_site_local
These attribute is true for the network as a whole if it is
true for both the network address and the broadcast address.
File: python.info, Node: Operators<4>, Prev: Network objects, Up: IP Network definitions
5.22.23.11 Operators
....................
Network objects support some operators. Unless stated otherwise,
operators can only be applied between compatible objects (i.e. IPv4
with IPv4, IPv6 with IPv6).
* Menu:
* Logical operators::
* Iteration::
* Networks as containers of addresses::
File: python.info, Node: Logical operators, Next: Iteration, Up: Operators<4>
5.22.23.12 Logical operators
............................
Network objects can be compared with the usual set of logical operators.
Network objects are ordered first by network address, then by net mask.
File: python.info, Node: Iteration, Next: Networks as containers of addresses, Prev: Logical operators, Up: Operators<4>
5.22.23.13 Iteration
....................
Network objects can be iterated to list all the addresses belonging to
the network. For iteration, 'all' hosts are returned, including
unusable hosts (for usable hosts, use the *note hosts(): 3c7c. method).
An example:
>>> for addr in IPv4Network('192.0.2.0/28'):
... addr
...
IPv4Address('192.0.2.0')
IPv4Address('192.0.2.1')
IPv4Address('192.0.2.2')
IPv4Address('192.0.2.3')
IPv4Address('192.0.2.4')
IPv4Address('192.0.2.5')
IPv4Address('192.0.2.6')
IPv4Address('192.0.2.7')
IPv4Address('192.0.2.8')
IPv4Address('192.0.2.9')
IPv4Address('192.0.2.10')
IPv4Address('192.0.2.11')
IPv4Address('192.0.2.12')
IPv4Address('192.0.2.13')
IPv4Address('192.0.2.14')
IPv4Address('192.0.2.15')
File: python.info, Node: Networks as containers of addresses, Prev: Iteration, Up: Operators<4>
5.22.23.14 Networks as containers of addresses
..............................................
Network objects can act as containers of addresses. Some examples:
>>> IPv4Network('192.0.2.0/28')[0]
IPv4Address('192.0.2.0')
>>> IPv4Network('192.0.2.0/28')[15]
IPv4Address('192.0.2.15')
>>> IPv4Address('192.0.2.6') in IPv4Network('192.0.2.0/28')
True
>>> IPv4Address('192.0.3.6') in IPv4Network('192.0.2.0/28')
False
File: python.info, Node: Interface objects, Next: Other Module Level Functions, Prev: IP Network definitions, Up: ipaddress — IPv4/IPv6 manipulation library
5.22.23.15 Interface objects
............................
Interface objects are *note hashable: 60c, so they can be used as keys
in dictionaries.
-- Class: ipaddress.IPv4Interface (address)
Construct an IPv4 interface. The meaning of 'address' is as in the
constructor of *note IPv4Network: 200, except that arbitrary host
addresses are always accepted.
*note IPv4Interface: 180d. is a subclass of *note IPv4Address: 1fe,
so it inherits all the attributes from that class. In addition,
the following attributes are available:
-- Attribute: ip
The address (*note IPv4Address: 1fe.) without network
information.
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.ip
IPv4Address('192.0.2.5')
-- Attribute: network
The network (*note IPv4Network: 200.) this interface belongs
to.
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.network
IPv4Network('192.0.2.0/24')
-- Attribute: with_prefixlen
A string representation of the interface with the mask in
prefix notation.
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.with_prefixlen
'192.0.2.5/24'
-- Attribute: with_netmask
A string representation of the interface with the network as a
net mask.
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.with_netmask
'192.0.2.5/255.255.255.0'
-- Attribute: with_hostmask
A string representation of the interface with the network as a
host mask.
>>> interface = IPv4Interface('192.0.2.5/24')
>>> interface.with_hostmask
'192.0.2.5/0.0.0.255'
-- Class: ipaddress.IPv6Interface (address)
Construct an IPv6 interface. The meaning of 'address' is as in the
constructor of *note IPv6Network: 201, except that arbitrary host
addresses are always accepted.
*note IPv6Interface: 180e. is a subclass of *note IPv6Address: 1ff,
so it inherits all the attributes from that class. In addition,
the following attributes are available:
-- Attribute: ip
-- Attribute: network
-- Attribute: with_prefixlen
-- Attribute: with_netmask
-- Attribute: with_hostmask
Refer to the corresponding attribute documentation in *note
IPv4Interface: 180d.
* Menu:
* Operators: Operators<5>.
File: python.info, Node: Operators<5>, Up: Interface objects
5.22.23.16 Operators
....................
Interface objects support some operators. Unless stated otherwise,
operators can only be applied between compatible objects (i.e. IPv4
with IPv4, IPv6 with IPv6).
* Menu:
* Logical operators: Logical operators<2>.
File: python.info, Node: Logical operators<2>, Up: Operators<5>
5.22.23.17 Logical operators
............................
Interface objects can be compared with the usual set of logical
operators.
For equality comparison (‘==’ and ‘!=’), both the IP address and network
must be the same for the objects to be equal. An interface will not
compare equal to any address or network object.
For ordering (‘<’, ‘>’, etc) the rules are different. Interface and
address objects with the same IP version can be compared, and the
address objects will always sort before the interface objects. Two
interface objects are first compared by their networks and, if those are
the same, then by their IP addresses.
File: python.info, Node: Other Module Level Functions, Next: Custom Exceptions, Prev: Interface objects, Up: ipaddress — IPv4/IPv6 manipulation library
5.22.23.18 Other Module Level Functions
.......................................
The module also provides the following module level functions:
-- Function: ipaddress.v4_int_to_packed (address)
Represent an address as 4 packed bytes in network (big-endian)
order. 'address' is an integer representation of an IPv4 IP
address. A *note ValueError: 204. is raised if the integer is
negative or too large to be an IPv4 IP address.
>>> ipaddress.ip_address(3221225985)
IPv4Address('192.0.2.1')
>>> ipaddress.v4_int_to_packed(3221225985)
b'\xc0\x00\x02\x01'
-- Function: ipaddress.v6_int_to_packed (address)
Represent an address as 16 packed bytes in network (big-endian)
order. 'address' is an integer representation of an IPv6 IP
address. A *note ValueError: 204. is raised if the integer is
negative or too large to be an IPv6 IP address.
-- Function: ipaddress.summarize_address_range (first, last)
Return an iterator of the summarized network range given the first
and last IP addresses. 'first' is the first *note IPv4Address:
1fe. or *note IPv6Address: 1ff. in the range and 'last' is the last
*note IPv4Address: 1fe. or *note IPv6Address: 1ff. in the range. A
*note TypeError: 534. is raised if 'first' or 'last' are not IP
addresses or are not of the same version. A *note ValueError: 204.
is raised if 'last' is not greater than 'first' or if 'first'
address version is not 4 or 6.
>>> [ipaddr for ipaddr in ipaddress.summarize_address_range(
... ipaddress.IPv4Address('192.0.2.0'),
... ipaddress.IPv4Address('192.0.2.130'))]
[IPv4Network('192.0.2.0/25'), IPv4Network('192.0.2.128/31'), IPv4Network('192.0.2.130/32')]
-- Function: ipaddress.collapse_addresses (addresses)
Return an iterator of the collapsed *note IPv4Network: 200. or
*note IPv6Network: 201. objects. 'addresses' is an *note iterable:
121a. of *note IPv4Network: 200. or *note IPv6Network: 201.
objects. A *note TypeError: 534. is raised if 'addresses' contains
mixed version objects.
>>> [ipaddr for ipaddr in
... ipaddress.collapse_addresses([ipaddress.IPv4Network('192.0.2.0/25'),
... ipaddress.IPv4Network('192.0.2.128/25')])]
[IPv4Network('192.0.2.0/24')]
-- Function: ipaddress.get_mixed_type_key (obj)
Return a key suitable for sorting between networks and addresses.
Address and Network objects are not sortable by default; they’re
fundamentally different, so the expression:
IPv4Address('192.0.2.0') <= IPv4Network('192.0.2.0/24')
doesn’t make sense. There are some times however, where you may
wish to have *note ipaddress: 80. sort these anyway. If you need
to do this, you can use this function as the 'key' argument to
*note sorted(): bce.
'obj' is either a network or address object.
File: python.info, Node: Custom Exceptions, Prev: Other Module Level Functions, Up: ipaddress — IPv4/IPv6 manipulation library
5.22.23.19 Custom Exceptions
............................
To support more specific error reporting from class constructors, the
module defines the following exceptions:
-- Exception: ipaddress.AddressValueError (ValueError)
Any value error related to the address.
-- Exception: ipaddress.NetmaskValueError (ValueError)
Any value error related to the net mask.
File: python.info, Node: Multimedia Services, Next: Internationalization, Prev: Internet Protocols and Support, Up: The Python Standard Library
5.23 Multimedia Services
========================
The modules described in this chapter implement various algorithms or
interfaces that are mainly useful for multimedia applications. They are
available at the discretion of the installation. Here’s an overview:
* Menu:
* wave — Read and write WAV files::
* colorsys — Conversions between color systems::
File: python.info, Node: wave — Read and write WAV files, Next: colorsys — Conversions between color systems, Up: Multimedia Services
5.23.1 ‘wave’ — Read and write WAV files
----------------------------------------
'Source code:' Lib/wave.py(1)
__________________________________________________________________
The *note wave: 113. module provides a convenient interface to the
Waveform Audio “WAVE” (or “WAV”) file format. Only uncompressed PCM
encoded wave files are supported.
Changed in version 3.12: Support for ‘WAVE_FORMAT_EXTENSIBLE’ headers
was added, provided that the extended format is
‘KSDATAFORMAT_SUBTYPE_PCM’.
The *note wave: 113. module defines the following function and
exception:
-- Function: wave.open (file, mode=None)
If 'file' is a string, open the file by that name, otherwise treat
it as a file-like object. 'mode' can be:
‘'rb'’
Read only mode.
‘'wb'’
Write only mode.
Note that it does not allow read/write WAV files.
A 'mode' of ‘'rb'’ returns a *note Wave_read: 2b8. object, while a
'mode' of ‘'wb'’ returns a *note Wave_write: 2b9. object. If
'mode' is omitted and a file-like object is passed as 'file',
‘file.mode’ is used as the default value for 'mode'.
If you pass in a file-like object, the wave object will not close
it when its ‘close()’ method is called; it is the caller’s
responsibility to close the file object.
The *note open(): 94b. function may be used in a *note with: 5ce.
statement. When the ‘with’ block completes, the *note
Wave_read.close(): 3cb9. or *note Wave_write.close(): 3cba. method
is called.
Changed in version 3.4: Added support for unseekable files.
-- Exception: wave.Error
An error raised when something is impossible because it violates
the WAV specification or hits an implementation deficiency.
* Menu:
* Wave_read Objects::
* Wave_write Objects::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/wave.py
File: python.info, Node: Wave_read Objects, Next: Wave_write Objects, Up: wave — Read and write WAV files
5.23.1.1 Wave_read Objects
..........................
-- Class: wave.Wave_read
Read a WAV file.
Wave_read objects, as returned by *note open(): 94b, have the
following methods:
-- Method: close ()
Close the stream if it was opened by *note wave: 113, and make
the instance unusable. This is called automatically on object
collection.
-- Method: getnchannels ()
Returns number of audio channels (‘1’ for mono, ‘2’ for
stereo).
-- Method: getsampwidth ()
Returns sample width in bytes.
-- Method: getframerate ()
Returns sampling frequency.
-- Method: getnframes ()
Returns number of audio frames.
-- Method: getcomptype ()
Returns compression type (‘'NONE'’ is the only supported
type).
-- Method: getcompname ()
Human-readable version of *note getcomptype(): 3cc2. Usually
‘'not compressed'’ parallels ‘'NONE'’.
-- Method: getparams ()
Returns a *note namedtuple(): 1ca. ‘(nchannels, sampwidth,
framerate, nframes, comptype, compname)’, equivalent to output
of the ‘get*()’ methods.
-- Method: readframes (n)
Reads and returns at most 'n' frames of audio, as a *note
bytes: 1c2. object.
-- Method: rewind ()
Rewind the file pointer to the beginning of the audio stream.
The following two methods are defined for compatibility with the
old ‘aifc’ module, and don’t do anything interesting.
-- Method: getmarkers ()
Returns ‘None’.
Deprecated since version 3.13, will be removed in version
3.15: The method only existed for compatibility with the
‘aifc’ module which has been removed in Python 3.13.
-- Method: getmark (id)
Raise an error.
Deprecated since version 3.13, will be removed in version
3.15: The method only existed for compatibility with the
‘aifc’ module which has been removed in Python 3.13.
The following two methods define a term “position” which is
compatible between them, and is otherwise implementation dependent.
-- Method: setpos (pos)
Set the file pointer to the specified position.
-- Method: tell ()
Return current file pointer position.
File: python.info, Node: Wave_write Objects, Prev: Wave_read Objects, Up: wave — Read and write WAV files
5.23.1.2 Wave_write Objects
...........................
-- Class: wave.Wave_write
Write a WAV file.
Wave_write objects, as returned by *note open(): 94b.
For seekable output streams, the ‘wave’ header will automatically
be updated to reflect the number of frames actually written. For
unseekable streams, the 'nframes' value must be accurate when the
first frame data is written. An accurate 'nframes' value can be
achieved either by calling *note setnframes(): 3cca. or *note
setparams(): 3ccb. with the number of frames that will be written
before *note close(): 3cba. is called and then using *note
writeframesraw(): fe5. to write the frame data, or by calling *note
writeframes(): fe6. with all of the frame data to be written. In
the latter case *note writeframes(): fe6. will calculate the number
of frames in the data and set 'nframes' accordingly before writing
the frame data.
Changed in version 3.4: Added support for unseekable files.
Wave_write objects have the following methods:
-- Method: close ()
Make sure 'nframes' is correct, and close the file if it was
opened by *note wave: 113. This method is called upon object
collection. It will raise an exception if the output stream
is not seekable and 'nframes' does not match the number of
frames actually written.
-- Method: setnchannels (n)
Set the number of channels.
-- Method: setsampwidth (n)
Set the sample width to 'n' bytes.
-- Method: setframerate (n)
Set the frame rate to 'n'.
Changed in version 3.2: A non-integral input to this method is
rounded to the nearest integer.
-- Method: setnframes (n)
Set the number of frames to 'n'. This will be changed later
if the number of frames actually written is different (this
update attempt will raise an error if the output stream is not
seekable).
-- Method: setcomptype (type, name)
Set the compression type and description. At the moment, only
compression type ‘NONE’ is supported, meaning no compression.
-- Method: setparams (tuple)
The 'tuple' should be ‘(nchannels, sampwidth, framerate,
nframes, comptype, compname)’, with values valid for the
‘set*()’ methods. Sets all parameters.
-- Method: tell ()
Return current position in the file, with the same disclaimer
for the *note Wave_read.tell(): 3cc8. and *note
Wave_read.setpos(): 3cc7. methods.
-- Method: writeframesraw (data)
Write audio frames, without correcting 'nframes'.
Changed in version 3.4: Any *note bytes-like object: d2d. is
now accepted.
-- Method: writeframes (data)
Write audio frames and make sure 'nframes' is correct. It
will raise an error if the output stream is not seekable and
the total number of frames that have been written after 'data'
has been written does not match the previously set value for
'nframes'.
Changed in version 3.4: Any *note bytes-like object: d2d. is
now accepted.
Note that it is invalid to set any parameters after calling
*note writeframes(): fe6. or *note writeframesraw(): fe5, and
any attempt to do so will raise *note wave.Error: 3cbb.
File: python.info, Node: colorsys — Conversions between color systems, Prev: wave — Read and write WAV files, Up: Multimedia Services
5.23.2 ‘colorsys’ — Conversions between color systems
-----------------------------------------------------
'Source code:' Lib/colorsys.py(1)
__________________________________________________________________
The *note colorsys: 1f. module defines bidirectional conversions of
color values between colors expressed in the RGB (Red Green Blue) color
space used in computer monitors and three other coordinate systems: YIQ,
HLS (Hue Lightness Saturation) and HSV (Hue Saturation Value).
Coordinates in all of these color spaces are floating-point values. In
the YIQ space, the Y coordinate is between 0 and 1, but the I and Q
coordinates can be positive or negative. In all other spaces, the
coordinates are all between 0 and 1.
See also
........
More information about color spaces can be found at
‘https://poynton.ca/ColorFAQ.html’ and
‘https://www.cambridgeincolour.com/tutorials/color-spaces.htm’.
The *note colorsys: 1f. module defines the following functions:
-- Function: colorsys.rgb_to_yiq (r, g, b)
Convert the color from RGB coordinates to YIQ coordinates.
-- Function: colorsys.yiq_to_rgb (y, i, q)
Convert the color from YIQ coordinates to RGB coordinates.
-- Function: colorsys.rgb_to_hls (r, g, b)
Convert the color from RGB coordinates to HLS coordinates.
-- Function: colorsys.hls_to_rgb (h, l, s)
Convert the color from HLS coordinates to RGB coordinates.
-- Function: colorsys.rgb_to_hsv (r, g, b)
Convert the color from RGB coordinates to HSV coordinates.
-- Function: colorsys.hsv_to_rgb (h, s, v)
Convert the color from HSV coordinates to RGB coordinates.
Example:
>>> import colorsys
>>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4)
(0.5, 0.5, 0.4)
>>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4)
(0.2, 0.4, 0.4)
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/colorsys.py
File: python.info, Node: Internationalization, Next: Program Frameworks, Prev: Multimedia Services, Up: The Python Standard Library
5.24 Internationalization
=========================
The modules described in this chapter help you write software that is
independent of language and locale by providing mechanisms for selecting
a language to be used in program messages or by tailoring output to
match local conventions.
The list of modules described in this chapter is:
* Menu:
* gettext — Multilingual internationalization services::
* locale — Internationalization services::
File: python.info, Node: gettext — Multilingual internationalization services, Next: locale — Internationalization services, Up: Internationalization
5.24.1 ‘gettext’ — Multilingual internationalization services
-------------------------------------------------------------
'Source code:' Lib/gettext.py(1)
__________________________________________________________________
The *note gettext: 63. module provides internationalization (I18N) and
localization (L10N) services for your Python modules and applications.
It supports both the GNU ‘gettext’ message catalog API and a higher
level, class-based API that may be more appropriate for Python files.
The interface described below allows you to write your module and
application messages in one natural language, and provide a catalog of
translated messages for running under different natural languages.
Some hints on localizing your Python modules and applications are also
given.
* Menu:
* GNU gettext API::
* Class-based API::
* Internationalizing your programs and modules::
* Acknowledgements: Acknowledgements<9>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/gettext.py
File: python.info, Node: GNU gettext API, Next: Class-based API, Up: gettext — Multilingual internationalization services
5.24.1.1 GNU ‘gettext’ API
..........................
The *note gettext: 63. module defines the following API, which is very
similar to the GNU ‘gettext’ API. If you use this API you will affect
the translation of your entire application globally. Often this is what
you want if your application is monolingual, with the choice of language
dependent on the locale of your user. If you are localizing a Python
module, or if your application needs to switch languages on the fly, you
probably want to use the class-based API instead.
-- Function: gettext.bindtextdomain (domain, localedir=None)
Bind the 'domain' to the locale directory 'localedir'. More
concretely, *note gettext: 63. will look for binary ‘.mo’ files for
the given domain using the path (on Unix):
‘`localedir'/`language'/LC_MESSAGES/`domain'.mo’, where 'language'
is searched for in the environment variables ‘LANGUAGE’, ‘LC_ALL’,
‘LC_MESSAGES’, and ‘LANG’ respectively.
If 'localedir' is omitted or ‘None’, then the current binding for
'domain' is returned. (1)
-- Function: gettext.textdomain (domain=None)
Change or query the current global domain. If 'domain' is ‘None’,
then the current global domain is returned, otherwise the global
domain is set to 'domain', which is returned.
-- Function: gettext.gettext (message)
Return the localized translation of 'message', based on the current
global domain, language, and locale directory. This function is
usually aliased as ‘_()’ in the local namespace (see examples
below).
-- Function: gettext.dgettext (domain, message)
Like *note gettext(): 1432, but look the message up in the
specified 'domain'.
-- Function: gettext.ngettext (singular, plural, n)
Like *note gettext(): 1432, but consider plural forms. If a
translation is found, apply the plural formula to 'n', and return
the resulting message (some languages have more than two plural
forms). If no translation is found, return 'singular' if 'n' is 1;
return 'plural' otherwise.
The Plural formula is taken from the catalog header. It is a C or
Python expression that has a free variable 'n'; the expression
evaluates to the index of the plural in the catalog. See the GNU
gettext documentation(2) for the precise syntax to be used in ‘.po’
files and the formulas for a variety of languages.
-- Function: gettext.dngettext (domain, singular, plural, n)
Like *note ngettext(): 3ce1, but look the message up in the
specified 'domain'.
-- Function: gettext.pgettext (context, message)
-- Function: gettext.dpgettext (domain, context, message)
-- Function: gettext.npgettext (context, singular, plural, n)
-- Function: gettext.dnpgettext (domain, context, singular, plural, n)
Similar to the corresponding functions without the ‘p’ in the
prefix (that is, *note gettext(): 63, *note dgettext(): 3ce0, *note
ngettext(): 3ce1, *note dngettext(): 3ce2.), but the translation is
restricted to the given message 'context'.
Added in version 3.8.
Note that GNU ‘gettext’ also defines a ‘dcgettext()’ method, but this
was deemed not useful and so it is currently unimplemented.
Here’s an example of typical usage for this API:
import gettext
gettext.bindtextdomain('myapplication', '/path/to/my/language/directory')
gettext.textdomain('myapplication')
_ = gettext.gettext
# ...
print(_('This is a translatable string.'))
---------- Footnotes ----------
(1) The default locale directory is system dependent; for example, on
Red Hat Linux it is ‘/usr/share/locale’, but on Solaris it is
‘/usr/lib/locale’. The ‘gettext’ module does not try to support these
system dependent defaults; instead its default is
‘`sys.base_prefix'/share/locale’ (see *note sys.base_prefix: 3eb.). For
this reason, it is always best to call *note bindtextdomain(): 3cde.
with an explicit absolute path at the start of your application.
(2) https://www.gnu.org/software/gettext/manual/gettext.html
File: python.info, Node: Class-based API, Next: Internationalizing your programs and modules, Prev: GNU gettext API, Up: gettext — Multilingual internationalization services
5.24.1.2 Class-based API
........................
The class-based API of the *note gettext: 63. module gives you more
flexibility and greater convenience than the GNU ‘gettext’ API. It is
the recommended way of localizing your Python applications and modules.
‘gettext’ defines a *note GNUTranslations: 3ce7. class which implements
the parsing of GNU ‘.mo’ format files, and has methods for returning
strings. Instances of this class can also install themselves in the
built-in namespace as the function ‘_()’.
-- Function: gettext.find (domain, localedir=None, languages=None,
all=False)
This function implements the standard ‘.mo’ file search algorithm.
It takes a 'domain', identical to what *note textdomain(): 3cdf.
takes. Optional 'localedir' is as in *note bindtextdomain(): 3cde.
Optional 'languages' is a list of strings, where each string is a
language code.
If 'localedir' is not given, then the default system locale
directory is used. (1) If 'languages' is not given, then the
following environment variables are searched: ‘LANGUAGE’, ‘LC_ALL’,
‘LC_MESSAGES’, and ‘LANG’. The first one returning a non-empty
value is used for the 'languages' variable. The environment
variables should contain a colon separated list of languages, which
will be split on the colon to produce the expected list of language
code strings.
*note find(): 3ce8. then expands and normalizes the languages, and
then iterates through them, searching for an existing file built of
these components:
‘`localedir'/`language'/LC_MESSAGES/`domain'.mo’
The first such file name that exists is returned by *note find():
3ce8. If no such file is found, then ‘None’ is returned. If 'all'
is given, it returns a list of all file names, in the order in
which they appear in the languages list or the environment
variables.
-- Function: gettext.translation (domain, localedir=None,
languages=None, class_=None, fallback=False)
Return a ‘*Translations’ instance based on the 'domain',
'localedir', and 'languages', which are first passed to *note
find(): 3ce8. to get a list of the associated ‘.mo’ file paths.
Instances with identical ‘.mo’ file names are cached. The actual
class instantiated is 'class_' if provided, otherwise *note
GNUTranslations: 3ce7. The class’s constructor must take a single
*note file object: 11b5. argument.
If multiple files are found, later files are used as fallbacks for
earlier ones. To allow setting the fallback, *note copy.copy():
52f. is used to clone each translation object from the cache; the
actual instance data is still shared with the cache.
If no ‘.mo’ file is found, this function raises *note OSError: 413.
if 'fallback' is false (which is the default), and returns a *note
NullTranslations: 3ce9. instance if 'fallback' is true.
Changed in version 3.3: *note IOError: 104b. used to be raised, it
is now an alias of *note OSError: 413.
Changed in version 3.11: 'codeset' parameter is removed.
-- Function: gettext.install (domain, localedir=None, *, names=None)
This installs the function ‘_()’ in Python’s builtins namespace,
based on 'domain' and 'localedir' which are passed to the function
*note translation(): a79.
For the 'names' parameter, please see the description of the
translation object’s *note install(): 3cea. method.
As seen below, you usually mark the strings in your application
that are candidates for translation, by wrapping them in a call to
the ‘_()’ function, like this:
print(_('This string will be translated.'))
For convenience, you want the ‘_()’ function to be installed in
Python’s builtins namespace, so it is easily accessible in all
modules of your application.
Changed in version 3.11: 'names' is now a keyword-only parameter.
* Menu:
* The NullTranslations class::
* The GNUTranslations class::
* Solaris message catalog support::
* The Catalog constructor::
---------- Footnotes ----------
(1) See the footnote for *note bindtextdomain(): 3cde. above.
File: python.info, Node: The NullTranslations class, Next: The GNUTranslations class, Up: Class-based API
5.24.1.3 The ‘NullTranslations’ class
.....................................
Translation classes are what actually implement the translation of
original source file message strings to translated message strings. The
base class used by all translation classes is *note NullTranslations:
3ce9.; this provides the basic interface you can use to write your own
specialized translation classes. Here are the methods of
‘NullTranslations’:
-- Class: gettext.NullTranslations (fp=None)
Takes an optional *note file object: 11b5. 'fp', which is ignored
by the base class. Initializes “protected” instance variables
'_info' and '_charset' which are set by derived classes, as well as
'_fallback', which is set through *note add_fallback(): 3cec. It
then calls ‘self._parse(fp)’ if 'fp' is not ‘None’.
-- Method: _parse (fp)
No-op in the base class, this method takes file object 'fp',
and reads the data from the file, initializing its message
catalog. If you have an unsupported message catalog file
format, you should override this method to parse your format.
-- Method: add_fallback (fallback)
Add 'fallback' as the fallback object for the current
translation object. A translation object should consult the
fallback if it cannot provide a translation for a given
message.
-- Method: gettext (message)
If a fallback has been set, forward ‘gettext()’ to the
fallback. Otherwise, return 'message'. Overridden in derived
classes.
-- Method: ngettext (singular, plural, n)
If a fallback has been set, forward ‘ngettext()’ to the
fallback. Otherwise, return 'singular' if 'n' is 1; return
'plural' otherwise. Overridden in derived classes.
-- Method: pgettext (context, message)
If a fallback has been set, forward *note pgettext(): 9f4. to
the fallback. Otherwise, return the translated message.
Overridden in derived classes.
Added in version 3.8.
-- Method: npgettext (context, singular, plural, n)
If a fallback has been set, forward *note npgettext(): 3ce4.
to the fallback. Otherwise, return the translated message.
Overridden in derived classes.
Added in version 3.8.
-- Method: info ()
Return a dictionary containing the metadata found in the
message catalog file.
-- Method: charset ()
Return the encoding of the message catalog file.
-- Method: install (names=None)
This method installs *note gettext(): 3cee. into the built-in
namespace, binding it to ‘_’.
If the 'names' parameter is given, it must be a sequence
containing the names of functions you want to install in the
builtins namespace in addition to ‘_()’. Supported names are
‘'gettext'’, ‘'ngettext'’, ‘'pgettext'’, and ‘'npgettext'’.
Note that this is only one way, albeit the most convenient
way, to make the ‘_()’ function available to your application.
Because it affects the entire application globally, and
specifically the built-in namespace, localized modules should
never install ‘_()’. Instead, they should use this code to
make ‘_()’ available to their module:
import gettext
t = gettext.translation('mymodule', ...)
_ = t.gettext
This puts ‘_()’ only in the module’s global namespace and so
only affects calls within this module.
Changed in version 3.8: Added ‘'pgettext'’ and ‘'npgettext'’.
File: python.info, Node: The GNUTranslations class, Next: Solaris message catalog support, Prev: The NullTranslations class, Up: Class-based API
5.24.1.4 The ‘GNUTranslations’ class
....................................
The ‘gettext’ module provides one additional class derived from *note
NullTranslations: 3ce9.: *note GNUTranslations: 3ce7. This class
overrides ‘_parse()’ to enable reading GNU ‘gettext’ format ‘.mo’ files
in both big-endian and little-endian format.
*note GNUTranslations: 3ce7. parses optional metadata out of the
translation catalog. It is convention with GNU ‘gettext’ to include
metadata as the translation for the empty string. This metadata is in
RFC 822(1)-style ‘key: value’ pairs, and should contain the
‘Project-Id-Version’ key. If the key ‘Content-Type’ is found, then the
‘charset’ property is used to initialize the “protected” ‘_charset’
instance variable, defaulting to ‘None’ if not found. If the charset
encoding is specified, then all message ids and message strings read
from the catalog are converted to Unicode using this encoding, else
ASCII is assumed.
Since message ids are read as Unicode strings too, all ‘*gettext()’
methods will assume message ids as Unicode strings, not byte strings.
The entire set of key/value pairs are placed into a dictionary and set
as the “protected” ‘_info’ instance variable.
If the ‘.mo’ file’s magic number is invalid, the major version number is
unexpected, or if other problems occur while reading the file,
instantiating a *note GNUTranslations: 3ce7. class can raise *note
OSError: 413.
-- Class: gettext.GNUTranslations
The following methods are overridden from the base class
implementation:
-- Method: gettext (message)
Look up the 'message' id in the catalog and return the
corresponding message string, as a Unicode string. If there
is no entry in the catalog for the 'message' id, and a
fallback has been set, the look up is forwarded to the
fallback’s *note gettext(): 3cee. method. Otherwise, the
'message' id is returned.
-- Method: ngettext (singular, plural, n)
Do a plural-forms lookup of a message id. 'singular' is used
as the message id for purposes of lookup in the catalog, while
'n' is used to determine which plural form to use. The
returned message string is a Unicode string.
If the message id is not found in the catalog, and a fallback
is specified, the request is forwarded to the fallback’s *note
ngettext(): 3cef. method. Otherwise, when 'n' is 1 'singular'
is returned, and 'plural' is returned in all other cases.
Here is an example:
n = len(os.listdir('.'))
cat = GNUTranslations(somefile)
message = cat.ngettext(
'There is %(num)d file in this directory',
'There are %(num)d files in this directory',
n) % {'num': n}
-- Method: pgettext (context, message)
Look up the 'context' and 'message' id in the catalog and
return the corresponding message string, as a Unicode string.
If there is no entry in the catalog for the 'message' id and
'context', and a fallback has been set, the look up is
forwarded to the fallback’s *note pgettext(): 9f4. method.
Otherwise, the 'message' id is returned.
Added in version 3.8.
-- Method: npgettext (context, singular, plural, n)
Do a plural-forms lookup of a message id. 'singular' is used
as the message id for purposes of lookup in the catalog, while
'n' is used to determine which plural form to use.
If the message id for 'context' is not found in the catalog,
and a fallback is specified, the request is forwarded to the
fallback’s *note npgettext(): 3ce4. method. Otherwise, when
'n' is 1 'singular' is returned, and 'plural' is returned in
all other cases.
Added in version 3.8.
---------- Footnotes ----------
(1) https://datatracker.ietf.org/doc/html/rfc822.html
File: python.info, Node: Solaris message catalog support, Next: The Catalog constructor, Prev: The GNUTranslations class, Up: Class-based API
5.24.1.5 Solaris message catalog support
........................................
The Solaris operating system defines its own binary ‘.mo’ file format,
but since no documentation can be found on this format, it is not
supported at this time.
File: python.info, Node: The Catalog constructor, Prev: Solaris message catalog support, Up: Class-based API
5.24.1.6 The Catalog constructor
................................
GNOME uses a version of the *note gettext: 63. module by James
Henstridge, but this version has a slightly different API. Its
documented usage was:
import gettext
cat = gettext.Catalog(domain, localedir)
_ = cat.gettext
print(_('hello world'))
For compatibility with this older module, the function ‘Catalog()’ is an
alias for the *note translation(): a79. function described above.
One difference between this module and Henstridge’s: his catalog objects
supported access through a mapping API, but this appears to be unused
and so is not currently supported.
File: python.info, Node: Internationalizing your programs and modules, Next: Acknowledgements<9>, Prev: Class-based API, Up: gettext — Multilingual internationalization services
5.24.1.7 Internationalizing your programs and modules
.....................................................
Internationalization (I18N) refers to the operation by which a program
is made aware of multiple languages. Localization (L10N) refers to the
adaptation of your program, once internationalized, to the local
language and cultural habits. In order to provide multilingual messages
for your Python programs, you need to take the following steps:
1. prepare your program or module by specially marking translatable
strings
2. run a suite of tools over your marked files to generate raw
messages catalogs
3. create language-specific translations of the message catalogs
4. use the *note gettext: 63. module so that message strings are
properly translated
In order to prepare your code for I18N, you need to look at all the
strings in your files. Any string that needs to be translated should be
marked by wrapping it in ‘_('...')’ — that is, a call to the function
*note _: 63. For example:
filename = 'mylog.txt'
message = _('writing a log message')
with open(filename, 'w') as fp:
fp.write(message)
In this example, the string ‘'writing a log message'’ is marked as a
candidate for translation, while the strings ‘'mylog.txt'’ and ‘'w'’ are
not.
There are a few tools to extract the strings meant for translation. The
original GNU ‘gettext’ only supported C or C++ source code but its
extended version ‘xgettext’ scans code written in a number of languages,
including Python, to find strings marked as translatable. Babel(1) is a
Python internationalization library that includes a ‘pybabel’ script to
extract and compile message catalogs. François Pinard’s program called
‘xpot’ does a similar job and is available as part of his po-utils
package(2).
(Python also includes pure-Python versions of these programs, called
‘pygettext.py’ and ‘msgfmt.py’; some Python distributions will install
them for you. ‘pygettext.py’ is similar to ‘xgettext’, but only
understands Python source code and cannot handle other programming
languages such as C or C++. ‘pygettext.py’ supports a command-line
interface similar to ‘xgettext’; for details on its use, run
‘pygettext.py --help’. ‘msgfmt.py’ is binary compatible with GNU
‘msgfmt’. With these two programs, you may not need the GNU ‘gettext’
package to internationalize your Python applications.)
‘xgettext’, ‘pygettext’, and similar tools generate ‘.po’ files that are
message catalogs. They are structured human-readable files that contain
every marked string in the source code, along with a placeholder for the
translated versions of these strings.
Copies of these ‘.po’ files are then handed over to the individual human
translators who write translations for every supported natural language.
They send back the completed language-specific versions as a
‘.po’ file that’s compiled into a machine-readable ‘.mo’
binary catalog file using the ‘msgfmt’ program. The ‘.mo’ files are
used by the *note gettext: 63. module for the actual translation
processing at run-time.
How you use the *note gettext: 63. module in your code depends on
whether you are internationalizing a single module or your entire
application. The next two sections will discuss each case.
* Menu:
* Localizing your module::
* Localizing your application::
* Changing languages on the fly::
* Deferred translations::
---------- Footnotes ----------
(1) https://babel.pocoo.org/
(2) https://github.com/pinard/po-utils
File: python.info, Node: Localizing your module, Next: Localizing your application, Up: Internationalizing your programs and modules
5.24.1.8 Localizing your module
...............................
If you are localizing your module, you must take care not to make global
changes, e.g. to the built-in namespace. You should not use the GNU
‘gettext’ API but instead the class-based API.
Let’s say your module is called “spam” and the module’s various natural
language translation ‘.mo’ files reside in ‘/usr/share/locale’ in GNU
‘gettext’ format. Here’s what you would put at the top of your module:
import gettext
t = gettext.translation('spam', '/usr/share/locale')
_ = t.gettext
File: python.info, Node: Localizing your application, Next: Changing languages on the fly, Prev: Localizing your module, Up: Internationalizing your programs and modules
5.24.1.9 Localizing your application
....................................
If you are localizing your application, you can install the ‘_()’
function globally into the built-in namespace, usually in the main
driver file of your application. This will let all your
application-specific files just use ‘_('...')’ without having to
explicitly install it in each file.
In the simple case then, you need only add the following bit of code to
the main driver file of your application:
import gettext
gettext.install('myapplication')
If you need to set the locale directory, you can pass it into the *note
install(): a7a. function:
import gettext
gettext.install('myapplication', '/usr/share/locale')
File: python.info, Node: Changing languages on the fly, Next: Deferred translations, Prev: Localizing your application, Up: Internationalizing your programs and modules
5.24.1.10 Changing languages on the fly
.......................................
If your program needs to support many languages at the same time, you
may want to create multiple translation instances and then switch
between them explicitly, like so:
import gettext
lang1 = gettext.translation('myapplication', languages=['en'])
lang2 = gettext.translation('myapplication', languages=['fr'])
lang3 = gettext.translation('myapplication', languages=['de'])
# start by using language1
lang1.install()
# ... time goes by, user selects language 2
lang2.install()
# ... more time goes by, user selects language 3
lang3.install()
File: python.info, Node: Deferred translations, Prev: Changing languages on the fly, Up: Internationalizing your programs and modules
5.24.1.11 Deferred translations
...............................
In most coding situations, strings are translated where they are coded.
Occasionally however, you need to mark strings for translation, but
defer actual translation until later. A classic example is:
animals = ['mollusk',
'albatross',
'rat',
'penguin',
'python', ]
# ...
for a in animals:
print(a)
Here, you want to mark the strings in the ‘animals’ list as being
translatable, but you don’t actually want to translate them until they
are printed.
Here is one way you can handle this situation:
def _(message): return message
animals = [_('mollusk'),
_('albatross'),
_('rat'),
_('penguin'),
_('python'), ]
del _
# ...
for a in animals:
print(_(a))
This works because the dummy definition of ‘_()’ simply returns the
string unchanged. And this dummy definition will temporarily override
any definition of ‘_()’ in the built-in namespace (until the *note del:
17a6. command). Take care, though if you have a previous definition of
‘_()’ in the local namespace.
Note that the second use of ‘_()’ will not identify “a” as being
translatable to the ‘gettext’ program, because the parameter is not a
string literal.
Another way to handle this is with the following example:
def N_(message): return message
animals = [N_('mollusk'),
N_('albatross'),
N_('rat'),
N_('penguin'),
N_('python'), ]
# ...
for a in animals:
print(_(a))
In this case, you are marking translatable strings with the function
‘N_()’, which won’t conflict with any definition of ‘_()’. However, you
will need to teach your message extraction program to look for
translatable strings marked with ‘N_()’. ‘xgettext’, ‘pygettext’,
‘pybabel extract’, and ‘xpot’ all support this through the use of the
‘-k’ command-line switch. The choice of ‘N_()’ here is totally
arbitrary; it could have just as easily been
‘MarkThisStringForTranslation()’.
File: python.info, Node: Acknowledgements<9>, Prev: Internationalizing your programs and modules, Up: gettext — Multilingual internationalization services
5.24.1.12 Acknowledgements
..........................
The following people contributed code, feedback, design suggestions,
previous implementations, and valuable experience to the creation of
this module:
* Peter Funk
* James Henstridge
* Juan David Ibáñez Palomar
* Marc-André Lemburg
* Martin von Löwis
* François Pinard
* Barry Warsaw
* Gustavo Niemeyer
File: python.info, Node: locale — Internationalization services, Prev: gettext — Multilingual internationalization services, Up: Internationalization
5.24.2 ‘locale’ — Internationalization services
-----------------------------------------------
'Source code:' Lib/locale.py(1)
__________________________________________________________________
The *note locale: 86. module opens access to the POSIX locale database
and functionality. The POSIX locale mechanism allows programmers to
deal with certain cultural issues in an application, without requiring
the programmer to know all the specifics of each country where the
software is executed.
The *note locale: 86. module is implemented on top of the ‘_locale’
module, which in turn uses an ANSI C locale implementation if available.
The *note locale: 86. module defines the following exception and
functions:
-- Exception: locale.Error
Exception raised when the locale passed to *note setlocale(): 2df.
is not recognized.
-- Function: locale.setlocale (category, locale=None)
If 'locale' is given and not ‘None’, *note setlocale(): 2df.
modifies the locale setting for the 'category'. The available
categories are listed in the data description below. 'locale' may
be a *note string: 3d04, or a pair, language code and encoding. An
empty string specifies the user’s default settings. If the
modification of the locale fails, the exception *note Error: 3d03.
is raised. If successful, the new locale setting is returned.
If 'locale' is a pair, it is converted to a locale name using the
locale aliasing engine. The language code has the same format as a
*note locale name: 3d04, but without encoding and ‘@’-modifier.
The language code and encoding can be ‘None’.
If 'locale' is omitted or ‘None’, the current setting for
'category' is returned.
*note setlocale(): 2df. is not thread-safe on most systems.
Applications typically start with a call of
import locale
locale.setlocale(locale.LC_ALL, '')
This sets the locale for all categories to the user’s default
setting (typically specified in the ‘LANG’ environment variable).
If the locale is not changed thereafter, using multithreading
should not cause problems.
-- Function: locale.localeconv ()
Returns the database of the local conventions as a dictionary.
This dictionary has the following strings as keys:
Category Key Meaning
----------------------------------------------------------------------------------------------------------
*note LC_NUMERIC: 3d05. ‘'decimal_point'’ Decimal point character.
‘'grouping'’ Sequence of numbers specifying
which relative positions the
‘'thousands_sep'’ is expected. If
the sequence is terminated with
*note CHAR_MAX: 3d06, no further
grouping is performed. If the
sequence terminates with a ‘0’,
the last group size is repeatedly
used.
‘'thousands_sep'’ Character used between groups.
*note LC_MONETARY: 3d07. ‘'int_curr_symbol'’ International currency symbol.
‘'currency_symbol'’ Local currency symbol.
‘'p_cs_precedes/n_cs_precedes'’ Whether the currency symbol
precedes the value (for positive
resp. negative values).
‘'p_sep_by_space/n_sep_by_space'’ Whether the currency symbol is
separated from the value by a
space (for positive resp.
negative values).
‘'mon_decimal_point'’ Decimal point used for monetary
values.
‘'frac_digits'’ Number of fractional digits used
in local formatting of monetary
values.
‘'int_frac_digits'’ Number of fractional digits used
in international formatting of
monetary values.
‘'mon_thousands_sep'’ Group separator used for monetary
values.
‘'mon_grouping'’ Equivalent to ‘'grouping'’, used
for monetary values.
‘'positive_sign'’ Symbol used to annotate a positive
monetary value.
‘'negative_sign'’ Symbol used to annotate a negative
monetary value.
‘'p_sign_posn/n_sign_posn'’ The position of the sign (for
positive resp. negative values),
see below.
All numeric values can be set to *note CHAR_MAX: 3d06. to indicate
that there is no value specified in this locale.
The possible values for ‘'p_sign_posn'’ and ‘'n_sign_posn'’ are
given below.
Value Explanation
-----------------------------------------------------------------
‘0’ Currency and value are surrounded by
parentheses.
‘1’ The sign should precede the value and
currency symbol.
‘2’ The sign should follow the value and
currency symbol.
‘3’ The sign should immediately precede the
value.
‘4’ The sign should immediately follow the
value.
‘CHAR_MAX’ Nothing is specified in this locale.
The function temporarily sets the ‘LC_CTYPE’ locale to the
‘LC_NUMERIC’ locale or the ‘LC_MONETARY’ locale if locales are
different and numeric or monetary strings are non-ASCII. This
temporary change affects other threads.
Changed in version 3.7: The function now temporarily sets the
‘LC_CTYPE’ locale to the ‘LC_NUMERIC’ locale in some cases.
-- Function: locale.nl_langinfo (option)
Return some locale-specific information as a string. This function
is not available on all systems, and the set of possible options
might also vary across platforms. The possible argument values are
numbers, for which symbolic constants are available in the locale
module.
The *note nl_langinfo(): 3d08. function accepts one of the
following keys. Most descriptions are taken from the corresponding
description in the GNU C library.
-- Data: locale.CODESET
Get a string with the name of the character encoding used in
the selected locale.
-- Data: locale.D_T_FMT
Get a string that can be used as a format string for *note
time.strftime(): 11db. to represent date and time in a
locale-specific way.
-- Data: locale.D_FMT
Get a string that can be used as a format string for *note
time.strftime(): 11db. to represent a date in a
locale-specific way.
-- Data: locale.T_FMT
Get a string that can be used as a format string for *note
time.strftime(): 11db. to represent a time in a
locale-specific way.
-- Data: locale.T_FMT_AMPM
Get a format string for *note time.strftime(): 11db. to
represent time in the am/pm format.
-- Data: locale.DAY_1
-- Data: locale.DAY_2
-- Data: locale.DAY_3
-- Data: locale.DAY_4
-- Data: locale.DAY_5
-- Data: locale.DAY_6
-- Data: locale.DAY_7
Get the name of the n-th day of the week.
Note: This follows the US convention of *note DAY_1:
3d0e. being Sunday, not the international convention (ISO
8601) that Monday is the first day of the week.
-- Data: locale.ABDAY_1
-- Data: locale.ABDAY_2
-- Data: locale.ABDAY_3
-- Data: locale.ABDAY_4
-- Data: locale.ABDAY_5
-- Data: locale.ABDAY_6
-- Data: locale.ABDAY_7
Get the abbreviated name of the n-th day of the week.
-- Data: locale.MON_1
-- Data: locale.MON_2
-- Data: locale.MON_3
-- Data: locale.MON_4
-- Data: locale.MON_5
-- Data: locale.MON_6
-- Data: locale.MON_7
-- Data: locale.MON_8
-- Data: locale.MON_9
-- Data: locale.MON_10
-- Data: locale.MON_11
-- Data: locale.MON_12
Get the name of the n-th month.
-- Data: locale.ABMON_1
-- Data: locale.ABMON_2
-- Data: locale.ABMON_3
-- Data: locale.ABMON_4
-- Data: locale.ABMON_5
-- Data: locale.ABMON_6
-- Data: locale.ABMON_7
-- Data: locale.ABMON_8
-- Data: locale.ABMON_9
-- Data: locale.ABMON_10
-- Data: locale.ABMON_11
-- Data: locale.ABMON_12
Get the abbreviated name of the n-th month.
-- Data: locale.RADIXCHAR
Get the radix character (decimal dot, decimal comma, etc.).
-- Data: locale.THOUSEP
Get the separator character for thousands (groups of three
digits).
-- Data: locale.YESEXPR
Get a regular expression that can be used with the regex
function to recognize a positive response to a yes/no
question.
-- Data: locale.NOEXPR
Get a regular expression that can be used with the ‘regex(3)’
function to recognize a negative response to a yes/no
question.
Note: The regular expressions for *note YESEXPR: 3d36.
and *note NOEXPR: 3d37. use syntax suitable for the
‘regex’ function from the C library, which might differ
from the syntax used in *note re: b9.
-- Data: locale.CRNCYSTR
Get the currency symbol, preceded by “-” if the symbol should
appear before the value, “+” if the symbol should appear after
the value, or “.” if the symbol should replace the radix
character.
-- Data: locale.ERA
Get a string which describes how years are counted and
displayed for each era in a locale.
Most locales do not define this value. An example of a locale
which does define this value is the Japanese one. In Japan,
the traditional representation of dates includes the name of
the era corresponding to the then-emperor’s reign.
Normally it should not be necessary to use this value
directly. Specifying the ‘E’ modifier in their format strings
causes the *note time.strftime(): 11db. function to use this
information. The format of the returned string is specified
in 'The Open Group Base Specifications Issue 8', paragraph
7.3.5.2 LC_TIME C-Language Access(2).
-- Data: locale.ERA_D_T_FMT
Get a format string for *note time.strftime(): 11db. to
represent date and time in a locale-specific era-based way.
-- Data: locale.ERA_D_FMT
Get a format string for *note time.strftime(): 11db. to
represent a date in a locale-specific era-based way.
-- Data: locale.ERA_T_FMT
Get a format string for *note time.strftime(): 11db. to
represent a time in a locale-specific era-based way.
-- Data: locale.ALT_DIGITS
Get a string consisting of up to 100 semicolon-separated
symbols used to represent the values 0 to 99 in a
locale-specific way. In most locales this is an empty string.
-- Function: locale.getdefaultlocale ([envvars])
Tries to determine the default locale settings and returns them as
a tuple of the form ‘(language code, encoding)’.
According to POSIX, a program which has not called
‘setlocale(LC_ALL, '')’ runs using the portable ‘'C'’ locale.
Calling ‘setlocale(LC_ALL, '')’ lets it use the default locale as
defined by the ‘LANG’ variable. Since we do not want to interfere
with the current locale setting we thus emulate the behavior in the
way described above.
To maintain compatibility with other platforms, not only the ‘LANG’
variable is tested, but a list of variables given as envvars
parameter. The first found to be defined will be used. 'envvars'
defaults to the search path used in GNU gettext; it must always
contain the variable name ‘'LANG'’. The GNU gettext search path
contains ‘'LC_ALL'’, ‘'LC_CTYPE'’, ‘'LANG'’ and ‘'LANGUAGE'’, in
that order.
The language code has the same format as a *note locale name: 3d04,
but without encoding and ‘@’-modifier. The language code and
encoding may be ‘None’ if their values cannot be determined. The
“C” locale is represented as ‘(None, None)’.
Deprecated since version 3.11, will be removed in version 3.15.
-- Function: locale.getlocale (category=LC_CTYPE)
Returns the current setting for the given locale category as a
tuple containing the language code and encoding. 'category' may be
one of the ‘LC_*’ values except *note LC_ALL: 3d3e. It defaults to
*note LC_CTYPE: 3d3f.
The language code has the same format as a *note locale name: 3d04,
but without encoding and ‘@’-modifier. The language code and
encoding may be ‘None’ if their values cannot be determined. The
“C” locale is represented as ‘(None, None)’.
-- Function: locale.getpreferredencoding (do_setlocale=True)
Return the *note locale encoding: 244. used for text data,
according to user preferences. User preferences are expressed
differently on different systems, and might not be available
programmatically on some systems, so this function only returns a
guess.
On some systems, it is necessary to invoke *note setlocale(): 2df.
to obtain the user preferences, so this function is not
thread-safe. If invoking setlocale is not necessary or desired,
'do_setlocale' should be set to ‘False’.
On Android or if the *note Python UTF-8 Mode: 652. is enabled,
always return ‘'utf-8'’, the *note locale encoding: 244. and the
'do_setlocale' argument are ignored.
The *note Python preinitialization: 3d40. configures the LC_CTYPE
locale. See also the *note filesystem encoding and error handler:
537.
Changed in version 3.7: The function now always returns ‘"utf-8"’
on Android or if the *note Python UTF-8 Mode: 652. is enabled.
-- Function: locale.getencoding ()
Get the current *note locale encoding: 244.:
* On Android and VxWorks, return ‘"utf-8"’.
* On Unix, return the encoding of the current *note LC_CTYPE:
3d3f. locale. Return ‘"utf-8"’ if ‘nl_langinfo(CODESET)’
returns an empty string: for example, if the current LC_CTYPE
locale is not supported.
* On Windows, return the ANSI code page.
The *note Python preinitialization: 3d40. configures the LC_CTYPE
locale. See also the *note filesystem encoding and error handler:
537.
This function is similar to *note getpreferredencoding(False): 536.
except this function ignores the *note Python UTF-8 Mode: 652.
Added in version 3.11.
-- Function: locale.normalize (localename)
Returns a normalized locale code for the given locale name. The
returned locale code is formatted for use with *note setlocale():
2df. If normalization fails, the original name is returned
unchanged.
If the given encoding is not known, the function defaults to the
default encoding for the locale code just like *note setlocale():
2df.
-- Function: locale.strcoll (string1, string2)
Compares two strings according to the current *note LC_COLLATE:
3d43. setting. As any other compare function, returns a negative,
or a positive value, or ‘0’, depending on whether 'string1'
collates before or after 'string2' or is equal to it.
-- Function: locale.strxfrm (string)
Transforms a string to one that can be used in locale-aware
comparisons. For example, ‘strxfrm(s1) < strxfrm(s2)’ is
equivalent to ‘strcoll(s1, s2) < 0’. This function can be used
when the same string is compared repeatedly, e.g. when collating a
sequence of strings.
-- Function: locale.format_string (format, val, grouping=False,
monetary=False)
Formats a number 'val' according to the current *note LC_NUMERIC:
3d05. setting. The format follows the conventions of the ‘%’
operator. For floating-point values, the decimal point is modified
if appropriate. If 'grouping' is ‘True’, also takes the grouping
into account.
If 'monetary' is true, the conversion uses monetary thousands
separator and grouping strings.
Processes formatting specifiers as in ‘format % val’, but takes the
current locale settings into account.
Changed in version 3.7: The 'monetary' keyword parameter was added.
-- Function: locale.currency (val, symbol=True, grouping=False,
international=False)
Formats a number 'val' according to the current *note LC_MONETARY:
3d07. settings.
The returned string includes the currency symbol if 'symbol' is
true, which is the default. If 'grouping' is ‘True’ (which is not
the default), grouping is done with the value. If 'international'
is ‘True’ (which is not the default), the international currency
symbol is used.
Note: This function will not work with the ‘C’ locale, so you
have to set a locale via *note setlocale(): 2df. first.
-- Function: locale.str (float)
Formats a floating-point number using the same format as the
built-in function ‘str(float)’, but takes the decimal point into
account.
-- Function: locale.delocalize (string)
Converts a string into a normalized number string, following the
*note LC_NUMERIC: 3d05. settings.
Added in version 3.5.
-- Function: locale.localize (string, grouping=False, monetary=False)
Converts a normalized number string into a formatted string
following the *note LC_NUMERIC: 3d05. settings.
Added in version 3.10.
-- Function: locale.atof (string, func=float)
Converts a string to a number, following the *note LC_NUMERIC:
3d05. settings, by calling 'func' on the result of calling *note
delocalize(): e24. on 'string'.
-- Function: locale.atoi (string)
Converts a string to an integer, following the *note LC_NUMERIC:
3d05. conventions.
-- Data: locale.LC_CTYPE
Locale category for the character type functions. Most
importantly, this category defines the text encoding, i.e. how
bytes are interpreted as Unicode codepoints. See PEP 538(3) and
PEP 540(4) for how this variable might be automatically coerced to
‘C.UTF-8’ to avoid issues created by invalid settings in containers
or incompatible settings passed over remote SSH connections.
Python doesn’t internally use locale-dependent character
transformation functions from ‘ctype.h’. Instead, an internal
‘pyctype.h’ provides locale-independent equivalents like
‘Py_TOLOWER’.
-- Data: locale.LC_COLLATE
Locale category for sorting strings. The functions *note
strcoll(): 3d42. and *note strxfrm(): 3d44. of the *note locale:
86. module are affected.
-- Data: locale.LC_TIME
Locale category for the formatting of time. The function *note
time.strftime(): 11db. follows these conventions.
-- Data: locale.LC_MONETARY
Locale category for formatting of monetary values. The available
options are available from the *note localeconv(): bfe. function.
-- Data: locale.LC_MESSAGES
Locale category for message display. Python currently does not
support application specific locale-aware messages. Messages
displayed by the operating system, like those returned by *note
os.strerror(): 2b2f. might be affected by this category.
This value may not be available on operating systems not conforming
to the POSIX standard, most notably Windows.
-- Data: locale.LC_NUMERIC
Locale category for formatting numbers. The functions *note
format_string(): 51a, *note atoi(): 3d49, *note atof(): 3d48. and
*note str(): 3d46. of the *note locale: 86. module are affected by
that category. All other numeric formatting operations are not
affected.
-- Data: locale.LC_ALL
Combination of all locale settings. If this flag is used when the
locale is changed, setting the locale for all categories is
attempted. If that fails for any category, no category is changed
at all. When the locale is retrieved using this flag, a string
indicating the setting for all categories is returned. This string
can be later used to restore the settings.
-- Data: locale.CHAR_MAX
This is a symbolic constant used for different values returned by
*note localeconv(): bfe.
Example:
>>> import locale
>>> loc = locale.getlocale() # get current locale
# use German locale; name might vary with platform
>>> locale.setlocale(locale.LC_ALL, 'de_DE')
>>> locale.strcoll('f\xe4n', 'foo') # compare a string containing an umlaut
>>> locale.setlocale(locale.LC_ALL, '') # use user's preferred locale
>>> locale.setlocale(locale.LC_ALL, 'C') # use default (C) locale
>>> locale.setlocale(locale.LC_ALL, loc) # restore saved locale
* Menu:
* Background, details, hints, tips and caveats: Background details hints tips and caveats.
* Locale names::
* For extension writers and programs that embed Python::
* Access to message catalogs::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/locale.py
(2)
https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap07.html#tag_07_03_05_02
(3) https://peps.python.org/pep-0538/
(4) https://peps.python.org/pep-0540/
File: python.info, Node: Background details hints tips and caveats, Next: Locale names, Up: locale — Internationalization services
5.24.2.1 Background, details, hints, tips and caveats
.....................................................
The C standard defines the locale as a program-wide property that may be
relatively expensive to change. On top of that, some implementations
are broken in such a way that frequent locale changes may cause core
dumps. This makes the locale somewhat painful to use correctly.
Initially, when a program is started, the locale is the ‘C’ locale, no
matter what the user’s preferred locale is. There is one exception: the
*note LC_CTYPE: 3d3f. category is changed at startup to set the current
locale encoding to the user’s preferred locale encoding. The program
must explicitly say that it wants the user’s preferred locale settings
for other categories by calling ‘setlocale(LC_ALL, '')’.
It is generally a bad idea to call *note setlocale(): 2df. in some
library routine, since as a side effect it affects the entire program.
Saving and restoring it is almost as bad: it is expensive and affects
other threads that happen to run before the settings have been restored.
If, when coding a module for general use, you need a locale independent
version of an operation that is affected by the locale (such as certain
formats used with *note time.strftime(): 11db.), you will have to find a
way to do it without using the standard library routine. Even better is
convincing yourself that using locale settings is okay. Only as a last
resort should you document that your module is not compatible with
non-‘C’ locale settings.
The only way to perform numeric operations according to the locale is to
use the special functions defined by this module: *note atof(): 3d48,
*note atoi(): 3d49, *note format_string(): 51a, *note str(): 3d46.
There is no way to perform case conversions and character
classifications according to the locale. For (Unicode) text strings
these are done according to the character value only, while for byte
strings, the conversions and classifications are done according to the
ASCII value of the byte, and bytes whose high bit is set (i.e.,
non-ASCII bytes) are never converted or considered part of a character
class such as letter or whitespace.
File: python.info, Node: Locale names, Next: For extension writers and programs that embed Python, Prev: Background details hints tips and caveats, Up: locale — Internationalization services
5.24.2.2 Locale names
.....................
The format of the locale name is platform dependent, and the set of
supported locales can depend on the system configuration.
On Posix platforms, it usually has the format (1):
language ["_" territory] ["." charset] ["@" modifier]
where 'language' is a two- or three-letter language code from ISO
639(2), 'territory' is a two-letter country or region code from ISO
3166(3), 'charset' is a locale encoding, and 'modifier' is a script
name, a language subtag, a sort order identifier, or other locale
modifier (for example, “latin”, “valencia”, “stroke” and “euro”).
On Windows, several formats are supported. (4) (5) A subset of IETF
BCP 47(6) tags:
language ["-" script] ["-" territory] ["." charset]
language ["-" script] "-" territory "-" modifier
where 'language' and 'territory' have the same meaning as in Posix,
'script' is a four-letter script code from ISO 15924(7), and 'modifier'
is a language subtag, a sort order identifier or custom modifier (for
example, “valencia”, “stroke” or “x-python”). Both hyphen (‘'-'’) and
underscore (‘'_'’) separators are supported. Only UTF-8 encoding is
allowed for BCP 47 tags.
Windows also supports locale names in the format:
language ["_" territory] ["." charset]
where 'language' and 'territory' are full names, such as “English” and
“United States”, and 'charset' is either a code page number (for
example, “1252”) or UTF-8. Only the underscore separator is supported
in this format.
The “C” locale is supported on all platforms.
---------- Footnotes ----------
(1) IEEE Std 1003.1-2024; 8.2 Internationalization Variables
(https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap08.html#tag_08_02)
(2) https://www.iso.org/iso-639-language-code
(3) https://www.iso.org/iso-3166-country-codes.html
(4) UCRT Locale names, Languages, and Country/Region strings
(https://learn.microsoft.com/en-us/cpp/c-runtime-library/locale-names-languages-and-country-region-strings)
(5) Locale Names
(https://learn.microsoft.com/en-us/windows/win32/intl/locale-names)
(6) https://www.rfc-editor.org/info/bcp47
(7) https://www.unicode.org/iso15924/
File: python.info, Node: For extension writers and programs that embed Python, Next: Access to message catalogs, Prev: Locale names, Up: locale — Internationalization services
5.24.2.3 For extension writers and programs that embed Python
.............................................................
Extension modules should never call *note setlocale(): 2df, except to
find out what the current locale is. But since the return value can
only be used portably to restore it, that is not very useful (except
perhaps to find out whether or not the locale is ‘C’).
When Python code uses the *note locale: 86. module to change the locale,
this also affects the embedding application. If the embedding
application doesn’t want this to happen, it should remove the ‘_locale’
extension module (which does all the work) from the table of built-in
modules in the ‘config.c’ file, and make sure that the ‘_locale’ module
is not accessible as a shared library.
File: python.info, Node: Access to message catalogs, Prev: For extension writers and programs that embed Python, Up: locale — Internationalization services
5.24.2.4 Access to message catalogs
...................................
-- Function: locale.gettext (msg)
-- Function: locale.dgettext (domain, msg)
-- Function: locale.dcgettext (domain, msg, category)
-- Function: locale.textdomain (domain)
-- Function: locale.bindtextdomain (domain, dir)
-- Function: locale.bind_textdomain_codeset (domain, codeset)
The locale module exposes the C library’s gettext interface on systems
that provide this interface. It consists of the functions *note
gettext(): 63, *note dgettext(): 3d53, *note dcgettext(): 3d54, *note
textdomain(): 3d55, *note bindtextdomain(): 3d56, and *note
bind_textdomain_codeset(): 3d57. These are similar to the same
functions in the *note gettext: 63. module, but use the C library’s
binary format for message catalogs, and the C library’s search
algorithms for locating message catalogs.
Python applications should normally find no need to invoke these
functions, and should use *note gettext: 63. instead. A known exception
to this rule are applications that link with additional C libraries
which internally invoke C functions ‘gettext’ or ‘dcgettext’. For these
applications, it may be necessary to bind the text domain, so that the
libraries can properly locate their message catalogs.
File: python.info, Node: Program Frameworks, Next: Graphical User Interfaces with Tk, Prev: Internationalization, Up: The Python Standard Library
5.25 Program Frameworks
=======================
The modules described in this chapter are frameworks that will largely
dictate the structure of your program. Currently the modules described
here are all oriented toward writing command-line interfaces.
The full list of modules described in this chapter is:
* Menu:
* turtle — Turtle graphics::
* cmd — Support for line-oriented command interpreters::
* shlex — Simple lexical analysis::
File: python.info, Node: turtle — Turtle graphics, Next: cmd — Support for line-oriented command interpreters, Up: Program Frameworks
5.25.1 ‘turtle’ — Turtle graphics
---------------------------------
'Source code:' Lib/turtle.py(1)
__________________________________________________________________
* Menu:
* Introduction: Introduction<11>.
* Get started::
* Tutorial: Tutorial<4>.
* How to…::
* Turtle graphics reference::
* Methods of RawTurtle/Turtle and corresponding functions::
* Methods of TurtleScreen/Screen and corresponding functions::
* Public classes::
* Explanation: Explanation<2>.
* Help and configuration::
* turtledemo — Demo scripts::
* Changes since Python 2.6: Changes since Python 2 6.
* Changes since Python 3.0: Changes since Python 3 0.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/turtle.py
File: python.info, Node: Introduction<11>, Next: Get started, Up: turtle — Turtle graphics
5.25.1.1 Introduction
.....................
Turtle graphics is an implementation of the popular geometric drawing
tools introduced in Logo(1), developed by Wally Feurzeig, Seymour Papert
and Cynthia Solomon in 1967.
---------- Footnotes ----------
(1) https://en.wikipedia.org/wiki/Turtle_(robot)
File: python.info, Node: Get started, Next: Tutorial<4>, Prev: Introduction<11>, Up: turtle — Turtle graphics
5.25.1.2 Get started
....................
Imagine a robotic turtle starting at (0, 0) in the x-y plane. After an
‘import turtle’, give it the command ‘turtle.forward(15)’, and it moves
(on-screen!) 15 pixels in the direction it is facing, drawing a line as
it moves. Give it the command ‘turtle.right(25)’, and it rotates
in-place 25 degrees clockwise.
Turtle star
...........
Turtle can draw intricate shapes using programs that repeat simple
moves.
[image src="python-figures/turtle-star.png" ]
In Python, turtle graphics provides a representation of a physical
“turtle” (a little robot with a pen) that draws on a sheet of paper on
the floor.
It’s an effective and well-proven way for learners to encounter
programming concepts and interaction with software, as it provides
instant, visible feedback. It also provides convenient access to
graphical output in general.
Turtle drawing was originally created as an educational tool, to be used
by teachers in the classroom. For the programmer who needs to produce
some graphical output it can be a way to do that without the overhead of
introducing more complex or external libraries into their work.
File: python.info, Node: Tutorial<4>, Next: How to…, Prev: Get started, Up: turtle — Turtle graphics
5.25.1.3 Tutorial
.................
New users should start here. In this tutorial we’ll explore some of the
basics of turtle drawing.
* Menu:
* Starting a turtle environment::
* Basic drawing::
* Making algorithmic patterns::
File: python.info, Node: Starting a turtle environment, Next: Basic drawing, Up: Tutorial<4>
5.25.1.4 Starting a turtle environment
......................................
In a Python shell, import all the objects of the ‘turtle’ module:
from turtle import *
If you run into a ‘No module named '_tkinter'’ error, you’ll have to
install the *note Tk interface package: f0. on your system.
File: python.info, Node: Basic drawing, Next: Making algorithmic patterns, Prev: Starting a turtle environment, Up: Tutorial<4>
5.25.1.5 Basic drawing
......................
Send the turtle forward 100 steps:
forward(100)
You should see (most likely, in a new window on your display) a line
drawn by the turtle, heading East. Change the direction of the turtle,
so that it turns 120 degrees left (anti-clockwise):
left(120)
Let’s continue by drawing a triangle:
forward(100)
left(120)
forward(100)
Notice how the turtle, represented by an arrow, points in different
directions as you steer it.
Experiment with those commands, and also with ‘backward()’ and
‘right()’.
* Menu:
* Pen control::
* The turtle’s position::
File: python.info, Node: Pen control, Next: The turtle’s position, Up: Basic drawing
5.25.1.6 Pen control
....................
Try changing the color - for example, ‘color('blue')’ - and width of the
line - for example, ‘width(3)’ - and then drawing again.
You can also move the turtle around without drawing, by lifting up the
pen: ‘up()’ before moving. To start drawing again, use ‘down()’.
File: python.info, Node: The turtle’s position, Prev: Pen control, Up: Basic drawing
5.25.1.7 The turtle’s position
..............................
Send your turtle back to its starting-point (useful if it has
disappeared off-screen):
home()
The home position is at the center of the turtle’s screen. If you ever
need to know them, get the turtle’s x-y coordinates with:
pos()
Home is at ‘(0, 0)’.
And after a while, it will probably help to clear the window so we can
start anew:
clearscreen()
File: python.info, Node: Making algorithmic patterns, Prev: Basic drawing, Up: Tutorial<4>
5.25.1.8 Making algorithmic patterns
....................................
Using loops, it’s possible to build up geometric patterns:
for steps in range(100):
for c in ('blue', 'red', 'green'):
color(c)
forward(steps)
right(30)
- which of course, are limited only by the imagination!
Let’s draw the star shape at the top of this page. We want red lines,
filled in with yellow:
color('red')
fillcolor('yellow')
Just as ‘up()’ and ‘down()’ determine whether lines will be drawn,
filling can be turned on and off:
begin_fill()
Next we’ll create a loop:
while True:
forward(200)
left(170)
if abs(pos()) < 1:
break
‘abs(pos()) < 1’ is a good way to know when the turtle is back at its
home position.
Finally, complete the filling:
end_fill()
(Note that filling only actually takes place when you give the
‘end_fill()’ command.)
File: python.info, Node: How to…, Next: Turtle graphics reference, Prev: Tutorial<4>, Up: turtle — Turtle graphics
5.25.1.9 How to…
................
This section covers some typical turtle use-cases and approaches.
* Menu:
* Get started as quickly as possible::
* Use the turtle module namespace::
* Use turtle graphics in a script::
* Use object-oriented turtle graphics::
File: python.info, Node: Get started as quickly as possible, Next: Use the turtle module namespace, Up: How to…
5.25.1.10 Get started as quickly as possible
............................................
One of the joys of turtle graphics is the immediate, visual feedback
that’s available from simple commands - it’s an excellent way to
introduce children to programming ideas, with a minimum of overhead (not
just children, of course).
The turtle module makes this possible by exposing all its basic
functionality as functions, available with ‘from turtle import *’. The
*note turtle graphics tutorial: 3d5f. covers this approach.
It’s worth noting that many of the turtle commands also have even more
terse equivalents, such as ‘fd()’ for *note forward(): 3d69. These are
especially useful when working with learners for whom typing is not a
skill.
You’ll need to have the *note Tk interface package: f0. installed
on your system for turtle graphics to work. Be warned that this is
not always straightforward, so check this in advance if you’re
planning to use turtle graphics with a learner.
File: python.info, Node: Use the turtle module namespace, Next: Use turtle graphics in a script, Prev: Get started as quickly as possible, Up: How to…
5.25.1.11 Use the ‘turtle’ module namespace
...........................................
Using ‘from turtle import *’ is convenient - but be warned that it
imports a rather large collection of objects, and if you’re doing
anything but turtle graphics you run the risk of a name conflict (this
becomes even more an issue if you’re using turtle graphics in a script
where other modules might be imported).
The solution is to use ‘import turtle’ - ‘fd()’ becomes ‘turtle.fd()’,
‘width()’ becomes ‘turtle.width()’ and so on. (If typing “turtle” over
and over again becomes tedious, use for example ‘import turtle as t’
instead.)
File: python.info, Node: Use turtle graphics in a script, Next: Use object-oriented turtle graphics, Prev: Use the turtle module namespace, Up: How to…
5.25.1.12 Use turtle graphics in a script
.........................................
It’s recommended to use the ‘turtle’ module namespace as described
immediately above, for example:
import turtle as t
from random import random
for i in range(100):
steps = int(random() * 100)
angle = int(random() * 360)
t.right(angle)
t.fd(steps)
Another step is also required though - as soon as the script ends,
Python will also close the turtle’s window. Add:
t.mainloop()
to the end of the script. The script will now wait to be dismissed and
will not exit until it is terminated, for example by closing the turtle
graphics window.
File: python.info, Node: Use object-oriented turtle graphics, Prev: Use turtle graphics in a script, Up: How to…
5.25.1.13 Use object-oriented turtle graphics
.............................................
See also
........
*note Explanation of the object-oriented interface: 3d6e.
Other than for very basic introductory purposes, or for trying things
out as quickly as possible, it’s more usual and much more powerful to
use the object-oriented approach to turtle graphics. For example, this
allows multiple turtles on screen at once.
In this approach, the various turtle commands are methods of objects
(mostly of ‘Turtle’ objects). You 'can' use the object-oriented
approach in the shell, but it would be more typical in a Python script.
The example above then becomes:
from turtle import Turtle
from random import random
t = Turtle()
for i in range(100):
steps = int(random() * 100)
angle = int(random() * 360)
t.right(angle)
t.fd(steps)
t.screen.mainloop()
Note the last line. ‘t.screen’ is an instance of the *note Screen:
3d6f. that a Turtle instance exists on; it’s created automatically along
with the turtle.
The turtle’s screen can be customised, for example:
t.screen.title('Object-oriented turtle demo')
t.screen.bgcolor("orange")
File: python.info, Node: Turtle graphics reference, Next: Methods of RawTurtle/Turtle and corresponding functions, Prev: How to…, Up: turtle — Turtle graphics
5.25.1.14 Turtle graphics reference
...................................
Note: In the following documentation the argument list for
functions is given. Methods, of course, have the additional first
argument 'self' which is omitted here.
* Menu:
* Turtle methods::
* Methods of TurtleScreen/Screen::
File: python.info, Node: Turtle methods, Next: Methods of TurtleScreen/Screen, Up: Turtle graphics reference
5.25.1.15 Turtle methods
........................
Turtle motion
Move and draw
*note forward(): 3d69. | *note fd(): 3d72.
*note backward(): 3d73. | *note bk(): 3d74. | *note back(): 3d75.
*note right(): 3d76. | *note rt(): 3d77.
*note left(): 3d78. | *note lt(): 3d79.
*note goto(): 3d7a. | *note setpos(): 3d7b. | *note setposition(): 3d7c.
*note teleport(): 1726.
*note setx(): 3d7d.
*note sety(): 3d7e.
*note setheading(): 3d7f. | *note seth(): 3d80.
*note home(): 3d81.
*note circle(): 3d82.
*note dot(): 3d83.
*note stamp(): 3d84.
*note clearstamp(): 3d85.
*note clearstamps(): 3d86.
*note undo(): 3d87.
*note speed(): 3d88.
Tell Turtle’s state
*note position(): 3d89. | *note pos(): 3d8a.
*note towards(): 3d8b.
*note xcor(): 3d8c.
*note ycor(): 3d8d.
*note heading(): 3d8e.
*note distance(): 3d8f.
Setting and measurement
*note degrees(): 3d90.
*note radians(): 3d91.
Pen control
Drawing state
*note pendown(): 3d92. | *note pd(): 3d93. | *note down(): 3d94.
*note penup(): 3d95. | *note pu(): 3d96. | *note up(): 3d97.
*note pensize(): 3d98. | *note width(): 3d99.
*note pen(): 3d9a.
*note isdown(): 3d9b.
Color control
*note color(): 3d9c.
*note pencolor(): 3d9d.
*note fillcolor(): 3d9e.
Filling
*note filling(): 3d9f.
*note begin_fill(): 3da0.
*note end_fill(): 3da1.
More drawing control
*note reset(): 3da2.
*note clear(): 3da3.
*note write(): 3da4.
Turtle state
Visibility
*note showturtle(): 3da5. | *note st(): 3da6.
*note hideturtle(): 3da7. | *note ht(): 3da8.
*note isvisible(): 3da9.
Appearance
*note shape(): 3daa.
*note resizemode(): 3dab.
*note shapesize(): 3dac. | *note turtlesize(): 3dad.
*note shearfactor(): 3dae.
*note tiltangle(): 720.
*note tilt(): 3daf.
*note shapetransform(): 3db0.
*note get_shapepoly(): 3db1.
Using events
*note onclick(): 3db2.
*note onrelease(): 3db3.
*note ondrag(): 3db4.
Special Turtle methods
*note begin_poly(): 3db5.
*note end_poly(): 3db6.
*note get_poly(): 3db7.
*note clone(): 3db8.
*note getturtle(): 3db9. | *note getpen(): 3dba.
*note getscreen(): 3dbb.
*note setundobuffer(): 3dbc.
*note undobufferentries(): 3dbd.
File: python.info, Node: Methods of TurtleScreen/Screen, Prev: Turtle methods, Up: Turtle graphics reference
5.25.1.16 Methods of TurtleScreen/Screen
........................................
Window control
*note bgcolor(): 3dbf.
*note bgpic(): 3dc0.
*note clearscreen(): 3dc1.
*note resetscreen(): 3dc2.
*note screensize(): 3dc3.
*note setworldcoordinates(): 3dc4.
Animation control
*note delay(): 3dc5.
*note tracer(): 3dc6.
*note update(): 3dc7.
Using screen events
*note listen(): 3dc8.
*note onkey(): 3dc9. | *note onkeyrelease(): 3dca.
*note onkeypress(): 3dcb.
*note onclick(): 3db2. | *note onscreenclick(): 3dcc.
*note ontimer(): 3dcd.
*note mainloop(): 3dce. | *note done(): 3dcf.
Settings and special methods
*note mode(): 3dd0.
*note colormode(): 3dd1.
*note getcanvas(): 3dd2.
*note getshapes(): 3dd3.
*note register_shape(): 3dd4. | *note addshape(): 3dd5.
*note turtles(): 3dd6.
*note window_height(): 3dd7.
*note window_width(): 3dd8.
Input methods
*note textinput(): 18c2.
*note numinput(): 18c3.
Methods specific to Screen
*note bye(): 3dd9.
*note exitonclick(): 3dda.
*note setup(): 3ddb.
*note title(): 3ddc.
File: python.info, Node: Methods of RawTurtle/Turtle and corresponding functions, Next: Methods of TurtleScreen/Screen and corresponding functions, Prev: Turtle graphics reference, Up: turtle — Turtle graphics
5.25.1.17 Methods of RawTurtle/Turtle and corresponding functions
.................................................................
Most of the examples in this section refer to a Turtle instance called
‘turtle’.
* Menu:
* Turtle motion::
* Tell Turtle’s state::
* Settings for measurement::
* Pen control: Pen control<2>.
* Turtle state::
* Using events::
* Special Turtle methods::
* Compound shapes::
File: python.info, Node: Turtle motion, Next: Tell Turtle’s state, Up: Methods of RawTurtle/Turtle and corresponding functions
5.25.1.18 Turtle motion
.......................
-- Function: turtle.forward (distance)
-- Function: turtle.fd (distance)
Parameters: ‘distance’ – a number (integer or float)
Move the turtle forward by the specified 'distance', in the
direction the turtle is headed.
>>> turtle.position()
(0.00,0.00)
>>> turtle.forward(25)
>>> turtle.position()
(25.00,0.00)
>>> turtle.forward(-75)
>>> turtle.position()
(-50.00,0.00)
-- Function: turtle.back (distance)
-- Function: turtle.bk (distance)
-- Function: turtle.backward (distance)
Parameters: ‘distance’ – a number
Move the turtle backward by 'distance', opposite to the direction
the turtle is headed. Do not change the turtle’s heading.
>>> turtle.position()
(0.00,0.00)
>>> turtle.backward(30)
>>> turtle.position()
(-30.00,0.00)
-- Function: turtle.right (angle)
-- Function: turtle.rt (angle)
Parameters: ‘angle’ – a number (integer or float)
Turn turtle right by 'angle' units. (Units are by default degrees,
but can be set via the *note degrees(): 3d90. and *note radians():
3d91. functions.) Angle orientation depends on the turtle mode,
see *note mode(): 3dd0.
>>> turtle.heading()
22.0
>>> turtle.right(45)
>>> turtle.heading()
337.0
-- Function: turtle.left (angle)
-- Function: turtle.lt (angle)
Parameters: ‘angle’ – a number (integer or float)
Turn turtle left by 'angle' units. (Units are by default degrees,
but can be set via the *note degrees(): 3d90. and *note radians():
3d91. functions.) Angle orientation depends on the turtle mode,
see *note mode(): 3dd0.
>>> turtle.heading()
22.0
>>> turtle.left(45)
>>> turtle.heading()
67.0
-- Function: turtle.goto (x, y=None)
-- Function: turtle.setpos (x, y=None)
-- Function: turtle.setposition (x, y=None)
Parameters:
* ‘x’ – a number or a pair/vector of numbers
* ‘y’ – a number or ‘None’
If 'y' is ‘None’, 'x' must be a pair of coordinates or a *note
Vec2D: 3ddf. (e.g. as returned by *note pos(): 3d8a.).
Move turtle to an absolute position. If the pen is down, draw
line. Do not change the turtle’s orientation.
>>> tp = turtle.pos()
>>> tp
(0.00,0.00)
>>> turtle.setpos(60,30)
>>> turtle.pos()
(60.00,30.00)
>>> turtle.setpos((20,80))
>>> turtle.pos()
(20.00,80.00)
>>> turtle.setpos(tp)
>>> turtle.pos()
(0.00,0.00)
-- Function: turtle.teleport (x, y=None, *, fill_gap=False)
Parameters:
* ‘x’ – a number or ‘None’
* ‘y’ – a number or ‘None’
* ‘fill_gap’ – a boolean
Move turtle to an absolute position. Unlike goto(x, y), a line
will not be drawn. The turtle’s orientation does not change. If
currently filling, the polygon(s) teleported from will be filled
after leaving, and filling will begin again after teleporting.
This can be disabled with fill_gap=True, which makes the imaginary
line traveled during teleporting act as a fill barrier like in
goto(x, y).
>>> tp = turtle.pos()
>>> tp
(0.00,0.00)
>>> turtle.teleport(60)
>>> turtle.pos()
(60.00,0.00)
>>> turtle.teleport(y=10)
>>> turtle.pos()
(60.00,10.00)
>>> turtle.teleport(20, 30)
>>> turtle.pos()
(20.00,30.00)
Added in version 3.12.
-- Function: turtle.setx (x)
Parameters: ‘x’ – a number (integer or float)
Set the turtle’s first coordinate to 'x', leave second coordinate
unchanged.
>>> turtle.position()
(0.00,240.00)
>>> turtle.setx(10)
>>> turtle.position()
(10.00,240.00)
-- Function: turtle.sety (y)
Parameters: ‘y’ – a number (integer or float)
Set the turtle’s second coordinate to 'y', leave first coordinate
unchanged.
>>> turtle.position()
(0.00,40.00)
>>> turtle.sety(-10)
>>> turtle.position()
(0.00,-10.00)
-- Function: turtle.setheading (to_angle)
-- Function: turtle.seth (to_angle)
Parameters: ‘to_angle’ – a number (integer or float)
Set the orientation of the turtle to 'to_angle'. Here are some
common directions in degrees:
standard mode logo mode
-------------------------------------------------
0 - east 0 - north
90 - north 90 - east
180 - west 180 - south
270 - south 270 - west
>>> turtle.setheading(90)
>>> turtle.heading()
90.0
-- Function: turtle.home ()
Move turtle to the origin – coordinates (0,0) – and set its heading
to its start-orientation (which depends on the mode, see *note
mode(): 3dd0.).
>>> turtle.heading()
90.0
>>> turtle.position()
(0.00,-10.00)
>>> turtle.home()
>>> turtle.position()
(0.00,0.00)
>>> turtle.heading()
0.0
-- Function: turtle.circle (radius, extent=None, steps=None)
Parameters:
* ‘radius’ – a number
* ‘extent’ – a number (or ‘None’)
* ‘steps’ – an integer (or ‘None’)
Draw a circle with given 'radius'. The center is 'radius' units
left of the turtle; 'extent' – an angle – determines which part of
the circle is drawn. If 'extent' is not given, draw the entire
circle. If 'extent' is not a full circle, one endpoint of the arc
is the current pen position. Draw the arc in counterclockwise
direction if 'radius' is positive, otherwise in clockwise
direction. Finally the direction of the turtle is changed by the
amount of 'extent'.
As the circle is approximated by an inscribed regular polygon,
'steps' determines the number of steps to use. If not given, it
will be calculated automatically. May be used to draw regular
polygons.
>>> turtle.home()
>>> turtle.position()
(0.00,0.00)
>>> turtle.heading()
0.0
>>> turtle.circle(50)
>>> turtle.position()
(-0.00,0.00)
>>> turtle.heading()
0.0
>>> turtle.circle(120, 180) # draw a semicircle
>>> turtle.position()
(0.00,240.00)
>>> turtle.heading()
180.0
-- Function: turtle.dot (size=None, *color)
Parameters:
* ‘size’ – an integer >= 1 (if given)
* ‘color’ – a colorstring or a numeric color tuple
Draw a circular dot with diameter 'size', using 'color'. If 'size'
is not given, the maximum of pensize+4 and 2*pensize is used.
>>> turtle.home()
>>> turtle.dot()
>>> turtle.fd(50); turtle.dot(20, "blue"); turtle.fd(50)
>>> turtle.position()
(100.00,-0.00)
>>> turtle.heading()
0.0
-- Function: turtle.stamp ()
Stamp a copy of the turtle shape onto the canvas at the current
turtle position. Return a stamp_id for that stamp, which can be
used to delete it by calling ‘clearstamp(stamp_id)’.
>>> turtle.color("blue")
>>> stamp_id = turtle.stamp()
>>> turtle.fd(50)
-- Function: turtle.clearstamp (stampid)
Parameters: ‘stampid’ – an integer, must be return value of
previous *note stamp(): 3d84. call
Delete stamp with given 'stampid'.
>>> turtle.position()
(150.00,-0.00)
>>> turtle.color("blue")
>>> astamp = turtle.stamp()
>>> turtle.fd(50)
>>> turtle.position()
(200.00,-0.00)
>>> turtle.clearstamp(astamp)
>>> turtle.position()
(200.00,-0.00)
-- Function: turtle.clearstamps (n=None)
Parameters: ‘n’ – an integer (or ‘None’)
Delete all or first/last 'n' of turtle’s stamps. If 'n' is ‘None’,
delete all stamps, if 'n' > 0 delete first 'n' stamps, else if 'n'
< 0 delete last 'n' stamps.
>>> for i in range(8):
... unused_stamp_id = turtle.stamp()
... turtle.fd(30)
>>> turtle.clearstamps(2)
>>> turtle.clearstamps(-2)
>>> turtle.clearstamps()
-- Function: turtle.undo ()
Undo (repeatedly) the last turtle action(s). Number of available
undo actions is determined by the size of the undobuffer.
>>> for i in range(4):
... turtle.fd(50); turtle.lt(80)
...
>>> for i in range(8):
... turtle.undo()
-- Function: turtle.speed (speed=None)
Parameters: ‘speed’ – an integer in the range 0..10 or a
speedstring (see below)
Set the turtle’s speed to an integer value in the range 0..10. If
no argument is given, return current speed.
If input is a number greater than 10 or smaller than 0.5, speed is
set to 0. Speedstrings are mapped to speedvalues as follows:
* “fastest”: 0
* “fast”: 10
* “normal”: 6
* “slow”: 3
* “slowest”: 1
Speeds from 1 to 10 enforce increasingly faster animation of line
drawing and turtle turning.
Attention: 'speed' = 0 means that 'no' animation takes place.
forward/back makes turtle jump and likewise left/right make the
turtle turn instantly.
>>> turtle.speed()
3
>>> turtle.speed('normal')
>>> turtle.speed()
6
>>> turtle.speed(9)
>>> turtle.speed()
9
File: python.info, Node: Tell Turtle’s state, Next: Settings for measurement, Prev: Turtle motion, Up: Methods of RawTurtle/Turtle and corresponding functions
5.25.1.19 Tell Turtle’s state
.............................
-- Function: turtle.position ()
-- Function: turtle.pos ()
Return the turtle’s current location (x,y) (as a *note Vec2D: 3ddf.
vector).
>>> turtle.pos()
(440.00,-0.00)
-- Function: turtle.towards (x, y=None)
Parameters:
* ‘x’ – a number or a pair/vector of numbers or a turtle
instance
* ‘y’ – a number if 'x' is a number, else ‘None’
Return the angle between the line from turtle position to position
specified by (x,y), the vector or the other turtle. This depends
on the turtle’s start orientation which depends on the mode -
“standard”/”world” or “logo”.
>>> turtle.goto(10, 10)
>>> turtle.towards(0,0)
225.0
-- Function: turtle.xcor ()
Return the turtle’s x coordinate.
>>> turtle.home()
>>> turtle.left(50)
>>> turtle.forward(100)
>>> turtle.pos()
(64.28,76.60)
>>> print(round(turtle.xcor(), 5))
64.27876
-- Function: turtle.ycor ()
Return the turtle’s y coordinate.
>>> turtle.home()
>>> turtle.left(60)
>>> turtle.forward(100)
>>> print(turtle.pos())
(50.00,86.60)
>>> print(round(turtle.ycor(), 5))
86.60254
-- Function: turtle.heading ()
Return the turtle’s current heading (value depends on the turtle
mode, see *note mode(): 3dd0.).
>>> turtle.home()
>>> turtle.left(67)
>>> turtle.heading()
67.0
-- Function: turtle.distance (x, y=None)
Parameters:
* ‘x’ – a number or a pair/vector of numbers or a turtle
instance
* ‘y’ – a number if 'x' is a number, else ‘None’
Return the distance from the turtle to (x,y), the given vector, or
the given other turtle, in turtle step units.
>>> turtle.home()
>>> turtle.distance(30,40)
50.0
>>> turtle.distance((30,40))
50.0
>>> joe = Turtle()
>>> joe.forward(77)
>>> turtle.distance(joe)
77.0
File: python.info, Node: Settings for measurement, Next: Pen control<2>, Prev: Tell Turtle’s state, Up: Methods of RawTurtle/Turtle and corresponding functions
5.25.1.20 Settings for measurement
..................................
-- Function: turtle.degrees (fullcircle=360.0)
Parameters: ‘fullcircle’ – a number
Set angle measurement units, i.e. set number of “degrees” for a
full circle. Default value is 360 degrees.
>>> turtle.home()
>>> turtle.left(90)
>>> turtle.heading()
90.0
>>> # Change angle measurement unit to grad (also known as gon,
>>> # grade, or gradian and equals 1/100-th of the right angle.)
>>> turtle.degrees(400.0)
>>> turtle.heading()
100.0
>>> turtle.degrees(360)
>>> turtle.heading()
90.0
-- Function: turtle.radians ()
Set the angle measurement units to radians. Equivalent to
‘degrees(2*math.pi)’.
>>> turtle.home()
>>> turtle.left(90)
>>> turtle.heading()
90.0
>>> turtle.radians()
>>> turtle.heading()
1.5707963267948966
File: python.info, Node: Pen control<2>, Next: Turtle state, Prev: Settings for measurement, Up: Methods of RawTurtle/Turtle and corresponding functions
5.25.1.21 Pen control
.....................
* Menu:
* Drawing state::
* Color control::
* Filling::
* More drawing control::
File: python.info, Node: Drawing state, Next: Color control, Up: Pen control<2>
5.25.1.22 Drawing state
.......................
-- Function: turtle.pendown ()
-- Function: turtle.pd ()
-- Function: turtle.down ()
Pull the pen down – drawing when moving.
-- Function: turtle.penup ()
-- Function: turtle.pu ()
-- Function: turtle.up ()
Pull the pen up – no drawing when moving.
-- Function: turtle.pensize (width=None)
-- Function: turtle.width (width=None)
Parameters: ‘width’ – a positive number
Set the line thickness to 'width' or return it. If resizemode is
set to “auto” and turtleshape is a polygon, that polygon is drawn
with the same line thickness. If no argument is given, the current
pensize is returned.
>>> turtle.pensize()
1
>>> turtle.pensize(10) # from here on lines of width 10 are drawn
-- Function: turtle.pen (pen=None, **pendict)
Parameters:
* ‘pen’ – a dictionary with some or all of the below listed keys
* ‘pendict’ – one or more keyword-arguments with the below
listed keys as keywords
Return or set the pen’s attributes in a “pen-dictionary” with the
following key/value pairs:
* “shown”: True/False
* “pendown”: True/False
* “pencolor”: color-string or color-tuple
* “fillcolor”: color-string or color-tuple
* “pensize”: positive number
* “speed”: number in range 0..10
* “resizemode”: “auto” or “user” or “noresize”
* “stretchfactor”: (positive number, positive number)
* “outline”: positive number
* “tilt”: number
This dictionary can be used as argument for a subsequent call to
*note pen(): 3d9a. to restore the former pen-state. Moreover one
or more of these attributes can be provided as keyword-arguments.
This can be used to set several pen attributes in one statement.
>>> turtle.pen(fillcolor="black", pencolor="red", pensize=10)
>>> sorted(turtle.pen().items())
[('fillcolor', 'black'), ('outline', 1), ('pencolor', 'red'),
('pendown', True), ('pensize', 10), ('resizemode', 'noresize'),
('shearfactor', 0.0), ('shown', True), ('speed', 9),
('stretchfactor', (1.0, 1.0)), ('tilt', 0.0)]
>>> penstate=turtle.pen()
>>> turtle.color("yellow", "")
>>> turtle.penup()
>>> sorted(turtle.pen().items())[:3]
[('fillcolor', ''), ('outline', 1), ('pencolor', 'yellow')]
>>> turtle.pen(penstate, fillcolor="green")
>>> sorted(turtle.pen().items())[:3]
[('fillcolor', 'green'), ('outline', 1), ('pencolor', 'red')]
-- Function: turtle.isdown ()
Return ‘True’ if pen is down, ‘False’ if it’s up.
>>> turtle.penup()
>>> turtle.isdown()
False
>>> turtle.pendown()
>>> turtle.isdown()
True
File: python.info, Node: Color control, Next: Filling, Prev: Drawing state, Up: Pen control<2>
5.25.1.23 Color control
.......................
-- Function: turtle.pencolor (*args)
Return or set the pencolor.
Four input formats are allowed:
‘pencolor()’
Return the current pencolor as color specification string or
as a tuple (see example). May be used as input to another
color/pencolor/fillcolor call.
‘pencolor(colorstring)’
Set pencolor to 'colorstring', which is a Tk color
specification string, such as ‘"red"’, ‘"yellow"’, or
‘"#33cc8c"’.
‘pencolor((r, g, b))’
Set pencolor to the RGB color represented by the tuple of 'r',
'g', and 'b'. Each of 'r', 'g', and 'b' must be in the range
0..colormode, where colormode is either 1.0 or 255 (see *note
colormode(): 3dd1.).
‘pencolor(r, g, b)’
Set pencolor to the RGB color represented by 'r', 'g', and
'b'. Each of 'r', 'g', and 'b' must be in the range
0..colormode.
If turtleshape is a polygon, the outline of that polygon is drawn
with the newly set pencolor.
>>> colormode()
1.0
>>> turtle.pencolor()
'red'
>>> turtle.pencolor("brown")
>>> turtle.pencolor()
'brown'
>>> tup = (0.2, 0.8, 0.55)
>>> turtle.pencolor(tup)
>>> turtle.pencolor()
(0.2, 0.8, 0.5490196078431373)
>>> colormode(255)
>>> turtle.pencolor()
(51.0, 204.0, 140.0)
>>> turtle.pencolor('#32c18f')
>>> turtle.pencolor()
(50.0, 193.0, 143.0)
-- Function: turtle.fillcolor (*args)
Return or set the fillcolor.
Four input formats are allowed:
‘fillcolor()’
Return the current fillcolor as color specification string,
possibly in tuple format (see example). May be used as input
to another color/pencolor/fillcolor call.
‘fillcolor(colorstring)’
Set fillcolor to 'colorstring', which is a Tk color
specification string, such as ‘"red"’, ‘"yellow"’, or
‘"#33cc8c"’.
‘fillcolor((r, g, b))’
Set fillcolor to the RGB color represented by the tuple of
'r', 'g', and 'b'. Each of 'r', 'g', and 'b' must be in the
range 0..colormode, where colormode is either 1.0 or 255 (see
*note colormode(): 3dd1.).
‘fillcolor(r, g, b)’
Set fillcolor to the RGB color represented by 'r', 'g', and
'b'. Each of 'r', 'g', and 'b' must be in the range
0..colormode.
If turtleshape is a polygon, the interior of that polygon is drawn
with the newly set fillcolor.
>>> turtle.fillcolor("violet")
>>> turtle.fillcolor()
'violet'
>>> turtle.pencolor()
(50.0, 193.0, 143.0)
>>> turtle.fillcolor((50, 193, 143)) # Integers, not floats
>>> turtle.fillcolor()
(50.0, 193.0, 143.0)
>>> turtle.fillcolor('#ffffff')
>>> turtle.fillcolor()
(255.0, 255.0, 255.0)
-- Function: turtle.color (*args)
Return or set pencolor and fillcolor.
Several input formats are allowed. They use 0 to 3 arguments as
follows:
‘color()’
Return the current pencolor and the current fillcolor as a
pair of color specification strings or tuples as returned by
*note pencolor(): 3d9d. and *note fillcolor(): 3d9e.
‘color(colorstring)’, ‘color((r,g,b))’, ‘color(r,g,b)’
Inputs as in *note pencolor(): 3d9d, set both, fillcolor and
pencolor, to the given value.
‘color(colorstring1, colorstring2)’, ‘color((r1,g1,b1), (r2,g2,b2))’
Equivalent to ‘pencolor(colorstring1)’ and
‘fillcolor(colorstring2)’ and analogously if the other input
format is used.
If turtleshape is a polygon, outline and interior of that polygon
is drawn with the newly set colors.
>>> turtle.color("red", "green")
>>> turtle.color()
('red', 'green')
>>> color("#285078", "#a0c8f0")
>>> color()
((40.0, 80.0, 120.0), (160.0, 200.0, 240.0))
See also: Screen method *note colormode(): 3dd1.
File: python.info, Node: Filling, Next: More drawing control, Prev: Color control, Up: Pen control<2>
5.25.1.24 Filling
.................
-- Function: turtle.filling ()
Return fillstate (‘True’ if filling, ‘False’ else).
>>> turtle.begin_fill()
>>> if turtle.filling():
... turtle.pensize(5)
... else:
... turtle.pensize(3)
-- Function: turtle.begin_fill ()
To be called just before drawing a shape to be filled.
-- Function: turtle.end_fill ()
Fill the shape drawn after the last call to *note begin_fill():
3da0.
Whether or not overlap regions for self-intersecting polygons or
multiple shapes are filled depends on the operating system
graphics, type of overlap, and number of overlaps. For example,
the Turtle star above may be either all yellow or have some white
regions.
>>> turtle.color("black", "red")
>>> turtle.begin_fill()
>>> turtle.circle(80)
>>> turtle.end_fill()
File: python.info, Node: More drawing control, Prev: Filling, Up: Pen control<2>
5.25.1.25 More drawing control
..............................
-- Function: turtle.reset ()
Delete the turtle’s drawings from the screen, re-center the turtle
and set variables to the default values.
>>> turtle.goto(0,-22)
>>> turtle.left(100)
>>> turtle.position()
(0.00,-22.00)
>>> turtle.heading()
100.0
>>> turtle.reset()
>>> turtle.position()
(0.00,0.00)
>>> turtle.heading()
0.0
-- Function: turtle.clear ()
Delete the turtle’s drawings from the screen. Do not move turtle.
State and position of the turtle as well as drawings of other
turtles are not affected.
-- Function: turtle.write (arg, move=False, align='left',
font=('Arial', 8, 'normal'))
Parameters:
* ‘arg’ – object to be written to the TurtleScreen
* ‘move’ – True/False
* ‘align’ – one of the strings “left”, “center” or right”
* ‘font’ – a triple (fontname, fontsize, fonttype)
Write text - the string representation of 'arg' - at the current
turtle position according to 'align' (“left”, “center” or “right”)
and with the given font. If 'move' is true, the pen is moved to
the bottom-right corner of the text. By default, 'move' is
‘False’.
>>> turtle.write("Home = ", True, align="center")
>>> turtle.write((0,0), True)
File: python.info, Node: Turtle state, Next: Using events, Prev: Pen control<2>, Up: Methods of RawTurtle/Turtle and corresponding functions
5.25.1.26 Turtle state
......................
* Menu:
* Visibility::
* Appearance::
File: python.info, Node: Visibility, Next: Appearance, Up: Turtle state
5.25.1.27 Visibility
....................
-- Function: turtle.hideturtle ()
-- Function: turtle.ht ()
Make the turtle invisible. It’s a good idea to do this while
you’re in the middle of doing some complex drawing, because hiding
the turtle speeds up the drawing observably.
>>> turtle.hideturtle()
-- Function: turtle.showturtle ()
-- Function: turtle.st ()
Make the turtle visible.
>>> turtle.showturtle()
-- Function: turtle.isvisible ()
Return ‘True’ if the Turtle is shown, ‘False’ if it’s hidden.
>>> turtle.hideturtle()
>>> turtle.isvisible()
False
>>> turtle.showturtle()
>>> turtle.isvisible()
True
File: python.info, Node: Appearance, Prev: Visibility, Up: Turtle state
5.25.1.28 Appearance
....................
-- Function: turtle.shape (name=None)
Parameters: ‘name’ – a string which is a valid shapename
Set turtle shape to shape with given 'name' or, if name is not
given, return name of current shape. Shape with 'name' must exist
in the TurtleScreen’s shape dictionary. Initially there are the
following polygon shapes: “arrow”, “turtle”, “circle”, “square”,
“triangle”, “classic”. To learn about how to deal with shapes see
Screen method *note register_shape(): 3dd4.
>>> turtle.shape()
'classic'
>>> turtle.shape("turtle")
>>> turtle.shape()
'turtle'
-- Function: turtle.resizemode (rmode=None)
Parameters: ‘rmode’ – one of the strings “auto”, “user”, “noresize”
Set resizemode to one of the values: “auto”, “user”, “noresize”.
If 'rmode' is not given, return current resizemode. Different
resizemodes have the following effects:
- “auto”: adapts the appearance of the turtle corresponding to
the value of pensize.
- “user”: adapts the appearance of the turtle according to the
values of stretchfactor and outlinewidth (outline), which are
set by *note shapesize(): 3dac.
- “noresize”: no adaption of the turtle’s appearance takes
place.
‘resizemode("user")’ is called by *note shapesize(): 3dac. when
used with arguments.
>>> turtle.resizemode()
'noresize'
>>> turtle.resizemode("auto")
>>> turtle.resizemode()
'auto'
-- Function: turtle.shapesize (stretch_wid=None, stretch_len=None,
outline=None)
-- Function: turtle.turtlesize (stretch_wid=None, stretch_len=None,
outline=None)
Parameters:
* ‘stretch_wid’ – positive number
* ‘stretch_len’ – positive number
* ‘outline’ – positive number
Return or set the pen’s attributes x/y-stretchfactors and/or
outline. Set resizemode to “user”. If and only if resizemode is
set to “user”, the turtle will be displayed stretched according to
its stretchfactors: 'stretch_wid' is stretchfactor perpendicular to
its orientation, 'stretch_len' is stretchfactor in direction of its
orientation, 'outline' determines the width of the shape’s outline.
>>> turtle.shapesize()
(1.0, 1.0, 1)
>>> turtle.resizemode("user")
>>> turtle.shapesize(5, 5, 12)
>>> turtle.shapesize()
(5, 5, 12)
>>> turtle.shapesize(outline=8)
>>> turtle.shapesize()
(5, 5, 8)
-- Function: turtle.shearfactor (shear=None)
Parameters: ‘shear’ – number (optional)
Set or return the current shearfactor. Shear the turtleshape
according to the given shearfactor shear, which is the tangent of
the shear angle. Do 'not' change the turtle’s heading (direction
of movement). If shear is not given: return the current
shearfactor, i. e. the tangent of the shear angle, by which lines
parallel to the heading of the turtle are sheared.
>>> turtle.shape("circle")
>>> turtle.shapesize(5,2)
>>> turtle.shearfactor(0.5)
>>> turtle.shearfactor()
0.5
-- Function: turtle.tilt (angle)
Parameters: ‘angle’ – a number
Rotate the turtleshape by 'angle' from its current tilt-angle, but
do 'not' change the turtle’s heading (direction of movement).
>>> turtle.reset()
>>> turtle.shape("circle")
>>> turtle.shapesize(5,2)
>>> turtle.tilt(30)
>>> turtle.fd(50)
>>> turtle.tilt(30)
>>> turtle.fd(50)
-- Function: turtle.tiltangle (angle=None)
Parameters: ‘angle’ – a number (optional)
Set or return the current tilt-angle. If angle is given, rotate
the turtleshape to point in the direction specified by angle,
regardless of its current tilt-angle. Do 'not' change the turtle’s
heading (direction of movement). If angle is not given: return the
current tilt-angle, i. e. the angle between the orientation of
the turtleshape and the heading of the turtle (its direction of
movement).
>>> turtle.reset()
>>> turtle.shape("circle")
>>> turtle.shapesize(5,2)
>>> turtle.tilt(45)
>>> turtle.tiltangle()
45.0
-- Function: turtle.shapetransform (t11=None, t12=None, t21=None,
t22=None)
Parameters:
* ‘t11’ – a number (optional)
* ‘t12’ – a number (optional)
* ‘t21’ – a number (optional)
* ‘t12’ – a number (optional)
Set or return the current transformation matrix of the turtle
shape.
If none of the matrix elements are given, return the transformation
matrix as a tuple of 4 elements. Otherwise set the given elements
and transform the turtleshape according to the matrix consisting of
first row t11, t12 and second row t21, t22. The determinant t11 *
t22 - t12 * t21 must not be zero, otherwise an error is raised.
Modify stretchfactor, shearfactor and tiltangle according to the
given matrix.
>>> turtle = Turtle()
>>> turtle.shape("square")
>>> turtle.shapesize(4,2)
>>> turtle.shearfactor(-0.5)
>>> turtle.shapetransform()
(4.0, -1.0, -0.0, 2.0)
-- Function: turtle.get_shapepoly ()
Return the current shape polygon as tuple of coordinate pairs.
This can be used to define a new shape or components of a compound
shape.
>>> turtle.shape("square")
>>> turtle.shapetransform(4, -1, 0, 2)
>>> turtle.get_shapepoly()
((50, -20), (30, 20), (-50, 20), (-30, -20))
File: python.info, Node: Using events, Next: Special Turtle methods, Prev: Turtle state, Up: Methods of RawTurtle/Turtle and corresponding functions
5.25.1.29 Using events
......................
-- Function: turtle.onclick (fun, btn=1, add=None)
Parameters:
* ‘fun’ – a function with two arguments which will be called
with the coordinates of the clicked point on the canvas
* ‘btn’ – number of the mouse-button, defaults to 1 (left mouse
button)
* ‘add’ – ‘True’ or ‘False’ – if ‘True’, a new binding will be
added, otherwise it will replace a former binding
Bind 'fun' to mouse-click events on this turtle. If 'fun' is
‘None’, existing bindings are removed. Example for the anonymous
turtle, i.e. the procedural way:
>>> def turn(x, y):
... left(180)
...
>>> onclick(turn) # Now clicking into the turtle will turn it.
>>> onclick(None) # event-binding will be removed
-- Function: turtle.onrelease (fun, btn=1, add=None)
Parameters:
* ‘fun’ – a function with two arguments which will be called
with the coordinates of the clicked point on the canvas
* ‘btn’ – number of the mouse-button, defaults to 1 (left mouse
button)
* ‘add’ – ‘True’ or ‘False’ – if ‘True’, a new binding will be
added, otherwise it will replace a former binding
Bind 'fun' to mouse-button-release events on this turtle. If 'fun'
is ‘None’, existing bindings are removed.
>>> class MyTurtle(Turtle):
... def glow(self,x,y):
... self.fillcolor("red")
... def unglow(self,x,y):
... self.fillcolor("")
...
>>> turtle = MyTurtle()
>>> turtle.onclick(turtle.glow) # clicking on turtle turns fillcolor red,
>>> turtle.onrelease(turtle.unglow) # releasing turns it to transparent.
-- Function: turtle.ondrag (fun, btn=1, add=None)
Parameters:
* ‘fun’ – a function with two arguments which will be called
with the coordinates of the clicked point on the canvas
* ‘btn’ – number of the mouse-button, defaults to 1 (left mouse
button)
* ‘add’ – ‘True’ or ‘False’ – if ‘True’, a new binding will be
added, otherwise it will replace a former binding
Bind 'fun' to mouse-move events on this turtle. If 'fun' is
‘None’, existing bindings are removed.
Remark: Every sequence of mouse-move-events on a turtle is preceded
by a mouse-click event on that turtle.
>>> turtle.ondrag(turtle.goto)
Subsequently, clicking and dragging the Turtle will move it across
the screen thereby producing handdrawings (if pen is down).
File: python.info, Node: Special Turtle methods, Next: Compound shapes, Prev: Using events, Up: Methods of RawTurtle/Turtle and corresponding functions
5.25.1.30 Special Turtle methods
................................
-- Function: turtle.begin_poly ()
Start recording the vertices of a polygon. Current turtle position
is first vertex of polygon.
-- Function: turtle.end_poly ()
Stop recording the vertices of a polygon. Current turtle position
is last vertex of polygon. This will be connected with the first
vertex.
-- Function: turtle.get_poly ()
Return the last recorded polygon.
>>> turtle.home()
>>> turtle.begin_poly()
>>> turtle.fd(100)
>>> turtle.left(20)
>>> turtle.fd(30)
>>> turtle.left(60)
>>> turtle.fd(50)
>>> turtle.end_poly()
>>> p = turtle.get_poly()
>>> register_shape("myFavouriteShape", p)
-- Function: turtle.clone ()
Create and return a clone of the turtle with same position, heading
and turtle properties.
>>> mick = Turtle()
>>> joe = mick.clone()
-- Function: turtle.getturtle ()
-- Function: turtle.getpen ()
Return the Turtle object itself. Only reasonable use: as a
function to return the “anonymous turtle”:
>>> pet = getturtle()
>>> pet.fd(50)
>>> pet
-- Function: turtle.getscreen ()
Return the *note TurtleScreen: 3dec. object the turtle is drawing
on. TurtleScreen methods can then be called for that object.
>>> ts = turtle.getscreen()
>>> ts
>>> ts.bgcolor("pink")
-- Function: turtle.setundobuffer (size)
Parameters: ‘size’ – an integer or ‘None’
Set or disable undobuffer. If 'size' is an integer, an empty
undobuffer of given size is installed. 'size' gives the maximum
number of turtle actions that can be undone by the *note undo():
3d87. method/function. If 'size' is ‘None’, the undobuffer is
disabled.
>>> turtle.setundobuffer(42)
-- Function: turtle.undobufferentries ()
Return number of entries in the undobuffer.
>>> while undobufferentries():
... undo()
File: python.info, Node: Compound shapes, Prev: Special Turtle methods, Up: Methods of RawTurtle/Turtle and corresponding functions
5.25.1.31 Compound shapes
.........................
To use compound turtle shapes, which consist of several polygons of
different color, you must use the helper class *note Shape: 3def.
explicitly as described below:
1. Create an empty Shape object of type “compound”.
2. Add as many components to this object as desired, using the *note
addcomponent(): 3df0. method.
For example:
>>> s = Shape("compound")
>>> poly1 = ((0,0),(10,-5),(0,10),(-10,-5))
>>> s.addcomponent(poly1, "red", "blue")
>>> poly2 = ((0,0),(10,-5),(-10,-5))
>>> s.addcomponent(poly2, "blue", "red")
3. Now add the Shape to the Screen’s shapelist and use it:
>>> register_shape("myshape", s)
>>> shape("myshape")
Note: The *note Shape: 3def. class is used internally by the *note
register_shape(): 3dd4. method in different ways. The application
programmer has to deal with the Shape class 'only' when using
compound shapes like shown above!
File: python.info, Node: Methods of TurtleScreen/Screen and corresponding functions, Next: Public classes, Prev: Methods of RawTurtle/Turtle and corresponding functions, Up: turtle — Turtle graphics
5.25.1.32 Methods of TurtleScreen/Screen and corresponding functions
....................................................................
Most of the examples in this section refer to a TurtleScreen instance
called ‘screen’.
* Menu:
* Window control::
* Animation control::
* Using screen events::
* Input methods::
* Settings and special methods::
* Methods specific to Screen, not inherited from TurtleScreen: Methods specific to Screen not inherited from TurtleScreen.
File: python.info, Node: Window control, Next: Animation control, Up: Methods of TurtleScreen/Screen and corresponding functions
5.25.1.33 Window control
........................
-- Function: turtle.bgcolor (*args)
Parameters: ‘args’ – a color string or three numbers in the range
0..colormode or a 3-tuple of such numbers
Set or return background color of the TurtleScreen.
>>> screen.bgcolor("orange")
>>> screen.bgcolor()
'orange'
>>> screen.bgcolor("#800080")
>>> screen.bgcolor()
(128.0, 0.0, 128.0)
-- Function: turtle.bgpic (picname=None)
Parameters: ‘picname’ – a string, name of a gif-file or ‘"nopic"’,
or ‘None’
Set background image or return name of current backgroundimage. If
'picname' is a filename, set the corresponding image as background.
If 'picname' is ‘"nopic"’, delete background image, if present. If
'picname' is ‘None’, return the filename of the current
backgroundimage.
>>> screen.bgpic()
'nopic'
>>> screen.bgpic("landscape.gif")
>>> screen.bgpic()
"landscape.gif"
-- Function: turtle.clear ()
Note: This TurtleScreen method is available as a global
function only under the name ‘clearscreen’. The global
function ‘clear’ is a different one derived from the Turtle
method ‘clear’.
-- Function: turtle.clearscreen ()
Delete all drawings and all turtles from the TurtleScreen. Reset
the now empty TurtleScreen to its initial state: white background,
no background image, no event bindings and tracing on.
-- Function: turtle.reset ()
Note: This TurtleScreen method is available as a global
function only under the name ‘resetscreen’. The global
function ‘reset’ is another one derived from the Turtle method
‘reset’.
-- Function: turtle.resetscreen ()
Reset all Turtles on the Screen to their initial state.
-- Function: turtle.screensize (canvwidth=None, canvheight=None,
bg=None)
Parameters:
* ‘canvwidth’ – positive integer, new width of canvas in pixels
* ‘canvheight’ – positive integer, new height of canvas in
pixels
* ‘bg’ – colorstring or color-tuple, new background color
If no arguments are given, return current (canvaswidth,
canvasheight). Else resize the canvas the turtles are drawing on.
Do not alter the drawing window. To observe hidden parts of the
canvas, use the scrollbars. With this method, one can make visible
those parts of a drawing which were outside the canvas before.
>>> screen.screensize()
(400, 300)
>>> screen.screensize(2000,1500)
>>> screen.screensize()
(2000, 1500)
e.g. to search for an erroneously escaped turtle ;-)
-- Function: turtle.setworldcoordinates (llx, lly, urx, ury)
Parameters:
* ‘llx’ – a number, x-coordinate of lower left corner of canvas
* ‘lly’ – a number, y-coordinate of lower left corner of canvas
* ‘urx’ – a number, x-coordinate of upper right corner of canvas
* ‘ury’ – a number, y-coordinate of upper right corner of canvas
Set up user-defined coordinate system and switch to mode “world” if
necessary. This performs a ‘screen.reset()’. If mode “world” is
already active, all drawings are redrawn according to the new
coordinates.
'ATTENTION': in user-defined coordinate systems angles may appear
distorted.
>>> screen.reset()
>>> screen.setworldcoordinates(-50,-7.5,50,7.5)
>>> for _ in range(72):
... left(10)
...
>>> for _ in range(8):
... left(45); fd(2) # a regular octagon
File: python.info, Node: Animation control, Next: Using screen events, Prev: Window control, Up: Methods of TurtleScreen/Screen and corresponding functions
5.25.1.34 Animation control
...........................
-- Function: turtle.delay (delay=None)
Parameters: ‘delay’ – positive integer
Set or return the drawing 'delay' in milliseconds. (This is
approximately the time interval between two consecutive canvas
updates.) The longer the drawing delay, the slower the animation.
Optional argument:
>>> screen.delay()
10
>>> screen.delay(5)
>>> screen.delay()
5
-- Function: turtle.tracer (n=None, delay=None)
Parameters:
* ‘n’ – nonnegative integer
* ‘delay’ – nonnegative integer
Turn turtle animation on/off and set delay for update drawings. If
'n' is given, only each n-th regular screen update is really
performed. (Can be used to accelerate the drawing of complex
graphics.) When called without arguments, returns the currently
stored value of n. Second argument sets delay value (see *note
delay(): 3dc5.).
>>> screen.tracer(8, 25)
>>> dist = 2
>>> for i in range(200):
... fd(dist)
... rt(90)
... dist += 2
-- Function: turtle.update ()
Perform a TurtleScreen update. To be used when tracer is turned
off.
See also the RawTurtle/Turtle method *note speed(): 3d88.
File: python.info, Node: Using screen events, Next: Input methods, Prev: Animation control, Up: Methods of TurtleScreen/Screen and corresponding functions
5.25.1.35 Using screen events
.............................
-- Function: turtle.listen (xdummy=None, ydummy=None)
Set focus on TurtleScreen (in order to collect key-events). Dummy
arguments are provided in order to be able to pass *note listen():
3dc8. to the onclick method.
-- Function: turtle.onkey (fun, key)
-- Function: turtle.onkeyrelease (fun, key)
Parameters:
* ‘fun’ – a function with no arguments or ‘None’
* ‘key’ – a string: key (e.g. “a”) or key-symbol (e.g.
“space”)
Bind 'fun' to key-release event of key. If 'fun' is ‘None’, event
bindings are removed. Remark: in order to be able to register
key-events, TurtleScreen must have the focus. (See method *note
listen(): 3dc8.)
>>> def f():
... fd(50)
... lt(60)
...
>>> screen.onkey(f, "Up")
>>> screen.listen()
-- Function: turtle.onkeypress (fun, key=None)
Parameters:
* ‘fun’ – a function with no arguments or ‘None’
* ‘key’ – a string: key (e.g. “a”) or key-symbol (e.g.
“space”)
Bind 'fun' to key-press event of key if key is given, or to any
key-press-event if no key is given. Remark: in order to be able to
register key-events, TurtleScreen must have focus. (See method
*note listen(): 3dc8.)
>>> def f():
... fd(50)
...
>>> screen.onkey(f, "Up")
>>> screen.listen()
-- Function: turtle.onclick (fun, btn=1, add=None)
-- Function: turtle.onscreenclick (fun, btn=1, add=None)
Parameters:
* ‘fun’ – a function with two arguments which will be called
with the coordinates of the clicked point on the canvas
* ‘btn’ – number of the mouse-button, defaults to 1 (left mouse
button)
* ‘add’ – ‘True’ or ‘False’ – if ‘True’, a new binding will be
added, otherwise it will replace a former binding
Bind 'fun' to mouse-click events on this screen. If 'fun' is
‘None’, existing bindings are removed.
Example for a TurtleScreen instance named ‘screen’ and a Turtle
instance named ‘turtle’:
>>> screen.onclick(turtle.goto) # Subsequently clicking into the TurtleScreen will
>>> # make the turtle move to the clicked point.
>>> screen.onclick(None) # remove event binding again
Note: This TurtleScreen method is available as a global
function only under the name ‘onscreenclick’. The global
function ‘onclick’ is another one derived from the Turtle
method ‘onclick’.
-- Function: turtle.ontimer (fun, t=0)
Parameters:
* ‘fun’ – a function with no arguments
* ‘t’ – a number >= 0
Install a timer that calls 'fun' after 't' milliseconds.
>>> running = True
>>> def f():
... if running:
... fd(50)
... lt(60)
... screen.ontimer(f, 250)
>>> f() ### makes the turtle march around
>>> running = False
-- Function: turtle.mainloop ()
-- Function: turtle.done ()
Starts event loop - calling Tkinter’s mainloop function. Must be
the last statement in a turtle graphics program. Must 'not' be
used if a script is run from within IDLE in -n mode (No subprocess)
- for interactive use of turtle graphics.
>>> screen.mainloop()
File: python.info, Node: Input methods, Next: Settings and special methods, Prev: Using screen events, Up: Methods of TurtleScreen/Screen and corresponding functions
5.25.1.36 Input methods
.......................
-- Function: turtle.textinput (title, prompt)
Parameters:
* ‘title’ – string
* ‘prompt’ – string
Pop up a dialog window for input of a string. Parameter title is
the title of the dialog window, prompt is a text mostly describing
what information to input. Return the string input. If the dialog
is canceled, return ‘None’.
>>> screen.textinput("NIM", "Name of first player:")
-- Function: turtle.numinput (title, prompt, default=None, minval=None,
maxval=None)
Parameters:
* ‘title’ – string
* ‘prompt’ – string
* ‘default’ – number (optional)
* ‘minval’ – number (optional)
* ‘maxval’ – number (optional)
Pop up a dialog window for input of a number. title is the title
of the dialog window, prompt is a text mostly describing what
numerical information to input. default: default value, minval:
minimum value for input, maxval: maximum value for input. The
number input must be in the range minval .. maxval if these are
given. If not, a hint is issued and the dialog remains open for
correction. Return the number input. If the dialog is canceled,
return ‘None’.
>>> screen.numinput("Poker", "Your stakes:", 1000, minval=10, maxval=10000)
File: python.info, Node: Settings and special methods, Next: Methods specific to Screen not inherited from TurtleScreen, Prev: Input methods, Up: Methods of TurtleScreen/Screen and corresponding functions
5.25.1.37 Settings and special methods
......................................
-- Function: turtle.mode (mode=None)
Parameters: ‘mode’ – one of the strings “standard”, “logo” or
“world”
Set turtle mode (“standard”, “logo” or “world”) and perform reset.
If mode is not given, current mode is returned.
Mode “standard” is compatible with old *note turtle: 101. Mode
“logo” is compatible with most Logo turtle graphics. Mode “world”
uses user-defined “world coordinates”. 'Attention': in this mode
angles appear distorted if ‘x/y’ unit-ratio doesn’t equal 1.
Mode Initial turtle heading positive angles
-----------------------------------------------------------------------
“standard” to the right (east) counterclockwise
“logo” upward (north) clockwise
>>> mode("logo") # resets turtle heading to north
>>> mode()
'logo'
-- Function: turtle.colormode (cmode=None)
Parameters: ‘cmode’ – one of the values 1.0 or 255
Return the colormode or set it to 1.0 or 255. Subsequently 'r',
'g', 'b' values of color triples have to be in the range
0..*cmode*.
>>> screen.colormode(1)
>>> turtle.pencolor(240, 160, 80)
Traceback (most recent call last):
...
TurtleGraphicsError: bad color sequence: (240, 160, 80)
>>> screen.colormode()
1.0
>>> screen.colormode(255)
>>> screen.colormode()
255
>>> turtle.pencolor(240,160,80)
-- Function: turtle.getcanvas ()
Return the Canvas of this TurtleScreen. Useful for insiders who
know what to do with a Tkinter Canvas.
>>> cv = screen.getcanvas()
>>> cv
-- Function: turtle.getshapes ()
Return a list of names of all currently available turtle shapes.
>>> screen.getshapes()
['arrow', 'blank', 'circle', ..., 'turtle']
-- Function: turtle.register_shape (name, shape=None)
-- Function: turtle.addshape (name, shape=None)
There are three different ways to call this function:
1. 'name' is the name of a gif-file and 'shape' is ‘None’:
Install the corresponding image shape.
>>> screen.register_shape("turtle.gif")
Note: Image shapes 'do not' rotate when turning the
turtle, so they do not display the heading of the turtle!
2. 'name' is an arbitrary string and 'shape' is a tuple of pairs
of coordinates: Install the corresponding polygon shape.
>>> screen.register_shape("triangle", ((5,-3), (0,5), (-5,-3)))
3. 'name' is an arbitrary string and 'shape' is a (compound)
*note Shape: 3def. object: Install the corresponding compound
shape.
Add a turtle shape to TurtleScreen’s shapelist. Only thusly
registered shapes can be used by issuing the command
‘shape(shapename)’.
-- Function: turtle.turtles ()
Return the list of turtles on the screen.
>>> for turtle in screen.turtles():
... turtle.color("red")
-- Function: turtle.window_height ()
Return the height of the turtle window.
>>> screen.window_height()
480
-- Function: turtle.window_width ()
Return the width of the turtle window.
>>> screen.window_width()
640
File: python.info, Node: Methods specific to Screen not inherited from TurtleScreen, Prev: Settings and special methods, Up: Methods of TurtleScreen/Screen and corresponding functions
5.25.1.38 Methods specific to Screen, not inherited from TurtleScreen
.....................................................................
-- Function: turtle.bye ()
Shut the turtlegraphics window.
-- Function: turtle.exitonclick ()
Bind ‘bye()’ method to mouse clicks on the Screen.
If the value “using_IDLE” in the configuration dictionary is
‘False’ (default value), also enter mainloop. Remark: If IDLE with
the ‘-n’ switch (no subprocess) is used, this value should be set
to ‘True’ in ‘turtle.cfg’. In this case IDLE’s own mainloop is
active also for the client script.
-- Function: turtle.setup (width=_CFG['width'], height=_CFG['height'],
startx=_CFG['leftright'], starty=_CFG['topbottom'])
Set the size and position of the main window. Default values of
arguments are stored in the configuration dictionary and can be
changed via a ‘turtle.cfg’ file.
Parameters:
* ‘width’ – if an integer, a size in pixels, if a float, a
fraction of the screen; default is 50% of screen
* ‘height’ – if an integer, the height in pixels, if a float, a
fraction of the screen; default is 75% of screen
* ‘startx’ – if positive, starting position in pixels from the
left edge of the screen, if negative from the right edge, if
‘None’, center window horizontally
* ‘starty’ – if positive, starting position in pixels from the
top edge of the screen, if negative from the bottom edge, if
‘None’, center window vertically
>>> screen.setup (width=200, height=200, startx=0, starty=0)
>>> # sets window to 200x200 pixels, in upper left of screen
>>> screen.setup(width=.75, height=0.5, startx=None, starty=None)
>>> # sets window to 75% of screen by 50% of screen and centers
-- Function: turtle.title (titlestring)
Parameters: ‘titlestring’ – a string that is shown in the titlebar
of the turtle graphics window
Set title of turtle window to 'titlestring'.
>>> screen.title("Welcome to the turtle zoo!")
File: python.info, Node: Public classes, Next: Explanation<2>, Prev: Methods of TurtleScreen/Screen and corresponding functions, Up: turtle — Turtle graphics
5.25.1.39 Public classes
........................
-- Class: turtle.RawTurtle (canvas)
-- Class: turtle.RawPen (canvas)
Parameters: ‘canvas’ – a ‘tkinter.Canvas’, a *note ScrolledCanvas:
3dfc. or a *note TurtleScreen: 3dec.
Create a turtle. The turtle has all methods described above as
“methods of Turtle/RawTurtle”.
-- Class: turtle.Turtle
Subclass of RawTurtle, has the same interface but draws on a
default *note Screen: 3d6f. object created automatically when
needed for the first time.
-- Class: turtle.TurtleScreen (cv)
Parameters: ‘cv’ – a ‘tkinter.Canvas’
Provides screen oriented methods like *note bgcolor(): 3dbf. etc.
that are described above.
-- Class: turtle.Screen
Subclass of TurtleScreen, with *note four methods added: 3df8.
-- Class: turtle.ScrolledCanvas (master)
Parameters: ‘master’ – some Tkinter widget to contain the
ScrolledCanvas, i.e. a Tkinter-canvas with scrollbars added
Used by class Screen, which thus automatically provides a
ScrolledCanvas as playground for the turtles.
-- Class: turtle.Shape (type_, data)
Parameters: ‘type_’ – one of the strings “polygon”, “image”,
“compound”
Data structure modeling shapes. The pair ‘(type_, data)’ must
follow this specification:
'type_' 'data'
-------------------------------------------------------------------------------
“polygon” a polygon-tuple, i.e. a tuple of pairs of coordinates
“image” an image (in this form only used internally!)
“compound” ‘None’ (a compound shape has to be constructed using the
*note addcomponent(): 3df0. method)
-- Method: addcomponent (poly, fill, outline=None)
Parameters:
* ‘poly’ – a polygon, i.e. a tuple of pairs of numbers
* ‘fill’ – a color the 'poly' will be filled with
* ‘outline’ – a color for the poly’s outline (if given)
Example:
>>> poly = ((0,0),(10,-5),(0,10),(-10,-5))
>>> s = Shape("compound")
>>> s.addcomponent(poly, "red", "blue")
>>> # ... add more components and then use register_shape()
See *note Compound shapes: 3dee.
-- Class: turtle.Vec2D (x, y)
A two-dimensional vector class, used as a helper class for
implementing turtle graphics. May be useful for turtle graphics
programs too. Derived from tuple, so a vector is a tuple!
Provides (for 'a', 'b' vectors, 'k' number):
* ‘a + b’ vector addition
* ‘a - b’ vector subtraction
* ‘a * b’ inner product
* ‘k * a’ and ‘a * k’ multiplication with scalar
* ‘abs(a)’ absolute value of a
* ‘a.rotate(angle)’ rotation
File: python.info, Node: Explanation<2>, Next: Help and configuration, Prev: Public classes, Up: turtle — Turtle graphics
5.25.1.40 Explanation
.....................
A turtle object draws on a screen object, and there a number of key
classes in the turtle object-oriented interface that can be used to
create them and relate them to each other.
A *note Turtle: 3dfd. instance will automatically create a *note Screen:
3d6f. instance if one is not already present.
‘Turtle’ is a subclass of *note RawTurtle: 3dfa, which 'doesn’t'
automatically create a drawing surface - a 'canvas' will need to be
provided or created for it. The 'canvas' can be a ‘tkinter.Canvas’,
*note ScrolledCanvas: 3dfc. or *note TurtleScreen: 3dec.
*note TurtleScreen: 3dec. is the basic drawing surface for a turtle.
*note Screen: 3d6f. is a subclass of ‘TurtleScreen’, and includes *note
some additional methods: 3df8. for managing its appearance (including
size and title) and behaviour. ‘TurtleScreen’’s constructor needs a
‘tkinter.Canvas’ or a *note ScrolledCanvas: 3dfc. as an argument.
The functional interface for turtle graphics uses the various methods of
‘Turtle’ and ‘TurtleScreen’/‘Screen’. Behind the scenes, a screen
object is automatically created whenever a function derived from a
‘Screen’ method is called. Similarly, a turtle object is automatically
created whenever any of the functions derived from a Turtle method is
called.
To use multiple turtles on a screen, the object-oriented interface must
be used.
File: python.info, Node: Help and configuration, Next: turtledemo — Demo scripts, Prev: Explanation<2>, Up: turtle — Turtle graphics
5.25.1.41 Help and configuration
................................
* Menu:
* How to use help::
* Translation of docstrings into different languages::
* How to configure Screen and Turtles::
File: python.info, Node: How to use help, Next: Translation of docstrings into different languages, Up: Help and configuration
5.25.1.42 How to use help
.........................
The public methods of the Screen and Turtle classes are documented
extensively via docstrings. So these can be used as online-help via the
Python help facilities:
- When using IDLE, tooltips show the signatures and first lines of
the docstrings of typed in function-/method calls.
- Calling *note help(): 8d6. on methods or functions displays the
docstrings:
>>> help(Screen.bgcolor)
Help on method bgcolor in module turtle:
bgcolor(self, *args) unbound turtle.Screen method
Set or return backgroundcolor of the TurtleScreen.
Arguments (if given): a color string or three numbers
in the range 0..colormode or a 3-tuple of such numbers.
>>> screen.bgcolor("orange")
>>> screen.bgcolor()
"orange"
>>> screen.bgcolor(0.5,0,0.5)
>>> screen.bgcolor()
"#800080"
>>> help(Turtle.penup)
Help on method penup in module turtle:
penup(self) unbound turtle.Turtle method
Pull the pen up -- no drawing when moving.
Aliases: penup | pu | up
No argument
>>> turtle.penup()
- The docstrings of the functions which are derived from methods have
a modified form:
>>> help(bgcolor)
Help on function bgcolor in module turtle:
bgcolor(*args)
Set or return backgroundcolor of the TurtleScreen.
Arguments (if given): a color string or three numbers
in the range 0..colormode or a 3-tuple of such numbers.
Example::
>>> bgcolor("orange")
>>> bgcolor()
"orange"
>>> bgcolor(0.5,0,0.5)
>>> bgcolor()
"#800080"
>>> help(penup)
Help on function penup in module turtle:
penup()
Pull the pen up -- no drawing when moving.
Aliases: penup | pu | up
No argument
Example:
>>> penup()
These modified docstrings are created automatically together with the
function definitions that are derived from the methods at import time.
File: python.info, Node: Translation of docstrings into different languages, Next: How to configure Screen and Turtles, Prev: How to use help, Up: Help and configuration
5.25.1.43 Translation of docstrings into different languages
............................................................
There is a utility to create a dictionary the keys of which are the
method names and the values of which are the docstrings of the public
methods of the classes Screen and Turtle.
-- Function: turtle.write_docstringdict
(filename='turtle_docstringdict')
Parameters: ‘filename’ – a string, used as filename
Create and write docstring-dictionary to a Python script with the
given filename. This function has to be called explicitly (it is
not used by the turtle graphics classes). The docstring dictionary
will be written to the Python script ‘`filename'.py’. It is
intended to serve as a template for translation of the docstrings
into different languages.
If you (or your students) want to use *note turtle: 101. with online
help in your native language, you have to translate the docstrings and
save the resulting file as e.g. ‘turtle_docstringdict_german.py’.
If you have an appropriate entry in your ‘turtle.cfg’ file this
dictionary will be read in at import time and will replace the original
English docstrings.
At the time of this writing there are docstring dictionaries in German
and in Italian. (Requests please to .)
File: python.info, Node: How to configure Screen and Turtles, Prev: Translation of docstrings into different languages, Up: Help and configuration
5.25.1.44 How to configure Screen and Turtles
.............................................
The built-in default configuration mimics the appearance and behaviour
of the old turtle module in order to retain best possible compatibility
with it.
If you want to use a different configuration which better reflects the
features of this module or which better fits to your needs, e.g. for
use in a classroom, you can prepare a configuration file ‘turtle.cfg’
which will be read at import time and modify the configuration according
to its settings.
The built in configuration would correspond to the following
‘turtle.cfg’:
width = 0.5
height = 0.75
leftright = None
topbottom = None
canvwidth = 400
canvheight = 300
mode = standard
colormode = 1.0
delay = 10
undobuffersize = 1000
shape = classic
pencolor = black
fillcolor = black
resizemode = noresize
visible = True
language = english
exampleturtle = turtle
examplescreen = screen
title = Python Turtle Graphics
using_IDLE = False
Short explanation of selected entries:
- The first four lines correspond to the arguments of the *note
Screen.setup: 3ddb. method.
- Line 5 and 6 correspond to the arguments of the method *note
Screen.screensize: 3dc3.
- 'shape' can be any of the built-in shapes, e.g: arrow, turtle, etc.
For more info try ‘help(shape)’.
- If you want to use no fill color (i.e. make the turtle
transparent), you have to write ‘fillcolor = ""’ (but all nonempty
strings must not have quotes in the cfg file).
- If you want to reflect the turtle its state, you have to use
‘resizemode = auto’.
- If you set e.g. ‘language = italian’ the docstringdict
‘turtle_docstringdict_italian.py’ will be loaded at import time (if
present on the import path, e.g. in the same directory as *note
turtle: 101.).
- The entries 'exampleturtle' and 'examplescreen' define the names of
these objects as they occur in the docstrings. The transformation
of method-docstrings to function-docstrings will delete these names
from the docstrings.
- 'using_IDLE': Set this to ‘True’ if you regularly work with IDLE
and its ‘-n’ switch (“no subprocess”). This will prevent *note
exitonclick(): 3dda. to enter the mainloop.
There can be a ‘turtle.cfg’ file in the directory where *note turtle:
101. is stored and an additional one in the current working directory.
The latter will override the settings of the first one.
The ‘Lib/turtledemo’ directory contains a ‘turtle.cfg’ file. You can
study it as an example and see its effects when running the demos
(preferably not from within the demo-viewer).
File: python.info, Node: turtledemo — Demo scripts, Next: Changes since Python 2 6, Prev: Help and configuration, Up: turtle — Turtle graphics
5.25.1.45 ‘turtledemo’ — Demo scripts
.....................................
The *note turtledemo: 102. package includes a set of demo scripts.
These scripts can be run and viewed using the supplied demo viewer as
follows:
python -m turtledemo
Alternatively, you can run the demo scripts individually. For example,
python -m turtledemo.bytedesign
The *note turtledemo: 102. package directory contains:
- A demo viewer ‘__main__.py’ which can be used to view the
sourcecode of the scripts and run them at the same time.
- Multiple scripts demonstrating different features of the *note
turtle: 101. module. Examples can be accessed via the Examples
menu. They can also be run standalone.
- A ‘turtle.cfg’ file which serves as an example of how to write and
use such files.
The demo scripts are:
Name Description Features
------------------------------------------------------------------------------------
bytedesign complex classical turtle *note tracer(): 3dc6,
graphics pattern delay,
*note update(): 3dc7.
chaos graphs Verhulst dynamics, shows world coordinates
that computer’s computations can
generate results sometimes
against the common sense
expectations
clock analog clock showing time of turtles as clock’s hands,
your computer ontimer
colormixer experiment with r, g, b *note ondrag(): 3db4.
forest 3 breadth-first trees randomization
fractalcurves Hilbert & Koch curves recursion
lindenmayer ethnomathematics (indian kolams) L-System
minimal_hanoi Towers of Hanoi Rectangular Turtles as
Hanoi discs (shape,
shapesize)
nim play the classical nim game with turtles as nimsticks,
three heaps of sticks against event driven (mouse,
the computer. keyboard)
paint super minimalistic drawing *note onclick(): 3db2.
program
peace elementary turtle: appearance and
animation
penrose aperiodic tiling with kites and *note stamp(): 3d84.
darts
planet_and_moon simulation of gravitational compound shapes,
system *note Vec2D: 3ddf.
rosette a pattern from the wikipedia *note clone(): 3db8,
article on turtle graphics *note undo(): 3d87.
round_dance dancing turtles rotating compound shapes, clone
pairwise in opposite direction shapesize, tilt,
get_shapepoly, update
sorting_animate visual demonstration of simple alignment,
different sorting methods randomization
tree a (graphical) breadth first tree *note clone(): 3db8.
(using generators)
two_canvases simple design turtles on two canvases
yinyang another elementary example *note circle(): 3d82.
Have fun!
File: python.info, Node: Changes since Python 2 6, Next: Changes since Python 3 0, Prev: turtledemo — Demo scripts, Up: turtle — Turtle graphics
5.25.1.46 Changes since Python 2.6
..................................
- The methods *note Turtle.tracer: 3dc6, *note Turtle.window_width:
3dd8. and *note Turtle.window_height: 3dd7. have been eliminated.
Methods with these names and functionality are now available only
as methods of *note Screen: 3d6f. The functions derived from these
remain available. (In fact already in Python 2.6 these methods
were merely duplications of the corresponding *note TurtleScreen:
3dec./*note Screen: 3d6f. methods.)
- The method ‘Turtle.fill()’ has been eliminated. The behaviour of
*note begin_fill(): 3da0. and *note end_fill(): 3da1. have changed
slightly: now every filling process must be completed with an
‘end_fill()’ call.
- A method *note Turtle.filling: 3d9f. has been added. It returns a
boolean value: ‘True’ if a filling process is under way, ‘False’
otherwise. This behaviour corresponds to a ‘fill()’ call without
arguments in Python 2.6.
File: python.info, Node: Changes since Python 3 0, Prev: Changes since Python 2 6, Up: turtle — Turtle graphics
5.25.1.47 Changes since Python 3.0
..................................
- The *note Turtle: 3dfd. methods *note shearfactor(): 3dae, *note
shapetransform(): 3db0. and *note get_shapepoly(): 3db1. have been
added. Thus the full range of regular linear transforms is now
available for transforming turtle shapes. *note tiltangle(): 720.
has been enhanced in functionality: it now can be used to get or
set the tilt angle.
- The *note Screen: 3d6f. method *note onkeypress(): 3dcb. has been
added as a complement to *note onkey(): 3dc9. As the latter binds
actions to the key release event, an alias: *note onkeyrelease():
3dca. was also added for it.
- The method *note Screen.mainloop: 3dce. has been added, so there is
no longer a need to use the standalone *note mainloop(): 3dce.
function when working with *note Screen: 3d6f. and *note Turtle:
3dfd. objects.
- Two input methods have been added: *note Screen.textinput: 18c2.
and *note Screen.numinput: 18c3. These pop up input dialogs and
return strings and numbers respectively.
- Two example scripts ‘tdemo_nim.py’ and ‘tdemo_round_dance.py’ have
been added to the ‘Lib/turtledemo’ directory.
File: python.info, Node: cmd — Support for line-oriented command interpreters, Next: shlex — Simple lexical analysis, Prev: turtle — Turtle graphics, Up: Program Frameworks
5.25.2 ‘cmd’ — Support for line-oriented command interpreters
-------------------------------------------------------------
'Source code:' Lib/cmd.py(1)
__________________________________________________________________
The *note Cmd: 16c5. class provides a simple framework for writing
line-oriented command interpreters. These are often useful for test
harnesses, administrative tools, and prototypes that will later be
wrapped in a more sophisticated interface.
-- Class: cmd.Cmd (completekey='tab', stdin=None, stdout=None)
A *note Cmd: 16c5. instance or subclass instance is a line-oriented
interpreter framework. There is no good reason to instantiate
*note Cmd: 16c5. itself; rather, it’s useful as a superclass of an
interpreter class you define yourself in order to inherit *note
Cmd: 16c5.’s methods and encapsulate action methods.
The optional argument 'completekey' is the *note readline: ba. name
of a completion key; it defaults to ‘Tab’. If 'completekey' is not
*note None: 671. and *note readline: ba. is available, command
completion is done automatically.
The default, ‘'tab'’, is treated specially, so that it refers to
the ‘Tab’ key on every *note readline.backend: 16af. Specifically,
if *note readline.backend: 16af. is ‘editline’, ‘Cmd’ will use
‘'^I'’ instead of ‘'tab'’. Note that other values are not treated
this way, and might only work with a specific backend.
The optional arguments 'stdin' and 'stdout' specify the input and
output file objects that the Cmd instance or subclass instance will
use for input and output. If not specified, they will default to
*note sys.stdin: 539. and *note sys.stdout: ad6.
If you want a given 'stdin' to be used, make sure to set the
instance’s *note use_rawinput: 3e09. attribute to ‘False’,
otherwise 'stdin' will be ignored.
Changed in version 3.13: ‘completekey='tab'’ is replaced by ‘'^I'’
for ‘editline’.
* Menu:
* Cmd Objects::
* Cmd Example::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/cmd.py
File: python.info, Node: Cmd Objects, Next: Cmd Example, Up: cmd — Support for line-oriented command interpreters
5.25.2.1 Cmd Objects
....................
A *note Cmd: 16c5. instance has the following methods:
-- Method: Cmd.cmdloop (intro=None)
Repeatedly issue a prompt, accept input, parse an initial prefix
off the received input, and dispatch to action methods, passing
them the remainder of the line as argument.
The optional argument is a banner or intro string to be issued
before the first prompt (this overrides the *note intro: 3e0d.
class attribute).
If the *note readline: ba. module is loaded, input will
automatically inherit ‘bash’-like history-list editing (e.g.
‘Control’-‘P’ scrolls back to the last command, ‘Control’-‘N’
forward to the next one, ‘Control’-‘F’ moves the cursor to the
right non-destructively, ‘Control’-‘B’ moves the cursor to the left
non-destructively, etc.).
An end-of-file on input is passed back as the string ‘'EOF'’.
An interpreter instance will recognize a command name ‘foo’ if and
only if it has a method ‘do_foo()’. As a special case, a line
beginning with the character ‘'?'’ is dispatched to the method
*note do_help(): 16b4. As another special case, a line beginning
with the character ‘'!'’ is dispatched to the method ‘do_shell()’
(if such a method is defined).
This method will return when the *note postcmd(): 3e0e. method
returns a true value. The 'stop' argument to *note postcmd():
3e0e. is the return value from the command’s corresponding ‘do_*()’
method.
If completion is enabled, completing commands will be done
automatically, and completing of commands args is done by calling
‘complete_foo()’ with arguments 'text', 'line', 'begidx', and
'endidx'. 'text' is the string prefix we are attempting to match:
all returned matches must begin with it. 'line' is the current
input line with leading whitespace removed, 'begidx' and 'endidx'
are the beginning and ending indexes of the prefix text, which
could be used to provide different completion depending upon which
position the argument is in.
-- Method: Cmd.do_help (arg)
All subclasses of *note Cmd: 16c5. inherit a predefined
‘do_help()’. This method, called with an argument ‘'bar'’, invokes
the corresponding method ‘help_bar()’, and if that is not present,
prints the docstring of ‘do_bar()’, if available. With no
argument, ‘do_help()’ lists all available help topics (that is, all
commands with corresponding ‘help_*()’ methods or commands that
have docstrings), and also lists any undocumented commands.
-- Method: Cmd.onecmd (str)
Interpret the argument as though it had been typed in response to
the prompt. This may be overridden, but should not normally need
to be; see the *note precmd(): 3e10. and *note postcmd(): 3e0e.
methods for useful execution hooks. The return value is a flag
indicating whether interpretation of commands by the interpreter
should stop. If there is a ‘do_*()’ method for the command 'str',
the return value of that method is returned, otherwise the return
value from the *note default(): 3e11. method is returned.
-- Method: Cmd.emptyline ()
Method called when an empty line is entered in response to the
prompt. If this method is not overridden, it repeats the last
nonempty command entered.
-- Method: Cmd.default (line)
Method called on an input line when the command prefix is not
recognized. If this method is not overridden, it prints an error
message and returns.
-- Method: Cmd.completedefault (text, line, begidx, endidx)
Method called to complete an input line when no command-specific
‘complete_*()’ method is available. By default, it returns an
empty list.
-- Method: Cmd.columnize (list, displaywidth=80)
Method called to display a list of strings as a compact set of
columns. Each column is only as wide as necessary. Columns are
separated by two spaces for readability.
-- Method: Cmd.precmd (line)
Hook method executed just before the command line 'line' is
interpreted, but after the input prompt is generated and issued.
This method is a stub in *note Cmd: 16c5.; it exists to be
overridden by subclasses. The return value is used as the command
which will be executed by the *note onecmd(): 3e0f. method; the
*note precmd(): 3e10. implementation may re-write the command or
simply return 'line' unchanged.
-- Method: Cmd.postcmd (stop, line)
Hook method executed just after a command dispatch is finished.
This method is a stub in *note Cmd: 16c5.; it exists to be
overridden by subclasses. 'line' is the command line which was
executed, and 'stop' is a flag which indicates whether execution
will be terminated after the call to *note postcmd(): 3e0e.; this
will be the return value of the *note onecmd(): 3e0f. method. The
return value of this method will be used as the new value for the
internal flag which corresponds to 'stop'; returning false will
cause interpretation to continue.
-- Method: Cmd.preloop ()
Hook method executed once when *note cmdloop(): 3e0c. is called.
This method is a stub in *note Cmd: 16c5.; it exists to be
overridden by subclasses.
-- Method: Cmd.postloop ()
Hook method executed once when *note cmdloop(): 3e0c. is about to
return. This method is a stub in *note Cmd: 16c5.; it exists to be
overridden by subclasses.
Instances of *note Cmd: 16c5. subclasses have some public instance
variables:
-- Attribute: Cmd.prompt
The prompt issued to solicit input.
-- Attribute: Cmd.identchars
The string of characters accepted for the command prefix.
-- Attribute: Cmd.lastcmd
The last nonempty command prefix seen.
-- Attribute: Cmd.cmdqueue
A list of queued input lines. The cmdqueue list is checked in
*note cmdloop(): 3e0c. when new input is needed; if it is nonempty,
its elements will be processed in order, as if entered at the
prompt.
-- Attribute: Cmd.intro
A string to issue as an intro or banner. May be overridden by
giving the *note cmdloop(): 3e0c. method an argument.
-- Attribute: Cmd.doc_header
The header to issue if the help output has a section for documented
commands.
-- Attribute: Cmd.misc_header
The header to issue if the help output has a section for
miscellaneous help topics (that is, there are ‘help_*()’ methods
without corresponding ‘do_*()’ methods).
-- Attribute: Cmd.undoc_header
The header to issue if the help output has a section for
undocumented commands (that is, there are ‘do_*()’ methods without
corresponding ‘help_*()’ methods).
-- Attribute: Cmd.ruler
The character used to draw separator lines under the help-message
headers. If empty, no ruler line is drawn. It defaults to ‘'='’.
-- Attribute: Cmd.use_rawinput
A flag, defaulting to true. If true, *note cmdloop(): 3e0c. uses
*note input(): 12cb. to display a prompt and read the next command;
if false, *note sys.stdout.write(): ad6. and *note
sys.stdin.readline(): 539. are used. (This means that by importing
*note readline: ba, on systems that support it, the interpreter
will automatically support ‘Emacs’-like line editing and
command-history keystrokes.)
File: python.info, Node: Cmd Example, Prev: Cmd Objects, Up: cmd — Support for line-oriented command interpreters
5.25.2.2 Cmd Example
....................
The *note cmd: 19. module is mainly useful for building custom shells
that let a user work with a program interactively.
This section presents a simple example of how to build a shell around a
few of the commands in the *note turtle: 101. module.
Basic turtle commands such as *note forward(): 3d69. are added to a
*note Cmd: 16c5. subclass with method named ‘do_forward()’. The
argument is converted to a number and dispatched to the turtle module.
The docstring is used in the help utility provided by the shell.
The example also includes a basic record and playback facility
implemented with the *note precmd(): 3e10. method which is responsible
for converting the input to lowercase and writing the commands to a
file. The ‘do_playback()’ method reads the file and adds the recorded
commands to the *note cmdqueue: 3e19. for immediate playback:
import cmd, sys
from turtle import *
class TurtleShell(cmd.Cmd):
intro = 'Welcome to the turtle shell. Type help or ? to list commands.\n'
prompt = '(turtle) '
file = None
# ----- basic turtle commands -----
def do_forward(self, arg):
'Move the turtle forward by the specified distance: FORWARD 10'
forward(*parse(arg))
def do_right(self, arg):
'Turn turtle right by given number of degrees: RIGHT 20'
right(*parse(arg))
def do_left(self, arg):
'Turn turtle left by given number of degrees: LEFT 90'
left(*parse(arg))
def do_goto(self, arg):
'Move turtle to an absolute position with changing orientation. GOTO 100 200'
goto(*parse(arg))
def do_home(self, arg):
'Return turtle to the home position: HOME'
home()
def do_circle(self, arg):
'Draw circle with given radius an options extent and steps: CIRCLE 50'
circle(*parse(arg))
def do_position(self, arg):
'Print the current turtle position: POSITION'
print('Current position is %d %d\n' % position())
def do_heading(self, arg):
'Print the current turtle heading in degrees: HEADING'
print('Current heading is %d\n' % (heading(),))
def do_color(self, arg):
'Set the color: COLOR BLUE'
color(arg.lower())
def do_undo(self, arg):
'Undo (repeatedly) the last turtle action(s): UNDO'
def do_reset(self, arg):
'Clear the screen and return turtle to center: RESET'
reset()
def do_bye(self, arg):
'Stop recording, close the turtle window, and exit: BYE'
print('Thank you for using Turtle')
self.close()
bye()
return True
# ----- record and playback -----
def do_record(self, arg):
'Save future commands to filename: RECORD rose.cmd'
self.file = open(arg, 'w')
def do_playback(self, arg):
'Playback commands from a file: PLAYBACK rose.cmd'
self.close()
with open(arg) as f:
self.cmdqueue.extend(f.read().splitlines())
def precmd(self, line):
line = line.lower()
if self.file and 'playback' not in line:
print(line, file=self.file)
return line
def close(self):
if self.file:
self.file.close()
self.file = None
def parse(arg):
'Convert a series of zero or more numbers to an argument tuple'
return tuple(map(int, arg.split()))
if __name__ == '__main__':
TurtleShell().cmdloop()
Here is a sample session with the turtle shell showing the help
functions, using blank lines to repeat commands, and the simple record
and playback facility:
Welcome to the turtle shell. Type help or ? to list commands.
(turtle) ?
Documented commands (type help ):
========================================
bye color goto home playback record right
circle forward heading left position reset undo
(turtle) help forward
Move the turtle forward by the specified distance: FORWARD 10
(turtle) record spiral.cmd
(turtle) position
Current position is 0 0
(turtle) heading
Current heading is 0
(turtle) reset
(turtle) circle 20
(turtle) right 30
(turtle) circle 40
(turtle) right 30
(turtle) circle 60
(turtle) right 30
(turtle) circle 80
(turtle) right 30
(turtle) circle 100
(turtle) right 30
(turtle) circle 120
(turtle) right 30
(turtle) circle 120
(turtle) heading
Current heading is 180
(turtle) forward 100
(turtle)
(turtle) right 90
(turtle) forward 100
(turtle)
(turtle) right 90
(turtle) forward 400
(turtle) right 90
(turtle) forward 500
(turtle) right 90
(turtle) forward 400
(turtle) right 90
(turtle) forward 300
(turtle) playback spiral.cmd
Current position is 0 0
Current heading is 0
Current heading is 180
(turtle) bye
Thank you for using Turtle
File: python.info, Node: shlex — Simple lexical analysis, Prev: cmd — Support for line-oriented command interpreters, Up: Program Frameworks
5.25.3 ‘shlex’ — Simple lexical analysis
----------------------------------------
'Source code:' Lib/shlex.py(1)
__________________________________________________________________
The *note shlex: ccd. class makes it easy to write lexical analyzers for
simple syntaxes resembling that of the Unix shell. This will often be
useful for writing minilanguages, (for example, in run control files for
Python applications) or for parsing quoted strings.
The *note shlex: c4. module defines the following functions:
-- Function: shlex.split (s, comments=False, posix=True)
Split the string 's' using shell-like syntax. If 'comments' is
*note False: b37. (the default), the parsing of comments in the
given string will be disabled (setting the *note commenters: 3e22.
attribute of the *note shlex: ccd. instance to the empty string).
This function operates in POSIX mode by default, but uses non-POSIX
mode if the 'posix' argument is false.
Changed in version 3.12: Passing ‘None’ for 's' argument now raises
an exception, rather than reading *note sys.stdin: 539.
-- Function: shlex.join (split_command)
Concatenate the tokens of the list 'split_command' and return a
string. This function is the inverse of *note split(): 538.
>>> from shlex import join
>>> print(join(['echo', '-n', 'Multiple words']))
echo -n 'Multiple words'
The returned value is shell-escaped to protect against injection
vulnerabilities (see *note quote(): 27f.).
Added in version 3.8.
-- Function: shlex.quote (s)
Return a shell-escaped version of the string 's'. The returned
value is a string that can safely be used as one token in a shell
command line, for cases where you cannot use a list.
Warning: The ‘shlex’ module is 'only designed for Unix
shells'.
The *note quote(): 27f. function is not guaranteed to be
correct on non-POSIX compliant shells or shells from other
operating systems such as Windows. Executing commands quoted
by this module on such shells can open up the possibility of a
command injection vulnerability.
Consider using functions that pass command arguments with
lists such as *note subprocess.run(): b86. with ‘shell=False’.
This idiom would be unsafe:
>>> filename = 'somefile; rm -rf ~'
>>> command = 'ls -l {}'.format(filename)
>>> print(command) # executed by a shell: boom!
ls -l somefile; rm -rf ~
*note quote(): 27f. lets you plug the security hole:
>>> from shlex import quote
>>> command = 'ls -l {}'.format(quote(filename))
>>> print(command)
ls -l 'somefile; rm -rf ~'
>>> remote_command = 'ssh home {}'.format(quote(command))
>>> print(remote_command)
ssh home 'ls -l '"'"'somefile; rm -rf ~'"'"''
The quoting is compatible with UNIX shells and with *note split():
538.:
>>> from shlex import split
>>> remote_command = split(remote_command)
>>> remote_command
['ssh', 'home', "ls -l 'somefile; rm -rf ~'"]
>>> command = split(remote_command[-1])
>>> command
['ls', '-l', 'somefile; rm -rf ~']
Added in version 3.3.
The *note shlex: c4. module defines the following class:
-- Class: shlex.shlex (instream=None, infile=None, posix=False,
punctuation_chars=False)
A *note shlex: ccd. instance or subclass instance is a lexical
analyzer object. The initialization argument, if present,
specifies where to read characters from. It must be a
file-/stream-like object with *note read(): 1c85. and *note
readline(): 1cbd. methods, or a string. If no argument is given,
input will be taken from ‘sys.stdin’. The second optional argument
is a filename string, which sets the initial value of the *note
infile: 3e23. attribute. If the 'instream' argument is omitted or
equal to ‘sys.stdin’, this second argument defaults to “stdin”.
The 'posix' argument defines the operational mode: when 'posix' is
not true (default), the *note shlex: ccd. instance will operate in
compatibility mode. When operating in POSIX mode, *note shlex:
ccd. will try to be as close as possible to the POSIX shell parsing
rules. The 'punctuation_chars' argument provides a way to make the
behaviour even closer to how real shells parse. This can take a
number of values: the default value, ‘False’, preserves the
behaviour seen under Python 3.5 and earlier. If set to ‘True’,
then parsing of the characters ‘();<>|&’ is changed: any run of
these characters (considered punctuation characters) is returned as
a single token. If set to a non-empty string of characters, those
characters will be used as the punctuation characters. Any
characters in the *note wordchars: 3e24. attribute that appear in
'punctuation_chars' will be removed from *note wordchars: 3e24.
See *note Improved Compatibility with Shells: cce. for more
information. 'punctuation_chars' can be set only upon *note shlex:
ccd. instance creation and can’t be modified later.
Changed in version 3.6: The 'punctuation_chars' parameter was
added.
See also
........
Module *note configparser: 22.
Parser for configuration files similar to the Windows ‘.ini’ files.
* Menu:
* shlex Objects::
* Parsing Rules::
* Improved Compatibility with Shells::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/shlex.py
File: python.info, Node: shlex Objects, Next: Parsing Rules, Up: shlex — Simple lexical analysis
5.25.3.1 shlex Objects
......................
A *note shlex: ccd. instance has the following methods:
-- Method: shlex.get_token ()
Return a token. If tokens have been stacked using *note
push_token(): 3e28, pop a token off the stack. Otherwise, read one
from the input stream. If reading encounters an immediate
end-of-file, *note eof: 3e29. is returned (the empty string (‘''’)
in non-POSIX mode, and ‘None’ in POSIX mode).
-- Method: shlex.push_token (str)
Push the argument onto the token stack.
-- Method: shlex.read_token ()
Read a raw token. Ignore the pushback stack, and do not interpret
source requests. (This is not ordinarily a useful entry point, and
is documented here only for the sake of completeness.)
-- Method: shlex.sourcehook (filename)
When *note shlex: ccd. detects a source request (see *note source:
3e2c. below) this method is given the following token as argument,
and expected to return a tuple consisting of a filename and an open
file-like object.
Normally, this method first strips any quotes off the argument. If
the result is an absolute pathname, or there was no previous source
request in effect, or the previous source was a stream (such as
‘sys.stdin’), the result is left alone. Otherwise, if the result
is a relative pathname, the directory part of the name of the file
immediately before it on the source inclusion stack is prepended
(this behavior is like the way the C preprocessor handles ‘#include
"file.h"’).
The result of the manipulations is treated as a filename, and
returned as the first component of the tuple, with *note open():
517. called on it to yield the second component. (Note: this is
the reverse of the order of arguments in instance initialization!)
This hook is exposed so that you can use it to implement directory
search paths, addition of file extensions, and other namespace
hacks. There is no corresponding ‘close’ hook, but a shlex
instance will call the *note close(): 1f8. method of the sourced
input stream when it returns EOF.
For more explicit control of source stacking, use the *note
push_source(): 3e2d. and *note pop_source(): 3e2e. methods.
-- Method: shlex.push_source (newstream, newfile=None)
Push an input source stream onto the input stack. If the filename
argument is specified it will later be available for use in error
messages. This is the same method used internally by the *note
sourcehook(): 3e2b. method.
-- Method: shlex.pop_source ()
Pop the last-pushed input source from the input stack. This is the
same method used internally when the lexer reaches EOF on a stacked
input stream.
-- Method: shlex.error_leader (infile=None, lineno=None)
This method generates an error message leader in the format of a
Unix C compiler error label; the format is ‘'"%s", line %d: '’,
where the ‘%s’ is replaced with the name of the current source file
and the ‘%d’ with the current input line number (the optional
arguments can be used to override these).
This convenience is provided to encourage *note shlex: c4. users to
generate error messages in the standard, parseable format
understood by Emacs and other Unix tools.
Instances of *note shlex: ccd. subclasses have some public instance
variables which either control lexical analysis or can be used for
debugging:
-- Attribute: shlex.commenters
The string of characters that are recognized as comment beginners.
All characters from the comment beginner to end of line are
ignored. Includes just ‘'#'’ by default.
-- Attribute: shlex.wordchars
The string of characters that will accumulate into multi-character
tokens. By default, includes all ASCII alphanumerics and
underscore. In POSIX mode, the accented characters in the Latin-1
set are also included. If *note punctuation_chars: 19d0. is not
empty, the characters ‘~-./*?=’, which can appear in filename
specifications and command line parameters, will also be included
in this attribute, and any characters which appear in
‘punctuation_chars’ will be removed from ‘wordchars’ if they are
present there. If *note whitespace_split: 3e30. is set to ‘True’,
this will have no effect.
-- Attribute: shlex.whitespace
Characters that will be considered whitespace and skipped.
Whitespace bounds tokens. By default, includes space, tab,
linefeed and carriage-return.
-- Attribute: shlex.escape
Characters that will be considered as escape. This will be only
used in POSIX mode, and includes just ‘'\'’ by default.
-- Attribute: shlex.quotes
Characters that will be considered string quotes. The token
accumulates until the same quote is encountered again (thus,
different quote types protect each other as in the shell.) By
default, includes ASCII single and double quotes.
-- Attribute: shlex.escapedquotes
Characters in *note quotes: 3e33. that will interpret escape
characters defined in *note escape: 3e32. This is only used in
POSIX mode, and includes just ‘'"'’ by default.
-- Attribute: shlex.whitespace_split
If ‘True’, tokens will only be split in whitespaces. This is
useful, for example, for parsing command lines with *note shlex:
ccd, getting tokens in a similar way to shell arguments. When used
in combination with *note punctuation_chars: 19d0, tokens will be
split on whitespace in addition to those characters.
Changed in version 3.8: The *note punctuation_chars: 19d0.
attribute was made compatible with the *note whitespace_split:
3e30. attribute.
-- Attribute: shlex.infile
The name of the current input file, as initially set at class
instantiation time or stacked by later source requests. It may be
useful to examine this when constructing error messages.
-- Attribute: shlex.instream
The input stream from which this *note shlex: ccd. instance is
reading characters.
-- Attribute: shlex.source
This attribute is ‘None’ by default. If you assign a string to it,
that string will be recognized as a lexical-level inclusion request
similar to the ‘source’ keyword in various shells. That is, the
immediately following token will be opened as a filename and input
will be taken from that stream until EOF, at which point the *note
close(): 1f8. method of that stream will be called and the input
source will again become the original input stream. Source
requests may be stacked any number of levels deep.
-- Attribute: shlex.debug
If this attribute is numeric and ‘1’ or more, a *note shlex: ccd.
instance will print verbose progress output on its behavior. If
you need to use this, you can read the module source code to learn
the details.
-- Attribute: shlex.lineno
Source line number (count of newlines seen so far plus one).
-- Attribute: shlex.token
The token buffer. It may be useful to examine this when catching
exceptions.
-- Attribute: shlex.eof
Token used to determine end of file. This will be set to the empty
string (‘''’), in non-POSIX mode, and to ‘None’ in POSIX mode.
-- Attribute: shlex.punctuation_chars
A read-only property. Characters that will be considered
punctuation. Runs of punctuation characters will be returned as a
single token. However, note that no semantic validity checking
will be performed: for example, ‘>>>’ could be returned as a token,
even though it may not be recognised as such by shells.
Added in version 3.6.
File: python.info, Node: Parsing Rules, Next: Improved Compatibility with Shells, Prev: shlex Objects, Up: shlex — Simple lexical analysis
5.25.3.2 Parsing Rules
......................
When operating in non-POSIX mode, *note shlex: ccd. will try to obey to
the following rules.
* Quote characters are not recognized within words (‘Do"Not"Separate’
is parsed as the single word ‘Do"Not"Separate’);
* Escape characters are not recognized;
* Enclosing characters in quotes preserve the literal value of all
characters within the quotes;
* Closing quotes separate words (‘"Do"Separate’ is parsed as ‘"Do"’
and ‘Separate’);
* If *note whitespace_split: 3e30. is ‘False’, any character not
declared to be a word character, whitespace, or a quote will be
returned as a single-character token. If it is ‘True’, *note
shlex: ccd. will only split words in whitespaces;
* EOF is signaled with an empty string (‘''’);
* It’s not possible to parse empty strings, even if quoted.
When operating in POSIX mode, *note shlex: ccd. will try to obey to the
following parsing rules.
* Quotes are stripped out, and do not separate words
(‘"Do"Not"Separate"’ is parsed as the single word ‘DoNotSeparate’);
* Non-quoted escape characters (e.g. ‘'\'’) preserve the literal
value of the next character that follows;
* Enclosing characters in quotes which are not part of *note
escapedquotes: 3e34. (e.g. ‘"'"’) preserve the literal value of
all characters within the quotes;
* Enclosing characters in quotes which are part of *note
escapedquotes: 3e34. (e.g. ‘'"'’) preserves the literal value of
all characters within the quotes, with the exception of the
characters mentioned in *note escape: 3e32. The escape characters
retain its special meaning only when followed by the quote in use,
or the escape character itself. Otherwise the escape character
will be considered a normal character.
* EOF is signaled with a *note None: 671. value;
* Quoted empty strings (‘''’) are allowed.
File: python.info, Node: Improved Compatibility with Shells, Prev: Parsing Rules, Up: shlex — Simple lexical analysis
5.25.3.3 Improved Compatibility with Shells
...........................................
Added in version 3.6.
The *note shlex: c4. class provides compatibility with the parsing
performed by common Unix shells like ‘bash’, ‘dash’, and ‘sh’. To take
advantage of this compatibility, specify the ‘punctuation_chars’
argument in the constructor. This defaults to ‘False’, which preserves
pre-3.6 behaviour. However, if it is set to ‘True’, then parsing of the
characters ‘();<>|&’ is changed: any run of these characters is returned
as a single token. While this is short of a full parser for shells
(which would be out of scope for the standard library, given the
multiplicity of shells out there), it does allow you to perform
processing of command lines more easily than you could otherwise. To
illustrate, you can see the difference in the following snippet:
>>> import shlex
>>> text = "a && b; c && d || e; f >'abc'; (def \"ghi\")"
>>> s = shlex.shlex(text, posix=True)
>>> s.whitespace_split = True
>>> list(s)
['a', '&&', 'b;', 'c', '&&', 'd', '||', 'e;', 'f', '>abc;', '(def', 'ghi)']
>>> s = shlex.shlex(text, posix=True, punctuation_chars=True)
>>> s.whitespace_split = True
>>> list(s)
['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', 'abc', ';',
'(', 'def', 'ghi', ')']
Of course, tokens will be returned which are not valid for shells, and
you’ll need to implement your own error checks on the returned tokens.
Instead of passing ‘True’ as the value for the punctuation_chars
parameter, you can pass a string with specific characters, which will be
used to determine which characters constitute punctuation. For example:
>>> import shlex
>>> s = shlex.shlex("a && b || c", punctuation_chars="|")
>>> list(s)
['a', '&', '&', 'b', '||', 'c']
Note: When ‘punctuation_chars’ is specified, the *note wordchars:
3e24. attribute is augmented with the characters ‘~-./*?=’. That
is because these characters can appear in file names (including
wildcards) and command-line arguments (e.g. ‘--color=auto’).
Hence:
>>> import shlex
>>> s = shlex.shlex('~/a && b-c --color=auto || d *.py?',
... punctuation_chars=True)
>>> list(s)
['~/a', '&&', 'b-c', '--color=auto', '||', 'd', '*.py?']
However, to match the shell as closely as possible, it is
recommended to always use ‘posix’ and *note whitespace_split: 3e30.
when using *note punctuation_chars: 19d0, which will negate *note
wordchars: 3e24. entirely.
For best effect, ‘punctuation_chars’ should be set in conjunction with
‘posix=True’. (Note that ‘posix=False’ is the default for *note shlex:
ccd.)
File: python.info, Node: Graphical User Interfaces with Tk, Next: Development Tools, Prev: Program Frameworks, Up: The Python Standard Library
5.26 Graphical User Interfaces with Tk
======================================
Tk/Tcl has long been an integral part of Python. It provides a robust
and platform independent windowing toolkit, that is available to Python
programmers using the *note tkinter: f0. package, and its extension, the
*note tkinter.ttk: f9. module.
The *note tkinter: f0. package is a thin object-oriented layer on top of
Tcl/Tk. To use *note tkinter: f0, you don’t need to write Tcl code, but
you will need to consult the Tk documentation, and occasionally the Tcl
documentation. *note tkinter: f0. is a set of wrappers that implement
the Tk widgets as Python classes.
*note tkinter: f0.’s chief virtues are that it is fast, and that it
usually comes bundled with Python. Although its standard documentation
is weak, good material is available, which includes: references,
tutorials, a book and others. *note tkinter: f0. is also famous for
having an outdated look and feel, which has been vastly improved in Tk
8.5. Nevertheless, there are many other GUI libraries that you could be
interested in. The Python wiki lists several alternative GUI frameworks
and tools(1).
* Menu:
* tkinter — Python interface to Tcl/Tk::
* tkinter.colorchooser — Color choosing dialog: tkinter colorchooser — Color choosing dialog.
* tkinter.font — Tkinter font wrapper: tkinter font — Tkinter font wrapper.
* Tkinter Dialogs::
* tkinter.messagebox — Tkinter message prompts: tkinter messagebox — Tkinter message prompts.
* tkinter.scrolledtext — Scrolled Text Widget: tkinter scrolledtext — Scrolled Text Widget.
* tkinter.dnd — Drag and drop support: tkinter dnd — Drag and drop support.
* tkinter.ttk — Tk themed widgets: tkinter ttk — Tk themed widgets.
* IDLE — Python editor and shell: IDLE — Python editor and shell<2>.
---------- Footnotes ----------
(1) https://wiki.python.org/moin/GuiProgramming
File: python.info, Node: tkinter — Python interface to Tcl/Tk, Next: tkinter colorchooser — Color choosing dialog, Up: Graphical User Interfaces with Tk
5.26.1 ‘tkinter’ — Python interface to Tcl/Tk
---------------------------------------------
'Source code:' Lib/tkinter/__init__.py(1)
__________________________________________________________________
The *note tkinter: f0. package (“Tk interface”) is the standard Python
interface to the Tcl/Tk GUI toolkit. Both Tk and *note tkinter: f0. are
available on most Unix platforms, including macOS, as well as on Windows
systems.
Running ‘python -m tkinter’ from the command line should open a window
demonstrating a simple Tk interface, letting you know that *note
tkinter: f0. is properly installed on your system, and also showing what
version of Tcl/Tk is installed, so you can read the Tcl/Tk documentation
specific to that version.
Tkinter supports a range of Tcl/Tk versions, built either with or
without thread support. The official Python binary release bundles
Tcl/Tk 8.6 threaded. See the source code for the *note _tkinter: 3.
module for more information about supported versions.
Tkinter is not a thin wrapper, but adds a fair amount of its own logic
to make the experience more pythonic. This documentation will
concentrate on these additions and changes, and refer to the official
Tcl/Tk documentation for details that are unchanged.
Note: Tcl/Tk 8.5 (2007) introduced a modern set of themed user
interface components along with a new API to use them. Both old
and new APIs are still available. Most documentation you will find
online still uses the old API and can be woefully outdated.
See also
........
*
TkDocs(2)
Extensive tutorial on creating user interfaces with Tkinter.
Explains key concepts, and illustrates recommended approaches
using the modern API.
*
Tkinter 8.5 reference: a GUI for Python(3)
Reference documentation for Tkinter 8.5 detailing available
classes, methods, and options.
Tcl/Tk Resources:
*
Tk commands(4)
Comprehensive reference to each of the underlying Tcl/Tk
commands used by Tkinter.
*
Tcl/Tk Home Page(5)
Additional documentation, and links to Tcl/Tk core
development.
Books:
*
Modern Tkinter for Busy Python Developers(6)
By Mark Roseman. (ISBN 978-1999149567)
*
Python GUI programming with Tkinter(7)
By Alan D. Moore. (ISBN 978-1788835886)
*
Programming Python(8)
By Mark Lutz; has excellent coverage of Tkinter. (ISBN
978-0596158101)
*
Tcl and the Tk Toolkit (2nd edition)(9)
By John Ousterhout, inventor of Tcl/Tk, and Ken Jones; does
not cover Tkinter. (ISBN 978-0321336330)
* Menu:
* Architecture::
* Tkinter Modules::
* Tkinter Life Preserver::
* Threading model::
* Handy Reference::
* File Handlers::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/tkinter/__init__.py
(2) https://tkdocs.com/
(3) https://www.tkdocs.com/shipman/
(4) https://www.tcl.tk/man/tcl8.6/TkCmd/contents.htm
(5) https://www.tcl.tk
(6) https://tkdocs.com/book.html
(7)
https://www.packtpub.com/en-us/product/python-gui-programming-with-tkinter-9781788835886
(8) https://learning-python.com/about-pp4e.html
(9) https://www.amazon.com/exec/obidos/ASIN/032133633X
File: python.info, Node: Architecture, Next: Tkinter Modules, Up: tkinter — Python interface to Tcl/Tk
5.26.1.1 Architecture
.....................
Tcl/Tk is not a single library but rather consists of a few distinct
modules, each with separate functionality and its own official
documentation. Python’s binary releases also ship an add-on module
together with it.
Tcl
Tcl is a dynamic interpreted programming language, just like
Python. Though it can be used on its own as a general-purpose
programming language, it is most commonly embedded into C
applications as a scripting engine or an interface to the Tk
toolkit. The Tcl library has a C interface to create and manage
one or more instances of a Tcl interpreter, run Tcl commands and
scripts in those instances, and add custom commands implemented in
either Tcl or C. Each interpreter has an event queue, and there are
facilities to send events to it and process them. Unlike Python,
Tcl’s execution model is designed around cooperative multitasking,
and Tkinter bridges this difference (see *note Threading model:
3e42. for details).
Tk
Tk is a Tcl package(1) implemented in C that adds custom commands
to create and manipulate GUI widgets. Each *note Tk: 1636. object
embeds its own Tcl interpreter instance with Tk loaded into it.
Tk’s widgets are very customizable, though at the cost of a dated
appearance. Tk uses Tcl’s event queue to generate and process GUI
events.
Ttk
Themed Tk (Ttk) is a newer family of Tk widgets that provide a much
better appearance on different platforms than many of the classic
Tk widgets. Ttk is distributed as part of Tk, starting with Tk
version 8.5. Python bindings are provided in a separate module,
*note tkinter.ttk: f9.
Internally, Tk and Ttk use facilities of the underlying operating
system, i.e., Xlib on Unix/X11, Cocoa on macOS, GDI on Windows.
When your Python application uses a class in Tkinter, e.g., to create a
widget, the *note tkinter: f0. module first assembles a Tcl/Tk command
string. It passes that Tcl command string to an internal *note
_tkinter: 3. binary module, which then calls the Tcl interpreter to
evaluate it. The Tcl interpreter will then call into the Tk and/or Ttk
packages, which will in turn make calls to Xlib, Cocoa, or GDI.
---------- Footnotes ----------
(1) https://wiki.tcl-lang.org/37432
File: python.info, Node: Tkinter Modules, Next: Tkinter Life Preserver, Prev: Architecture, Up: tkinter — Python interface to Tcl/Tk
5.26.1.2 Tkinter Modules
........................
Support for Tkinter is spread across several modules. Most applications
will need the main *note tkinter: f0. module, as well as the *note
tkinter.ttk: f9. module, which provides the modern themed widget set and
API:
from tkinter import *
from tkinter import ttk
-- Class: tkinter.Tk (screenName=None, baseName=None, className='Tk',
useTk=True, sync=False, use=None)
Construct a toplevel Tk widget, which is usually the main window of
an application, and initialize a Tcl interpreter for this widget.
Each instance has its own associated Tcl interpreter.
The *note Tk: 1636. class is typically instantiated using all
default values. However, the following keyword arguments are
currently recognized:
'screenName'
When given (as a string), sets the ‘DISPLAY’ environment
variable. (X11 only)
'baseName'
Name of the profile file. By default, 'baseName' is derived
from the program name (‘sys.argv[0]’).
'className'
Name of the widget class. Used as a profile file and also as
the name with which Tcl is invoked ('argv0' in 'interp').
'useTk'
If ‘True’, initialize the Tk subsystem. The *note
tkinter.Tcl(): 3e44. function sets this to ‘False’.
'sync'
If ‘True’, execute all X server commands synchronously, so
that errors are reported immediately. Can be used for
debugging. (X11 only)
'use'
Specifies the 'id' of the window in which to embed the
application, instead of it being created as an independent
toplevel window. 'id' must be specified in the same way as
the value for the -use option for toplevel widgets (that is,
it has a form like that returned by ‘winfo_id()’).
Note that on some platforms this will only work correctly if
'id' refers to a Tk frame or toplevel that has its -container
option enabled.
*note Tk: 1636. reads and interprets profile files, named
‘.`className'.tcl’ and ‘.`baseName'.tcl’, into the Tcl interpreter
and calls *note exec(): 17f. on the contents of ‘.`className'.py’
and ‘.`baseName'.py’. The path for the profile files is the ‘HOME’
environment variable or, if that isn’t defined, then *note
os.curdir: 27e4.
-- Attribute: tk
The Tk application object created by instantiating *note Tk:
1636. This provides access to the Tcl interpreter. Each
widget that is attached the same instance of *note Tk: 1636.
has the same value for its *note tk: 3e45. attribute.
-- Attribute: master
The widget object that contains this widget. For *note Tk:
1636, the 'master' is *note None: 671. because it is the main
window. The terms 'master' and 'parent' are similar and
sometimes used interchangeably as argument names; however,
calling ‘winfo_parent()’ returns a string of the widget name
whereas *note master: 3e46. returns the object.
'parent'/'child' reflects the tree-like relationship while
'master'/'slave' reflects the container structure.
-- Attribute: children
The immediate descendants of this widget as a *note dict: 258.
with the child widget names as the keys and the child instance
objects as the values.
-- Function: tkinter.Tcl (screenName=None, baseName=None,
className='Tk', useTk=False)
The *note Tcl(): 3e44. function is a factory function which creates
an object much like that created by the *note Tk: 1636. class,
except that it does not initialize the Tk subsystem. This is most
often useful when driving the Tcl interpreter in an environment
where one doesn’t want to create extraneous toplevel windows, or
where one cannot (such as Unix/Linux systems without an X server).
An object created by the *note Tcl(): 3e44. object can have a
Toplevel window created (and the Tk subsystem initialized) by
calling its ‘loadtk()’ method.
The modules that provide Tk support include:
*note tkinter: f0.
Main Tkinter module.
*note tkinter.colorchooser: f1.
Dialog to let the user choose a color.
*note tkinter.commondialog: f2.
Base class for the dialogs defined in the other modules listed
here.
*note tkinter.filedialog: f4.
Common dialogs to allow the user to specify a file to open or save.
*note tkinter.font: f5.
Utilities to help work with fonts.
*note tkinter.messagebox: f6.
Access to standard Tk dialog boxes.
*note tkinter.scrolledtext: f7.
Text widget with a vertical scroll bar built in.
*note tkinter.simpledialog: f8.
Basic dialogs and convenience functions.
*note tkinter.ttk: f9.
Themed widget set introduced in Tk 8.5, providing modern
alternatives for many of the classic widgets in the main *note
tkinter: f0. module.
Additional modules:
*note _tkinter: 3.
A binary module that contains the low-level interface to Tcl/Tk.
It is automatically imported by the main *note tkinter: f0. module,
and should never be used directly by application programmers. It
is usually a shared library (or DLL), but might in some cases be
statically linked with the Python interpreter.
*note idlelib: 73.
Python’s Integrated Development and Learning Environment (IDLE).
Based on *note tkinter: f0.
‘tkinter.constants’
Symbolic constants that can be used in place of strings when
passing various parameters to Tkinter calls. Automatically
imported by the main *note tkinter: f0. module.
*note tkinter.dnd: f3.
(experimental) Drag-and-drop support for *note tkinter: f0. This
will become deprecated when it is replaced with the Tk DND.
*note turtle: 101.
Turtle graphics in a Tk window.
File: python.info, Node: Tkinter Life Preserver, Next: Threading model, Prev: Tkinter Modules, Up: tkinter — Python interface to Tcl/Tk
5.26.1.3 Tkinter Life Preserver
...............................
This section is not designed to be an exhaustive tutorial on either Tk
or Tkinter. For that, refer to one of the external resources noted
earlier. Instead, this section provides a very quick orientation to
what a Tkinter application looks like, identifies foundational Tk
concepts, and explains how the Tkinter wrapper is structured.
The remainder of this section will help you to identify the classes,
methods, and options you’ll need in your Tkinter application, and where
to find more detailed documentation on them, including in the official
Tcl/Tk reference manual.
* Menu:
* A Hello World Program::
* Important Tk Concepts::
* Understanding How Tkinter Wraps Tcl/Tk::
* How do I…? What option does…?::
* Navigating the Tcl/Tk Reference Manual::
File: python.info, Node: A Hello World Program, Next: Important Tk Concepts, Up: Tkinter Life Preserver
5.26.1.4 A Hello World Program
..............................
We’ll start by walking through a “Hello World” application in Tkinter.
This isn’t the smallest one we could write, but has enough to illustrate
some key concepts you’ll need to know.
from tkinter import *
from tkinter import ttk
root = Tk()
frm = ttk.Frame(root, padding=10)
frm.grid()
ttk.Label(frm, text="Hello World!").grid(column=0, row=0)
ttk.Button(frm, text="Quit", command=root.destroy).grid(column=1, row=0)
root.mainloop()
After the imports, the next line creates an instance of the ‘Tk’ class,
which initializes Tk and creates its associated Tcl interpreter. It
also creates a toplevel window, known as the root window, which serves
as the main window of the application.
The following line creates a frame widget, which in this case will
contain a label and a button we’ll create next. The frame is fit inside
the root window.
The next line creates a label widget holding a static text string. The
‘grid()’ method is used to specify the relative layout (position) of the
label within its containing frame widget, similar to how tables in HTML
work.
A button widget is then created, and placed to the right of the label.
When pressed, it will call the ‘destroy()’ method of the root window.
Finally, the ‘mainloop()’ method puts everything on the display, and
responds to user input until the program terminates.
File: python.info, Node: Important Tk Concepts, Next: Understanding How Tkinter Wraps Tcl/Tk, Prev: A Hello World Program, Up: Tkinter Life Preserver
5.26.1.5 Important Tk Concepts
..............................
Even this simple program illustrates the following key Tk concepts:
widgets
A Tkinter user interface is made up of individual 'widgets'. Each
widget is represented as a Python object, instantiated from classes
like ‘ttk.Frame’, ‘ttk.Label’, and ‘ttk.Button’.
widget hierarchy
Widgets are arranged in a 'hierarchy'. The label and button were
contained within a frame, which in turn was contained within the
root window. When creating each 'child' widget, its 'parent'
widget is passed as the first argument to the widget constructor.
configuration options
Widgets have 'configuration options', which modify their appearance
and behavior, such as the text to display in a label or button.
Different classes of widgets will have different sets of options.
geometry management
Widgets aren’t automatically added to the user interface when they
are created. A 'geometry manager' like ‘grid’ controls where in
the user interface they are placed.
event loop
Tkinter reacts to user input, changes from your program, and even
refreshes the display only when actively running an 'event loop'.
If your program isn’t running the event loop, your user interface
won’t update.
File: python.info, Node: Understanding How Tkinter Wraps Tcl/Tk, Next: How do I…? What option does…?, Prev: Important Tk Concepts, Up: Tkinter Life Preserver
5.26.1.6 Understanding How Tkinter Wraps Tcl/Tk
...............................................
When your application uses Tkinter’s classes and methods, internally
Tkinter is assembling strings representing Tcl/Tk commands, and
executing those commands in the Tcl interpreter attached to your
application’s ‘Tk’ instance.
Whether it’s trying to navigate reference documentation, trying to find
the right method or option, adapting some existing code, or debugging
your Tkinter application, there are times that it will be useful to
understand what those underlying Tcl/Tk commands look like.
To illustrate, here is the Tcl/Tk equivalent of the main part of the
Tkinter script above.
ttk::frame .frm -padding 10
grid .frm
grid [ttk::label .frm.lbl -text "Hello World!"] -column 0 -row 0
grid [ttk::button .frm.btn -text "Quit" -command "destroy ."] -column 1 -row 0
Tcl’s syntax is similar to many shell languages, where the first word is
the command to be executed, with arguments to that command following it,
separated by spaces. Without getting into too many details, notice the
following:
* The commands used to create widgets (like ‘ttk::frame’) correspond
to widget classes in Tkinter.
* Tcl widget options (like ‘-text’) correspond to keyword arguments
in Tkinter.
* Widgets are referred to by a 'pathname' in Tcl (like ‘.frm.btn’),
whereas Tkinter doesn’t use names but object references.
* A widget’s place in the widget hierarchy is encoded in its
(hierarchical) pathname, which uses a ‘.’ (dot) as a path
separator. The pathname for the root window is just ‘.’ (dot). In
Tkinter, the hierarchy is defined not by pathname but by specifying
the parent widget when creating each child widget.
* Operations which are implemented as separate 'commands' in Tcl
(like ‘grid’ or ‘destroy’) are represented as 'methods' on Tkinter
widget objects. As you’ll see shortly, at other times Tcl uses
what appear to be method calls on widget objects, which more
closely mirror what would is used in Tkinter.
File: python.info, Node: How do I…? What option does…?, Next: Navigating the Tcl/Tk Reference Manual, Prev: Understanding How Tkinter Wraps Tcl/Tk, Up: Tkinter Life Preserver
5.26.1.7 How do I…? What option does…?
......................................
If you’re not sure how to do something in Tkinter, and you can’t
immediately find it in the tutorial or reference documentation you’re
using, there are a few strategies that can be helpful.
First, remember that the details of how individual widgets work may vary
across different versions of both Tkinter and Tcl/Tk. If you’re
searching documentation, make sure it corresponds to the Python and
Tcl/Tk versions installed on your system.
When searching for how to use an API, it helps to know the exact name of
the class, option, or method that you’re using. Introspection, either
in an interactive Python shell or with *note print(): f70, can help you
identify what you need.
To find out what configuration options are available on any widget, call
its ‘configure()’ method, which returns a dictionary containing a
variety of information about each object, including its default and
current values. Use ‘keys()’ to get just the names of each option.
btn = ttk.Button(frm, ...)
print(btn.configure().keys())
As most widgets have many configuration options in common, it can be
useful to find out which are specific to a particular widget class.
Comparing the list of options to that of a simpler widget, like a frame,
is one way to do that.
print(set(btn.configure().keys()) - set(frm.configure().keys()))
Similarly, you can find the available methods for a widget object using
the standard *note dir(): 62e. function. If you try it, you’ll see
there are over 200 common widget methods, so again identifying those
specific to a widget class is helpful.
print(dir(btn))
print(set(dir(btn)) - set(dir(frm)))
File: python.info, Node: Navigating the Tcl/Tk Reference Manual, Prev: How do I…? What option does…?, Up: Tkinter Life Preserver
5.26.1.8 Navigating the Tcl/Tk Reference Manual
...............................................
As noted, the official Tk commands(1) reference manual (man pages) is
often the most accurate description of what specific operations on
widgets do. Even when you know the name of the option or method that
you need, you may still have a few places to look.
While all operations in Tkinter are implemented as method calls on
widget objects, you’ve seen that many Tcl/Tk operations appear as
commands that take a widget pathname as its first parameter, followed by
optional parameters, e.g.
destroy .
grid .frm.btn -column 0 -row 0
Others, however, look more like methods called on a widget object (in
fact, when you create a widget in Tcl/Tk, it creates a Tcl command with
the name of the widget pathname, with the first parameter to that
command being the name of a method to call).
.frm.btn invoke
.frm.lbl configure -text "Goodbye"
In the official Tcl/Tk reference documentation, you’ll find most
operations that look like method calls on the man page for a specific
widget (e.g., you’ll find the ‘invoke()’ method on the ttk::button(2)
man page), while functions that take a widget as a parameter often have
their own man page (e.g., grid(3)).
You’ll find many common options and methods in the options(4) or
ttk::widget(5) man pages, while others are found in the man page for a
specific widget class.
You’ll also find that many Tkinter methods have compound names, e.g.,
‘winfo_x()’, ‘winfo_height()’, ‘winfo_viewable()’. You’d find
documentation for all of these in the winfo(6) man page.
Note: Somewhat confusingly, there are also methods on all Tkinter
widgets that don’t actually operate on the widget, but operate at a
global scope, independent of any widget. Examples are methods for
accessing the clipboard or the system bell. (They happen to be
implemented as methods in the base ‘Widget’ class that all Tkinter
widgets inherit from).
---------- Footnotes ----------
(1) https://www.tcl.tk/man/tcl8.6/TkCmd/contents.htm
(2) https://www.tcl.tk/man/tcl8.6/TkCmd/ttk_button.htm
(3) https://www.tcl.tk/man/tcl8.6/TkCmd/grid.htm
(4) https://www.tcl.tk/man/tcl8.6/TkCmd/options.htm
(5) https://www.tcl.tk/man/tcl8.6/TkCmd/ttk_widget.htm
(6) https://www.tcl.tk/man/tcl8.6/TkCmd/winfo.htm
File: python.info, Node: Threading model, Next: Handy Reference, Prev: Tkinter Life Preserver, Up: tkinter — Python interface to Tcl/Tk
5.26.1.9 Threading model
........................
Python and Tcl/Tk have very different threading models, which *note
tkinter: f0. tries to bridge. If you use threads, you may need to be
aware of this.
A Python interpreter may have many threads associated with it. In Tcl,
multiple threads can be created, but each thread has a separate Tcl
interpreter instance associated with it. Threads can also create more
than one interpreter instance, though each interpreter instance can be
used only by the one thread that created it.
Each ‘Tk’ object created by *note tkinter: f0. contains a Tcl
interpreter. It also keeps track of which thread created that
interpreter. Calls to *note tkinter: f0. can be made from any Python
thread. Internally, if a call comes from a thread other than the one
that created the ‘Tk’ object, an event is posted to the interpreter’s
event queue, and when executed, the result is returned to the calling
Python thread.
Tcl/Tk applications are normally event-driven, meaning that after
initialization, the interpreter runs an event loop (i.e.
‘Tk.mainloop()’) and responds to events. Because it is single-threaded,
event handlers must respond quickly, otherwise they will block other
events from being processed. To avoid this, any long-running
computations should not run in an event handler, but are either broken
into smaller pieces using timers, or run in another thread. This is
different from many GUI toolkits where the GUI runs in a completely
separate thread from all application code including event handlers.
If the Tcl interpreter is not running the event loop and processing
events, any *note tkinter: f0. calls made from threads other than the
one running the Tcl interpreter will fail.
A number of special cases exist:
* Tcl/Tk libraries can be built so they are not thread-aware. In
this case, *note tkinter: f0. calls the library from the
originating Python thread, even if this is different than the
thread that created the Tcl interpreter. A global lock ensures
only one call occurs at a time.
* While *note tkinter: f0. allows you to create more than one
instance of a ‘Tk’ object (with its own interpreter), all
interpreters that are part of the same thread share a common event
queue, which gets ugly fast. In practice, don’t create more than
one instance of ‘Tk’ at a time. Otherwise, it’s best to create
them in separate threads and ensure you’re running a thread-aware
Tcl/Tk build.
* Blocking event handlers are not the only way to prevent the Tcl
interpreter from reentering the event loop. It is even possible to
run multiple nested event loops or abandon the event loop entirely.
If you’re doing anything tricky when it comes to events or threads,
be aware of these possibilities.
* There are a few select *note tkinter: f0. functions that presently
work only when called from the thread that created the Tcl
interpreter.
File: python.info, Node: Handy Reference, Next: File Handlers, Prev: Threading model, Up: tkinter — Python interface to Tcl/Tk
5.26.1.10 Handy Reference
.........................
* Menu:
* Setting Options::
* The Packer::
* Packer Options::
* Coupling Widget Variables::
* The Window Manager::
* Tk Option Data Types::
* Bindings and Events::
* The index Parameter::
* Images::
File: python.info, Node: Setting Options, Next: The Packer, Up: Handy Reference
5.26.1.11 Setting Options
.........................
Options control things like the color and border width of a widget.
Options can be set in three ways:
At object creation time, using keyword arguments
fred = Button(self, fg="red", bg="blue")
After object creation, treating the option name like a dictionary index
fred["fg"] = "red"
fred["bg"] = "blue"
Use the config() method to update multiple attrs subsequent to object creation
fred.config(fg="red", bg="blue")
For a complete explanation of a given option and its behavior, see the
Tk man pages for the widget in question.
Note that the man pages list “STANDARD OPTIONS” and “WIDGET SPECIFIC
OPTIONS” for each widget. The former is a list of options that are
common to many widgets, the latter are the options that are
idiosyncratic to that particular widget. The Standard Options are
documented on the ‘options(3)(1)’ man page.
No distinction between standard and widget-specific options is made in
this document. Some options don’t apply to some kinds of widgets.
Whether a given widget responds to a particular option depends on the
class of the widget; buttons have a ‘command’ option, labels do not.
The options supported by a given widget are listed in that widget’s man
page, or can be queried at runtime by calling the ‘config()’ method
without arguments, or by calling the ‘keys()’ method on that widget.
The return value of these calls is a dictionary whose key is the name of
the option as a string (for example, ‘'relief'’) and whose values are
5-tuples.
Some options, like ‘bg’ are synonyms for common options with long names
(‘bg’ is shorthand for “background”). Passing the ‘config()’ method the
name of a shorthand option will return a 2-tuple, not 5-tuple. The
2-tuple passed back will contain the name of the synonym and the “real”
option (such as ‘('bg', 'background')’).
Index Meaning Example
---------------------------------------------------------------------
0 option name ‘'relief'’
1 option name for database lookup ‘'relief'’
2 option class for database lookup ‘'Relief'’
3 default value ‘'raised'’
4 current value ‘'groove'’
Example:
>>> print(fred.config())
{'relief': ('relief', 'relief', 'Relief', 'raised', 'groove')}
Of course, the dictionary printed will include all the options available
and their values. This is meant only as an example.
---------- Footnotes ----------
(1) https://manpages.debian.org/options(3)
File: python.info, Node: The Packer, Next: Packer Options, Prev: Setting Options, Up: Handy Reference
5.26.1.12 The Packer
....................
The packer is one of Tk’s geometry-management mechanisms. Geometry
managers are used to specify the relative positioning of widgets within
their container - their mutual 'master'. In contrast to the more
cumbersome 'placer' (which is used less commonly, and we do not cover
here), the packer takes qualitative relationship specification -
'above', 'to the left of', 'filling', etc - and works everything out to
determine the exact placement coordinates for you.
The size of any 'master' widget is determined by the size of the “slave
widgets” inside. The packer is used to control where slave widgets
appear inside the master into which they are packed. You can pack
widgets into frames, and frames into other frames, in order to achieve
the kind of layout you desire. Additionally, the arrangement is
dynamically adjusted to accommodate incremental changes to the
configuration, once it is packed.
Note that widgets do not appear until they have had their geometry
specified with a geometry manager. It’s a common early mistake to leave
out the geometry specification, and then be surprised when the widget is
created but nothing appears. A widget will appear only after it has
had, for example, the packer’s ‘pack()’ method applied to it.
The pack() method can be called with keyword-option/value pairs that
control where the widget is to appear within its container, and how it
is to behave when the main application window is resized. Here are some
examples:
fred.pack() # defaults to side = "top"
fred.pack(side="left")
fred.pack(expand=1)
File: python.info, Node: Packer Options, Next: Coupling Widget Variables, Prev: The Packer, Up: Handy Reference
5.26.1.13 Packer Options
........................
For more extensive information on the packer and the options that it can
take, see the man pages and page 183 of John Ousterhout’s book.
anchor
Anchor type. Denotes where the packer is to place each slave in
its parcel.
expand
Boolean, ‘0’ or ‘1’.
fill
Legal values: ‘'x'’, ‘'y'’, ‘'both'’, ‘'none'’.
ipadx and ipady
A distance - designating internal padding on each side of the slave
widget.
padx and pady
A distance - designating external padding on each side of the slave
widget.
side
Legal values are: ‘'left'’, ‘'right'’, ‘'top'’, ‘'bottom'’.
File: python.info, Node: Coupling Widget Variables, Next: The Window Manager, Prev: Packer Options, Up: Handy Reference
5.26.1.14 Coupling Widget Variables
...................................
The current-value setting of some widgets (like text entry widgets) can
be connected directly to application variables by using special options.
These options are ‘variable’, ‘textvariable’, ‘onvalue’, ‘offvalue’, and
‘value’. This connection works both ways: if the variable changes for
any reason, the widget it’s connected to will be updated to reflect the
new value.
Unfortunately, in the current implementation of *note tkinter: f0. it is
not possible to hand over an arbitrary Python variable to a widget
through a ‘variable’ or ‘textvariable’ option. The only kinds of
variables for which this works are variables that are subclassed from a
class called Variable, defined in *note tkinter: f0.
There are many useful subclasses of Variable already defined:
‘StringVar’, ‘IntVar’, ‘DoubleVar’, and ‘BooleanVar’. To read the
current value of such a variable, call the ‘get()’ method on it, and to
change its value you call the ‘set()’ method. If you follow this
protocol, the widget will always track the value of the variable, with
no further intervention on your part.
For example:
import tkinter as tk
class App(tk.Frame):
def __init__(self, master):
super().__init__(master)
self.pack()
self.entrythingy = tk.Entry()
self.entrythingy.pack()
# Create the application variable.
self.contents = tk.StringVar()
# Set it to some value.
self.contents.set("this is a variable")
# Tell the entry widget to watch this variable.
self.entrythingy["textvariable"] = self.contents
# Define a callback for when the user hits return.
# It prints the current value of the variable.
self.entrythingy.bind('',
self.print_contents)
def print_contents(self, event):
print("Hi. The current entry content is:",
self.contents.get())
root = tk.Tk()
myapp = App(root)
myapp.mainloop()
File: python.info, Node: The Window Manager, Next: Tk Option Data Types, Prev: Coupling Widget Variables, Up: Handy Reference
5.26.1.15 The Window Manager
............................
In Tk, there is a utility command, ‘wm’, for interacting with the window
manager. Options to the ‘wm’ command allow you to control things like
titles, placement, icon bitmaps, and the like. In *note tkinter: f0,
these commands have been implemented as methods on the ‘Wm’ class.
Toplevel widgets are subclassed from the ‘Wm’ class, and so can call the
‘Wm’ methods directly.
To get at the toplevel window that contains a given widget, you can
often just refer to the widget’s master. Of course if the widget has
been packed inside of a frame, the master won’t represent a toplevel
window. To get at the toplevel window that contains an arbitrary
widget, you can call the ‘_root()’ method. This method begins with an
underscore to denote the fact that this function is part of the
implementation, and not an interface to Tk functionality.
Here are some examples of typical usage:
import tkinter as tk
class App(tk.Frame):
def __init__(self, master=None):
super().__init__(master)
self.pack()
# create the application
myapp = App()
#
# here are method calls to the window manager class
#
myapp.master.title("My Do-Nothing Application")
myapp.master.maxsize(1000, 400)
# start the program
myapp.mainloop()
File: python.info, Node: Tk Option Data Types, Next: Bindings and Events, Prev: The Window Manager, Up: Handy Reference
5.26.1.16 Tk Option Data Types
..............................
anchor
Legal values are points of the compass: ‘"n"’, ‘"ne"’, ‘"e"’,
‘"se"’, ‘"s"’, ‘"sw"’, ‘"w"’, ‘"nw"’, and also ‘"center"’.
bitmap
There are eight built-in, named bitmaps: ‘'error'’, ‘'gray25'’,
‘'gray50'’, ‘'hourglass'’, ‘'info'’, ‘'questhead'’, ‘'question'’,
‘'warning'’. To specify an X bitmap filename, give the full path
to the file, preceded with an ‘@’, as in
‘"@/usr/contrib/bitmap/gumby.bit"’.
boolean
You can pass integers 0 or 1 or the strings ‘"yes"’ or ‘"no"’.
callback
This is any Python function that takes no arguments. For example:
def print_it():
print("hi there")
fred["command"] = print_it
color
Colors can be given as the names of X colors in the rgb.txt file,
or as strings representing RGB values in 4 bit: ‘"#RGB"’, 8 bit:
‘"#RRGGBB"’, 12 bit: ‘"#RRRGGGBBB"’, or 16 bit: ‘"#RRRRGGGGBBBB"’
ranges, where R,G,B here represent any legal hex digit. See page
160 of Ousterhout’s book for details.
cursor
The standard X cursor names from ‘cursorfont.h’ can be used,
without the ‘XC_’ prefix. For example to get a hand cursor
(‘XC_hand2’), use the string ‘"hand2"’. You can also specify a
bitmap and mask file of your own. See page 179 of Ousterhout’s
book.
distance
Screen distances can be specified in either pixels or absolute
distances. Pixels are given as numbers and absolute distances as
strings, with the trailing character denoting units: ‘c’ for
centimetres, ‘i’ for inches, ‘m’ for millimetres, ‘p’ for printer’s
points. For example, 3.5 inches is expressed as ‘"3.5i"’.
font
Tk uses a list font name format, such as ‘{courier 10 bold}’. Font
sizes with positive numbers are measured in points; sizes with
negative numbers are measured in pixels.
geometry
This is a string of the form ‘widthxheight’, where width and height
are measured in pixels for most widgets (in characters for widgets
displaying text). For example: ‘fred["geometry"] = "200x100"’.
justify
Legal values are the strings: ‘"left"’, ‘"center"’, ‘"right"’, and
‘"fill"’.
region
This is a string with four space-delimited elements, each of which
is a legal distance (see above). For example: ‘"2 3 4 5"’ and ‘"3i
2i 4.5i 2i"’ and ‘"3c 2c 4c 10.43c"’ are all legal regions.
relief
Determines what the border style of a widget will be. Legal values
are: ‘"raised"’, ‘"sunken"’, ‘"flat"’, ‘"groove"’, and ‘"ridge"’.
scrollcommand
This is almost always the ‘set()’ method of some scrollbar widget,
but can be any widget method that takes a single argument.
wrap
Must be one of: ‘"none"’, ‘"char"’, or ‘"word"’.
File: python.info, Node: Bindings and Events, Next: The index Parameter, Prev: Tk Option Data Types, Up: Handy Reference
5.26.1.17 Bindings and Events
.............................
The bind method from the widget command allows you to watch for certain
events and to have a callback function trigger when that event type
occurs. The form of the bind method is:
def bind(self, sequence, func, add=''):
where:
sequence
is a string that denotes the target kind of event. (See the
‘bind(3tk)(1)’ man page, and page 201 of John Ousterhout’s book,
‘Tcl and the Tk Toolkit (2nd edition)’, for details).
func
is a Python function, taking one argument, to be invoked when the
event occurs. An Event instance will be passed as the argument.
(Functions deployed this way are commonly known as 'callbacks'.)
add
is optional, either ‘''’ or ‘'+'’. Passing an empty string denotes
that this binding is to replace any other bindings that this event
is associated with. Passing a ‘'+'’ means that this function is to
be added to the list of functions bound to this event type.
For example:
def turn_red(self, event):
event.widget["activeforeground"] = "red"
self.button.bind("", self.turn_red)
Notice how the widget field of the event is being accessed in the
‘turn_red()’ callback. This field contains the widget that caught the X
event. The following table lists the other event fields you can access,
and how they are denoted in Tk, which can be useful when referring to
the Tk man pages.
Tk Tkinter Event Field Tk Tkinter Event Field
----------------------------------------------------------------------
%f focus %A char
%h height %E send_event
%k keycode %K keysym
%s state %N keysym_num
%t time %T type
%w width %W widget
%x x %X x_root
%y y %Y y_root
---------- Footnotes ----------
(1) https://manpages.debian.org/bind(3tk)
File: python.info, Node: The index Parameter, Next: Images, Prev: Bindings and Events, Up: Handy Reference
5.26.1.18 The index Parameter
.............................
A number of widgets require “index” parameters to be passed. These are
used to point at a specific place in a Text widget, or to particular
characters in an Entry widget, or to particular menu items in a Menu
widget.
Entry widget indexes (index, view index, etc.)
Entry widgets have options that refer to character positions in the
text being displayed. You can use these *note tkinter: f0.
functions to access these special points in text widgets:
Text widget indexes
The index notation for Text widgets is very rich and is best
described in the Tk man pages.
Menu indexes (menu.invoke(), menu.entryconfig(), etc.)
Some options and methods for menus manipulate specific menu
entries. Anytime a menu index is needed for an option or a
parameter, you may pass in:
* an integer which refers to the numeric position of the entry
in the widget, counted from the top, starting with 0;
* the string ‘"active"’, which refers to the menu position that
is currently under the cursor;
* the string ‘"last"’ which refers to the last menu item;
* An integer preceded by ‘@’, as in ‘@6’, where the integer is
interpreted as a y pixel coordinate in the menu’s coordinate
system;
* the string ‘"none"’, which indicates no menu entry at all,
most often used with menu.activate() to deactivate all
entries, and finally,
* a text string that is pattern matched against the label of the
menu entry, as scanned from the top of the menu to the bottom.
Note that this index type is considered after all the others,
which means that matches for menu items labelled ‘last’,
‘active’, or ‘none’ may be interpreted as the above literals,
instead.
File: python.info, Node: Images, Prev: The index Parameter, Up: Handy Reference
5.26.1.19 Images
................
Images of different formats can be created through the corresponding
subclass of ‘tkinter.Image’:
* ‘BitmapImage’ for images in XBM format.
* ‘PhotoImage’ for images in PGM, PPM, GIF and PNG formats. The
latter is supported starting with Tk 8.6.
Either type of image is created through either the ‘file’ or the ‘data’
option (other options are available as well).
Changed in version 3.13: Added the ‘PhotoImage’ method ‘copy_replace()’
to copy a region from one image to other image, possibly with pixel
zooming and/or subsampling. Add 'from_coords' parameter to ‘PhotoImage’
methods ‘copy()’, ‘zoom()’ and ‘subsample()’. Add 'zoom' and
'subsample' parameters to ‘PhotoImage’ method ‘copy()’.
The image object can then be used wherever an ‘image’ option is
supported by some widget (e.g. labels, buttons, menus). In these
cases, Tk will not keep a reference to the image. When the last Python
reference to the image object is deleted, the image data is deleted as
well, and Tk will display an empty box wherever the image was used.
See also
........
The Pillow(1) package adds support for formats such as BMP, JPEG, TIFF,
and WebP, among others.
---------- Footnotes ----------
(1) https://python-pillow.org/
File: python.info, Node: File Handlers, Prev: Handy Reference, Up: tkinter — Python interface to Tcl/Tk
5.26.1.20 File Handlers
.......................
Tk allows you to register and unregister a callback function which will
be called from the Tk mainloop when I/O is possible on a file
descriptor. Only one handler may be registered per file descriptor.
Example code:
import tkinter
widget = tkinter.Tk()
mask = tkinter.READABLE | tkinter.WRITABLE
widget.tk.createfilehandler(file, mask, callback)
...
widget.tk.deletefilehandler(file)
This feature is not available on Windows.
Since you don’t know how many bytes are available for reading, you may
not want to use the *note BufferedIOBase: 690. or *note TextIOBase: 691.
*note read(): 131b. or *note readline(): 131c. methods, since these will
insist on reading a predefined number of bytes. For sockets, the *note
recv(): da3. or *note recvfrom(): da4. methods will work fine; for other
files, use raw reads or ‘os.read(file.fileno(), maxbytecount)’.
-- Method: Widget.tk.createfilehandler (file, mask, func)
Registers the file handler callback function 'func'. The 'file'
argument may either be an object with a *note fileno(): 2856.
method (such as a file or socket object), or an integer file
descriptor. The 'mask' argument is an ORed combination of any of
the three constants below. The callback is called as follows:
callback(file, mask)
-- Method: Widget.tk.deletefilehandler (file)
Unregisters a file handler.
-- Data: _tkinter.READABLE
-- Data: _tkinter.WRITABLE
-- Data: _tkinter.EXCEPTION
Constants used in the 'mask' arguments.
File: python.info, Node: tkinter colorchooser — Color choosing dialog, Next: tkinter font — Tkinter font wrapper, Prev: tkinter — Python interface to Tcl/Tk, Up: Graphical User Interfaces with Tk
5.26.2 ‘tkinter.colorchooser’ — Color choosing dialog
-----------------------------------------------------
'Source code:' Lib/tkinter/colorchooser.py(1)
__________________________________________________________________
The *note tkinter.colorchooser: f1. module provides the *note Chooser:
3e63. class as an interface to the native color picker dialog.
‘Chooser’ implements a modal color choosing dialog window. The
‘Chooser’ class inherits from the *note Dialog: 3e64. class.
-- Class: tkinter.colorchooser.Chooser (master=None, **options)
-- Function: tkinter.colorchooser.askcolor (color=None, **options)
Create a color choosing dialog. A call to this method will show
the window, wait for the user to make a selection, and return the
selected color (or ‘None’) to the caller.
See also
........
Module *note tkinter.commondialog: f2.
Tkinter standard dialog module
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/tkinter/colorchooser.py
File: python.info, Node: tkinter font — Tkinter font wrapper, Next: Tkinter Dialogs, Prev: tkinter colorchooser — Color choosing dialog, Up: Graphical User Interfaces with Tk
5.26.3 ‘tkinter.font’ — Tkinter font wrapper
--------------------------------------------
'Source code:' Lib/tkinter/font.py(1)
__________________________________________________________________
The *note tkinter.font: f5. module provides the *note Font: 190a. class
for creating and using named fonts.
The different font weights and slants are:
-- Data: tkinter.font.NORMAL
-- Data: tkinter.font.BOLD
-- Data: tkinter.font.ITALIC
-- Data: tkinter.font.ROMAN
-- Class: tkinter.font.Font (root=None, font=None, name=None,
exists=False, **options)
The *note Font: 190a. class represents a named font. 'Font'
instances are given unique names and can be specified by their
family, size, and style configuration. Named fonts are Tk’s method
of creating and identifying fonts as a single object, rather than
specifying a font by its attributes with each occurrence.
arguments:
'font' - font specifier tuple (family, size, options)
'name' - unique font name
'exists' - self points to existing named font if true
additional keyword options (ignored if 'font' is specified):
'family' - font family i.e. Courier, Times
'size' - font size
If 'size' is positive it is interpreted as size in points.
If 'size' is a negative number its absolute value is treated
as size in pixels.
'weight' - font emphasis (NORMAL, BOLD)
'slant' - ROMAN, ITALIC
'underline' - font underlining (0 - none, 1 - underline)
'overstrike' - font strikeout (0 - none, 1 - strikeout)
-- Method: actual (option=None, displayof=None)
Return the attributes of the font.
-- Method: cget (option)
Retrieve an attribute of the font.
-- Method: config (**options)
Modify attributes of the font.
-- Method: copy ()
Return new instance of the current font.
-- Method: measure (text, displayof=None)
Return amount of space the text would occupy on the specified
display when formatted in the current font. If no display is
specified then the main application window is assumed.
-- Method: metrics (*options, **kw)
Return font-specific data. Options include:
'ascent' - distance between baseline and highest point that a
character of the font can occupy
'descent' - distance between baseline and lowest point that a
character of the font can occupy
'linespace' - minimum vertical separation necessary between any two
characters of the font that ensures no vertical overlap
between lines.
'fixed' - 1 if font is fixed-width else 0
-- Function: tkinter.font.families (root=None, displayof=None)
Return the different font families.
-- Function: tkinter.font.names (root=None)
Return the names of defined fonts.
-- Function: tkinter.font.nametofont (name, root=None)
Return a *note Font: 190a. representation of a tk named font.
Changed in version 3.10: The 'root' parameter was added.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/tkinter/font.py
File: python.info, Node: Tkinter Dialogs, Next: tkinter messagebox — Tkinter message prompts, Prev: tkinter font — Tkinter font wrapper, Up: Graphical User Interfaces with Tk
5.26.4 Tkinter Dialogs
----------------------
* Menu:
* tkinter.simpledialog — Standard Tkinter input dialogs: tkinter simpledialog — Standard Tkinter input dialogs.
* tkinter.filedialog — File selection dialogs: tkinter filedialog — File selection dialogs.
* tkinter.commondialog — Dialog window templates: tkinter commondialog — Dialog window templates.
File: python.info, Node: tkinter simpledialog — Standard Tkinter input dialogs, Next: tkinter filedialog — File selection dialogs, Up: Tkinter Dialogs
5.26.4.1 ‘tkinter.simpledialog’ — Standard Tkinter input dialogs
................................................................
'Source code:' Lib/tkinter/simpledialog.py(1)
__________________________________________________________________
The *note tkinter.simpledialog: f8. module contains convenience classes
and functions for creating simple modal dialogs to get a value from the
user.
-- Function: tkinter.simpledialog.askfloat (title, prompt, **kw)
-- Function: tkinter.simpledialog.askinteger (title, prompt, **kw)
-- Function: tkinter.simpledialog.askstring (title, prompt, **kw)
The above three functions provide dialogs that prompt the user to
enter a value of the desired type.
-- Class: tkinter.simpledialog.Dialog (parent, title=None)
The base class for custom dialogs.
-- Method: body (master)
Override to construct the dialog’s interface and return the
widget that should have initial focus.
-- Method: buttonbox ()
Default behaviour adds OK and Cancel buttons. Override for
custom button layouts.
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/tkinter/simpledialog.py
File: python.info, Node: tkinter filedialog — File selection dialogs, Next: tkinter commondialog — Dialog window templates, Prev: tkinter simpledialog — Standard Tkinter input dialogs, Up: Tkinter Dialogs
5.26.4.2 ‘tkinter.filedialog’ — File selection dialogs
......................................................
'Source code:' Lib/tkinter/filedialog.py(1)
__________________________________________________________________
The *note tkinter.filedialog: f4. module provides classes and factory
functions for creating file/directory selection windows.
* Menu:
* Native Load/Save Dialogs::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/tkinter/filedialog.py
File: python.info, Node: Native Load/Save Dialogs, Up: tkinter filedialog — File selection dialogs
5.26.4.3 Native Load/Save Dialogs
.................................
The following classes and functions provide file dialog windows that
combine a native look-and-feel with configuration options to customize
behaviour. The following keyword arguments are applicable to the
classes and functions listed below:
'parent' - the window to place the dialog on top of
'title' - the title of the window
'initialdir' - the directory that the dialog starts in
'initialfile' - the file selected upon opening of the dialog
'filetypes' - a sequence of (label, pattern) tuples, ‘*’ wildcard is allowed
'defaultextension' - default extension to append to file (save dialogs)
'multiple' - when true, selection of multiple items is allowed
'Static factory functions'
The below functions when called create a modal, native look-and-feel
dialog, wait for the user’s selection, then return the selected value(s)
or ‘None’ to the caller.
-- Function: tkinter.filedialog.askopenfile (mode='r', **options)
-- Function: tkinter.filedialog.askopenfiles (mode='r', **options)
The above two functions create an *note Open: 3e80. dialog and
return the opened file object(s) in read-only mode.
-- Function: tkinter.filedialog.asksaveasfile (mode='w', **options)
Create a *note SaveAs: 3e82. dialog and return a file object opened
in write-only mode.
-- Function: tkinter.filedialog.askopenfilename (**options)
-- Function: tkinter.filedialog.askopenfilenames (**options)
The above two functions create an *note Open: 3e80. dialog and
return the selected filename(s) that correspond to existing
file(s).
-- Function: tkinter.filedialog.asksaveasfilename (**options)
Create a *note SaveAs: 3e82. dialog and return the selected
filename.
-- Function: tkinter.filedialog.askdirectory (**options)
Prompt user to select a directory.
Additional keyword option:
'mustexist' - determines if selection must be an existing directory.
-- Class: tkinter.filedialog.Open (master=None, **options)
-- Class: tkinter.filedialog.SaveAs (master=None, **options)
The above two classes provide native dialog windows for saving and
loading files.
'Convenience classes'
The below classes are used for creating file/directory windows from
scratch. These do not emulate the native look-and-feel of the platform.
-- Class: tkinter.filedialog.Directory (master=None, **options)
Create a dialog prompting the user to select a directory.
Note: The 'FileDialog' class should be subclassed for custom event
handling and behaviour.
-- Class: tkinter.filedialog.FileDialog (master, title=None)
Create a basic file selection dialog.
-- Method: cancel_command (event=None)
Trigger the termination of the dialog window.
-- Method: dirs_double_event (event)
Event handler for double-click event on directory.
-- Method: dirs_select_event (event)
Event handler for click event on directory.
-- Method: files_double_event (event)
Event handler for double-click event on file.
-- Method: files_select_event (event)
Event handler for single-click event on file.
-- Method: filter_command (event=None)
Filter the files by directory.
-- Method: get_filter ()
Retrieve the file filter currently in use.
-- Method: get_selection ()
Retrieve the currently selected item.
-- Method: go (dir_or_file=os.curdir, pattern='*', default='',
key=None)
Render dialog and start event loop.
-- Method: ok_event (event)
Exit dialog returning current selection.
-- Method: quit (how=None)
Exit dialog returning filename, if any.
-- Method: set_filter (dir, pat)
Set the file filter.
-- Method: set_selection (file)
Update the current file selection to 'file'.
-- Class: tkinter.filedialog.LoadFileDialog (master, title=None)
A subclass of FileDialog that creates a dialog window for selecting
an existing file.
-- Method: ok_command ()
Test that a file is provided and that the selection indicates
an already existing file.
-- Class: tkinter.filedialog.SaveFileDialog (master, title=None)
A subclass of FileDialog that creates a dialog window for selecting
a destination file.
-- Method: ok_command ()
Test whether or not the selection points to a valid file that
is not a directory. Confirmation is required if an already
existing file is selected.
File: python.info, Node: tkinter commondialog — Dialog window templates, Prev: tkinter filedialog — File selection dialogs, Up: Tkinter Dialogs
5.26.4.4 ‘tkinter.commondialog’ — Dialog window templates
.........................................................
'Source code:' Lib/tkinter/commondialog.py(1)
__________________________________________________________________
The *note tkinter.commondialog: f2. module provides the *note Dialog:
3e64. class that is the base class for dialogs defined in other
supporting modules.
-- Class: tkinter.commondialog.Dialog (master=None, **options)
-- Method: show (**options)
Render the Dialog window.
See also
........
Modules *note tkinter.messagebox: f6, *note Reading and Writing Files:
1c7f.
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/tkinter/commondialog.py
File: python.info, Node: tkinter messagebox — Tkinter message prompts, Next: tkinter scrolledtext — Scrolled Text Widget, Prev: Tkinter Dialogs, Up: Graphical User Interfaces with Tk
5.26.5 ‘tkinter.messagebox’ — Tkinter message prompts
-----------------------------------------------------
'Source code:' Lib/tkinter/messagebox.py(1)
__________________________________________________________________
The *note tkinter.messagebox: f6. module provides a template base class
as well as a variety of convenience methods for commonly used
configurations. The message boxes are modal and will return a subset of
(‘True’, ‘False’, ‘None’, *note OK: 3e9e, *note CANCEL: 3e9f, *note YES:
3ea0, *note NO: 3ea1.) based on the user’s selection. Common message
box styles and layouts include but are not limited to:
[image src="python-figures/tk_msg.png" ]
Figure
-- Class: tkinter.messagebox.Message (master=None, **options)
Create a message window with an application-specified message, an
icon and a set of buttons. Each of the buttons in the message
window is identified by a unique symbolic name (see the 'type'
options).
The following options are supported:
'command'
Specifies the function to invoke when the user closes the
dialog. The name of the button clicked by the user to
close the dialog is passed as argument. This is only
available on macOS.
'default'
Gives the *note symbolic name: 3ea3. of the default
button for this message window (*note OK: 3e9e, *note
CANCEL: 3e9f, and so on). If this option is not
specified, the first button in the dialog will be made
the default.
'detail'
Specifies an auxiliary message to the main message given
by the 'message' option. The message detail will be
presented beneath the main message and, where supported
by the OS, in a less emphasized font than the main
message.
'icon'
Specifies an *note icon: 3ea4. to display. If this
option is not specified, then the *note INFO: 3ea5. icon
will be displayed.
'message'
Specifies the message to display in this message box.
The default value is an empty string.
'parent'
Makes the specified window the logical parent of the
message box. The message box is displayed on top of its
parent window.
'title'
Specifies a string to display as the title of the message
box. This option is ignored on macOS, where platform
guidelines forbid the use of a title on this kind of
dialog.
'type'
Arranges for a *note predefined set of buttons: 3ea6. to
be displayed.
-- Method: show (**options)
Display a message window and wait for the user to select one
of the buttons. Then return the symbolic name of the selected
button. Keyword arguments can override options specified in
the constructor.
'Information message box'
-- Function: tkinter.messagebox.showinfo (title=None, message=None,
**options)
Creates and displays an information message box with the specified
title and message.
'Warning message boxes'
-- Function: tkinter.messagebox.showwarning (title=None, message=None,
**options)
Creates and displays a warning message box with the specified title
and message.
-- Function: tkinter.messagebox.showerror (title=None, message=None,
**options)
Creates and displays an error message box with the specified title
and message.
'Question message boxes'
-- Function: tkinter.messagebox.askquestion (title=None, message=None,
*, type=YESNO, **options)
Ask a question. By default shows buttons *note YES: 3ea0. and
*note NO: 3ea1. Returns the symbolic name of the selected button.
-- Function: tkinter.messagebox.askokcancel (title=None, message=None,
**options)
Ask if operation should proceed. Shows buttons *note OK: 3e9e. and
*note CANCEL: 3e9f. Returns ‘True’ if the answer is ok and ‘False’
otherwise.
-- Function: tkinter.messagebox.askretrycancel (title=None,
message=None, **options)
Ask if operation should be retried. Shows buttons *note RETRY:
3eae. and *note CANCEL: 3e9f. Return ‘True’ if the answer is yes
and ‘False’ otherwise.
-- Function: tkinter.messagebox.askyesno (title=None, message=None,
**options)
Ask a question. Shows buttons *note YES: 3ea0. and *note NO: 3ea1.
Returns ‘True’ if the answer is yes and ‘False’ otherwise.
-- Function: tkinter.messagebox.askyesnocancel (title=None,
message=None, **options)
Ask a question. Shows buttons *note YES: 3ea0, *note NO: 3ea1. and
*note CANCEL: 3e9f. Return ‘True’ if the answer is yes, ‘None’ if
cancelled, and ‘False’ otherwise.
Symbolic names of buttons:
-- Data: tkinter.messagebox.ABORT = 'abort'
-- Data: tkinter.messagebox.RETRY = 'retry'
-- Data: tkinter.messagebox.IGNORE = 'ignore'
-- Data: tkinter.messagebox.OK = 'ok'
-- Data: tkinter.messagebox.CANCEL = 'cancel'
-- Data: tkinter.messagebox.YES = 'yes'
-- Data: tkinter.messagebox.NO = 'no'
Predefined sets of buttons:
-- Data: tkinter.messagebox.ABORTRETRYIGNORE = 'abortretryignore'
Displays three buttons whose symbolic names are *note ABORT: 3eb1,
*note RETRY: 3eae. and *note IGNORE: 3eb2.
-- Data: tkinter.messagebox.OK = 'ok'
Displays one button whose symbolic name is *note OK: 3e9e.
-- Data: tkinter.messagebox.OKCANCEL = 'okcancel'
Displays two buttons whose symbolic names are *note OK: 3e9e. and
*note CANCEL: 3e9f.
-- Data: tkinter.messagebox.RETRYCANCEL = 'retrycancel'
Displays two buttons whose symbolic names are *note RETRY: 3eae.
and *note CANCEL: 3e9f.
-- Data: tkinter.messagebox.YESNO = 'yesno'
Displays two buttons whose symbolic names are *note YES: 3ea0. and
*note NO: 3ea1.
-- Data: tkinter.messagebox.YESNOCANCEL = 'yesnocancel'
Displays three buttons whose symbolic names are *note YES: 3ea0,
*note NO: 3ea1. and *note CANCEL: 3e9f.
Icon images:
-- Data: tkinter.messagebox.ERROR = 'error'
-- Data: tkinter.messagebox.INFO = 'info'
-- Data: tkinter.messagebox.QUESTION = 'question'
-- Data: tkinter.messagebox.WARNING = 'warning'
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/tkinter/messagebox.py
File: python.info, Node: tkinter scrolledtext — Scrolled Text Widget, Next: tkinter dnd — Drag and drop support, Prev: tkinter messagebox — Tkinter message prompts, Up: Graphical User Interfaces with Tk
5.26.6 ‘tkinter.scrolledtext’ — Scrolled Text Widget
----------------------------------------------------
'Source code:' Lib/tkinter/scrolledtext.py(1)
__________________________________________________________________
The *note tkinter.scrolledtext: f7. module provides a class of the same
name which implements a basic text widget which has a vertical scroll
bar configured to do the “right thing.” Using the *note ScrolledText:
3ebd. class is a lot easier than setting up a text widget and scroll bar
directly.
The text widget and scrollbar are packed together in a ‘Frame’, and the
methods of the ‘Grid’ and ‘Pack’ geometry managers are acquired from the
‘Frame’ object. This allows the *note ScrolledText: 3ebd. widget to be
used directly to achieve most normal geometry management behavior.
Should more specific control be necessary, the following attributes are
available:
-- Class: tkinter.scrolledtext.ScrolledText (master=None, **kw)
-- Attribute: frame
The frame which surrounds the text and scroll bar widgets.
-- Attribute: vbar
The scroll bar widget.
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/tkinter/scrolledtext.py
File: python.info, Node: tkinter dnd — Drag and drop support, Next: tkinter ttk — Tk themed widgets, Prev: tkinter scrolledtext — Scrolled Text Widget, Up: Graphical User Interfaces with Tk
5.26.7 ‘tkinter.dnd’ — Drag and drop support
--------------------------------------------
'Source code:' Lib/tkinter/dnd.py(1)
__________________________________________________________________
Note: This is experimental and due to be deprecated when it is
replaced with the Tk DND.
The *note tkinter.dnd: f3. module provides drag-and-drop support for
objects within a single application, within the same window or between
windows. To enable an object to be dragged, you must create an event
binding for it that starts the drag-and-drop process. Typically, you
bind a ButtonPress event to a callback function that you write (see
*note Bindings and Events: 3e56.). The function should call *note
dnd_start(): 3ec2, where ‘source’ is the object to be dragged, and
‘event’ is the event that invoked the call (the argument to your
callback function).
Selection of a target object occurs as follows:
1. Top-down search of area under mouse for target widget
* Target widget should have a callable 'dnd_accept' attribute
* If 'dnd_accept' is not present or returns ‘None’, search moves
to parent widget
* If no target widget is found, then the target object is ‘None’
2. Call to '.dnd_leave(source, event)'
3. Call to '.dnd_enter(source, event)'
4. Call to '.dnd_commit(source, event)' to notify of drop
5. Call to '.dnd_end(target, event)' to signal end of
drag-and-drop
-- Class: tkinter.dnd.DndHandler (source, event)
The 'DndHandler' class handles drag-and-drop events tracking Motion
and ButtonRelease events on the root of the event widget.
-- Method: cancel (event=None)
Cancel the drag-and-drop process.
-- Method: finish (event, commit=0)
Execute end of drag-and-drop functions.
-- Method: on_motion (event)
Inspect area below mouse for target objects while drag is
performed.
-- Method: on_release (event)
Signal end of drag when the release pattern is triggered.
-- Function: tkinter.dnd.dnd_start (source, event)
Factory function for drag-and-drop process.
See also
........
*note Bindings and Events: 3e56.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/tkinter/dnd.py
File: python.info, Node: tkinter ttk — Tk themed widgets, Next: IDLE — Python editor and shell<2>, Prev: tkinter dnd — Drag and drop support, Up: Graphical User Interfaces with Tk
5.26.8 ‘tkinter.ttk’ — Tk themed widgets
----------------------------------------
'Source code:' Lib/tkinter/ttk.py(1)
__________________________________________________________________
The *note tkinter.ttk: f9. module provides access to the Tk themed
widget set, introduced in Tk 8.5. It provides additional benefits
including anti-aliased font rendering under X11 and window transparency
(requiring a composition window manager on X11).
The basic idea for *note tkinter.ttk: f9. is to separate, to the extent
possible, the code implementing a widget’s behavior from the code
implementing its appearance.
See also
........
Tk Widget Styling Support(2)
A document introducing theming support for Tk
* Menu:
* Using Ttk::
* Ttk Widgets::
* Widget::
* Combobox::
* Spinbox::
* Notebook::
* Progressbar::
* Separator::
* Sizegrip::
* Treeview::
* Ttk Styling::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/tkinter/ttk.py
(2) https://core.tcl.tk/tips/doc/trunk/tip/48.md
File: python.info, Node: Using Ttk, Next: Ttk Widgets, Up: tkinter ttk — Tk themed widgets
5.26.8.1 Using Ttk
..................
To start using Ttk, import its module:
from tkinter import ttk
To override the basic Tk widgets, the import should follow the Tk
import:
from tkinter import *
from tkinter.ttk import *
That code causes several *note tkinter.ttk: f9. widgets (‘Button’,
‘Checkbutton’, ‘Entry’, ‘Frame’, ‘Label’, ‘LabelFrame’, ‘Menubutton’,
‘PanedWindow’, ‘Radiobutton’, ‘Scale’ and ‘Scrollbar’) to automatically
replace the Tk widgets.
This has the direct benefit of using the new widgets which gives a
better look and feel across platforms; however, the replacement widgets
are not completely compatible. The main difference is that widget
options such as “fg”, “bg” and others related to widget styling are no
longer present in Ttk widgets. Instead, use the ‘ttk.Style’ class for
improved styling effects.
See also
........
Converting existing applications to use Tile widgets(1)
A monograph (using Tcl terminology) about differences typically
encountered when moving applications to use the new widgets.
---------- Footnotes ----------
(1) https://tktable.sourceforge.net/tile/doc/converting.txt
File: python.info, Node: Ttk Widgets, Next: Widget, Prev: Using Ttk, Up: tkinter ttk — Tk themed widgets
5.26.8.2 Ttk Widgets
....................
Ttk comes with 18 widgets, twelve of which already existed in tkinter:
‘Button’, ‘Checkbutton’, ‘Entry’, ‘Frame’, ‘Label’, ‘LabelFrame’,
‘Menubutton’, ‘PanedWindow’, ‘Radiobutton’, ‘Scale’, ‘Scrollbar’, and
*note Spinbox: b94. The other six are new: *note Combobox: 3ecc, *note
Notebook: 3ecd, *note Progressbar: 3ece, ‘Separator’, ‘Sizegrip’ and
*note Treeview: a94. And all them are subclasses of *note Widget: 3ecf.
Using the Ttk widgets gives the application an improved look and feel.
As discussed above, there are differences in how the styling is coded.
Tk code:
l1 = tkinter.Label(text="Test", fg="black", bg="white")
l2 = tkinter.Label(text="Test", fg="black", bg="white")
Ttk code:
style = ttk.Style()
style.configure("BW.TLabel", foreground="black", background="white")
l1 = ttk.Label(text="Test", style="BW.TLabel")
l2 = ttk.Label(text="Test", style="BW.TLabel")
For more information about *note TtkStyling: 3ed0, see the *note Style:
25b. class documentation.
File: python.info, Node: Widget, Next: Combobox, Prev: Ttk Widgets, Up: tkinter ttk — Tk themed widgets
5.26.8.3 Widget
...............
‘ttk.Widget’ defines standard options and methods supported by Tk themed
widgets and is not supposed to be directly instantiated.
* Menu:
* Standard Options::
* Scrollable Widget Options::
* Label Options::
* Compatibility Options::
* Widget States::
* ttk.Widget: ttk Widget.
File: python.info, Node: Standard Options, Next: Scrollable Widget Options, Up: Widget
5.26.8.4 Standard Options
.........................
All the ‘ttk’ Widgets accept the following options:
Option Description
-----------------------------------------------------------------------------------
class Specifies the window class. The class is used when querying the
option database for the window’s other options, to determine the
default bindtags for the window, and to select the widget’s
default layout and style. This option is read-only, and may
only be specified when the window is created.
cursor Specifies the mouse cursor to be used for the widget. If set to
the empty string (the default), the cursor is inherited for the
parent widget.
takefocus Determines whether the window accepts the focus during keyboard
traversal. 0, 1 or an empty string is returned. If 0 is
returned, it means that the window should be skipped entirely
during keyboard traversal. If 1, it means that the window
should receive the input focus as long as it is viewable. And
an empty string means that the traversal scripts make the
decision about whether or not to focus on the window.
style May be used to specify a custom widget style.
File: python.info, Node: Scrollable Widget Options, Next: Label Options, Prev: Standard Options, Up: Widget
5.26.8.5 Scrollable Widget Options
..................................
The following options are supported by widgets that are controlled by a
scrollbar.
Option Description
-----------------------------------------------------------------------------------
xscrollcommand Used to communicate with horizontal scrollbars.
When the view in the widget’s window change, the widget
will generate a Tcl command based on the scrollcommand.
Usually this option consists of the method
‘Scrollbar.set()’ of some scrollbar. This will cause the
scrollbar to be updated whenever the view in the window
changes.
yscrollcommand Used to communicate with vertical scrollbars. For some
more information, see above.
File: python.info, Node: Label Options, Next: Compatibility Options, Prev: Scrollable Widget Options, Up: Widget
5.26.8.6 Label Options
......................
The following options are supported by labels, buttons and other
button-like widgets.
Option Description
-----------------------------------------------------------------------------------
text Specifies a text string to be displayed inside the widget.
textvariable Specifies a name whose value will be used in place of the
text option resource.
underline If set, specifies the index (0-based) of a character to
underline in the text string. The underline character is
used for mnemonic activation.
image Specifies an image to display. This is a list of 1 or more
elements. The first element is the default image name. The
rest of the list if a sequence of statespec/value pairs as
defined by *note Style.map(): 191d, specifying different
images to use when the widget is in a particular state or a
combination of states. All images in the list should have
the same size.
compound Specifies how to display the image relative to the text, in
the case both text and images options are present. Valid
values are:
* text: display text only
* image: display image only
* top, bottom, left, right: display image above, below,
left of, or right of the text, respectively.
* none: the default. display the image if present,
otherwise the text.
width If greater than zero, specifies how much space, in character
widths, to allocate for the text label, if less than zero,
specifies a minimum width. If zero or unspecified, the
natural width of the text label is used.
File: python.info, Node: Compatibility Options, Next: Widget States, Prev: Label Options, Up: Widget
5.26.8.7 Compatibility Options
..............................
Option Description
----------------------------------------------------------------------------------
state May be set to “normal” or “disabled” to control the “disabled”
state bit. This is a write-only option: setting it changes the
widget state, but the *note Widget.state(): 3ed6. method does not
affect this option.
File: python.info, Node: Widget States, Next: ttk Widget, Prev: Compatibility Options, Up: Widget
5.26.8.8 Widget States
......................
The widget state is a bitmap of independent state flags.
Flag Description
-----------------------------------------------------------------------------------
active The mouse cursor is over the widget and pressing a mouse button
will cause some action to occur
disabled Widget is disabled under program control
focus Widget has keyboard focus
pressed Widget is being pressed
selected “On”, “true”, or “current” for things like Checkbuttons and
radiobuttons
background Windows and Mac have a notion of an “active” or foreground
window. The 'background' state is set for widgets in a
background window, and cleared for those in the foreground
window
readonly Widget should not allow user modification
alternate A widget-specific alternate display format
invalid The widget’s value is invalid
A state specification is a sequence of state names, optionally prefixed
with an exclamation point indicating that the bit is off.
File: python.info, Node: ttk Widget, Prev: Widget States, Up: Widget
5.26.8.9 ttk.Widget
...................
Besides the methods described below, the ‘ttk.Widget’ supports the
methods ‘tkinter.Widget.cget()’ and ‘tkinter.Widget.configure()’.
-- Class: tkinter.ttk.Widget
-- Method: identify (x, y)
Returns the name of the element at position 'x' 'y', or the
empty string if the point does not lie within any element.
'x' and 'y' are pixel coordinates relative to the widget.
-- Method: instate (statespec, callback=None, *args, **kw)
Test the widget’s state. If a callback is not specified,
returns ‘True’ if the widget state matches 'statespec' and
‘False’ otherwise. If callback is specified then it is called
with args if widget state matches 'statespec'.
-- Method: state (statespec=None)
Modify or inquire widget state. If 'statespec' is specified,
sets the widget state according to it and return a new
'statespec' indicating which flags were changed. If
'statespec' is not specified, returns the currently enabled
state flags.
'statespec' will usually be a list or a tuple.
File: python.info, Node: Combobox, Next: Spinbox, Prev: Widget, Up: tkinter ttk — Tk themed widgets
5.26.8.10 Combobox
..................
The ‘ttk.Combobox’ widget combines a text field with a pop-down list of
values. This widget is a subclass of ‘Entry’.
Besides the methods inherited from *note Widget: 3ecf.: ‘Widget.cget()’,
‘Widget.configure()’, *note Widget.identify(): 3ed9, *note
Widget.instate(): 3eda. and *note Widget.state(): 3ed6, and the
following inherited from ‘Entry’: ‘Entry.bbox()’, ‘Entry.delete()’,
‘Entry.icursor()’, ‘Entry.index()’, ‘Entry.insert()’,
‘Entry.selection()’, ‘Entry.xview()’, it has some other methods,
described at ‘ttk.Combobox’.
* Menu:
* Options::
* Virtual events::
* ttk.Combobox: ttk Combobox.
File: python.info, Node: Options, Next: Virtual events, Up: Combobox
5.26.8.11 Options
.................
This widget accepts the following specific options:
Option Description
-----------------------------------------------------------------------------------
exportselection Boolean value. If set, the widget selection is linked to
the Window Manager selection (which can be returned by
invoking Misc.selection_get, for example).
justify Specifies how the text is aligned within the widget. One
of “left”, “center”, or “right”.
height Specifies the height of the pop-down listbox, in rows.
postcommand A script (possibly registered with Misc.register) that is
called immediately before displaying the values. It may
specify which values to display.
state One of “normal”, “readonly”, or “disabled”. In the
“readonly” state, the value may not be edited directly,
and the user can only selection of the values from the
dropdown list. In the “normal” state, the text field is
directly editable. In the “disabled” state, no
interaction is possible.
textvariable Specifies a name whose value is linked to the widget
value. Whenever the value associated with that name
changes, the widget value is updated, and vice versa. See
‘tkinter.StringVar’.
values Specifies the list of values to display in the drop-down
listbox.
width Specifies an integer value indicating the desired width of
the entry window, in average-size characters of the
widget’s font.
File: python.info, Node: Virtual events, Next: ttk Combobox, Prev: Options, Up: Combobox
5.26.8.12 Virtual events
........................
The combobox widgets generates a '<>' virtual event
when the user selects an element from the list of values.
File: python.info, Node: ttk Combobox, Prev: Virtual events, Up: Combobox
5.26.8.13 ttk.Combobox
......................
-- Class: tkinter.ttk.Combobox
-- Method: current (newindex=None)
If 'newindex' is specified, sets the combobox value to the
element position 'newindex'. Otherwise, returns the index of
the current value or -1 if the current value is not in the
values list.
-- Method: get ()
Returns the current value of the combobox.
-- Method: set (value)
Sets the value of the combobox to 'value'.
File: python.info, Node: Spinbox, Next: Notebook, Prev: Combobox, Up: tkinter ttk — Tk themed widgets
5.26.8.14 Spinbox
.................
The ‘ttk.Spinbox’ widget is a ‘ttk.Entry’ enhanced with increment and
decrement arrows. It can be used for numbers or lists of string values.
This widget is a subclass of ‘Entry’.
Besides the methods inherited from *note Widget: 3ecf.: ‘Widget.cget()’,
‘Widget.configure()’, *note Widget.identify(): 3ed9, *note
Widget.instate(): 3eda. and *note Widget.state(): 3ed6, and the
following inherited from ‘Entry’: ‘Entry.bbox()’, ‘Entry.delete()’,
‘Entry.icursor()’, ‘Entry.index()’, ‘Entry.insert()’, ‘Entry.xview()’,
it has some other methods, described at ‘ttk.Spinbox’.
* Menu:
* Options: Options<2>.
* Virtual events: Virtual events<2>.
* ttk.Spinbox: ttk Spinbox.
File: python.info, Node: Options<2>, Next: Virtual events<2>, Up: Spinbox
5.26.8.15 Options
.................
This widget accepts the following specific options:
Option Description
--------------------------------------------------------------------------------------
from Float value. If set, this is the minimum value to which
the decrement button will decrement. Must be spelled as
‘from_’ when used as an argument, since ‘from’ is a
Python keyword.
to Float value. If set, this is the maximum value to which
the increment button will increment.
increment Float value. Specifies the amount which the
increment/decrement buttons change the value. Defaults
to 1.0.
values Sequence of string or float values. If specified, the
increment/decrement buttons will cycle through the items
in this sequence rather than incrementing or
decrementing numbers.
wrap Boolean value. If ‘True’, increment and decrement
buttons will cycle from the ‘to’ value to the ‘from’
value or the ‘from’ value to the ‘to’ value,
respectively.
format String value. This specifies the format of numbers set
by the increment/decrement buttons. It must be in the
form “%W.Pf”, where W is the padded width of the value,
P is the precision, and ‘%’ and ‘f’ are literal.
command Python callable. Will be called with no arguments
whenever either of the increment or decrement buttons
are pressed.
File: python.info, Node: Virtual events<2>, Next: ttk Spinbox, Prev: Options<2>, Up: Spinbox
5.26.8.16 Virtual events
........................
The spinbox widget generates an '<>' virtual event when the
user presses , and a '<>' virtual event when the user
presses .
File: python.info, Node: ttk Spinbox, Prev: Virtual events<2>, Up: Spinbox
5.26.8.17 ttk.Spinbox
.....................
-- Class: tkinter.ttk.Spinbox
-- Method: get ()
Returns the current value of the spinbox.
-- Method: set (value)
Sets the value of the spinbox to 'value'.
File: python.info, Node: Notebook, Next: Progressbar, Prev: Spinbox, Up: tkinter ttk — Tk themed widgets
5.26.8.18 Notebook
..................
Ttk Notebook widget manages a collection of windows and displays a
single one at a time. Each child window is associated with a tab, which
the user may select to change the currently displayed window.
* Menu:
* Options: Options<3>.
* Tab Options::
* Tab Identifiers::
* Virtual Events::
* ttk.Notebook: ttk Notebook.
File: python.info, Node: Options<3>, Next: Tab Options, Up: Notebook
5.26.8.19 Options
.................
This widget accepts the following specific options:
Option Description
-----------------------------------------------------------------------------------
height If present and greater than zero, specifies the desired height of
the pane area (not including internal padding or tabs).
Otherwise, the maximum height of all panes is used.
padding Specifies the amount of extra space to add around the outside of
the notebook. The padding is a list up to four length
specifications left top right bottom. If fewer than four elements
are specified, bottom defaults to top, right defaults to left, and
top defaults to left.
width If present and greater than zero, specified the desired width of
the pane area (not including internal padding). Otherwise, the
maximum width of all panes is used.
File: python.info, Node: Tab Options, Next: Tab Identifiers, Prev: Options<3>, Up: Notebook
5.26.8.20 Tab Options
.....................
There are also specific options for tabs:
Option Description
-----------------------------------------------------------------------------------
state Either “normal”, “disabled” or “hidden”. If “disabled”, then
the tab is not selectable. If “hidden”, then the tab is not
shown.
sticky Specifies how the child window is positioned within the pane
area. Value is a string containing zero or more of the
characters “n”, “s”, “e” or “w”. Each letter refers to a side
(north, south, east or west) that the child window will stick
to, as per the ‘grid()’ geometry manager.
padding Specifies the amount of extra space to add between the notebook
and this pane. Syntax is the same as for the option padding
used by this widget.
text Specifies a text to be displayed in the tab.
image Specifies an image to display in the tab. See the option image
described in *note Widget: 3ecf.
compound Specifies how to display the image relative to the text, in the
case both options text and image are present. See
*note Label Options: 3ed4. for legal values.
underline Specifies the index (0-based) of a character to underline in the
text string. The underlined character is used for mnemonic
activation if *note Notebook.enable_traversal(): 3eeb. is
called.
File: python.info, Node: Tab Identifiers, Next: Virtual Events, Prev: Tab Options, Up: Notebook
5.26.8.21 Tab Identifiers
.........................
The tab_id present in several methods of ‘ttk.Notebook’ may take any of
the following forms:
* An integer between zero and the number of tabs
* The name of a child window
* A positional specification of the form “@x,y”, which identifies the
tab
* The literal string “current”, which identifies the currently
selected tab
* The literal string “end”, which returns the number of tabs (only
valid for *note Notebook.index(): 3eed.)
File: python.info, Node: Virtual Events, Next: ttk Notebook, Prev: Tab Identifiers, Up: Notebook
5.26.8.22 Virtual Events
........................
This widget generates a '<>' virtual event after a
new tab is selected.
File: python.info, Node: ttk Notebook, Prev: Virtual Events, Up: Notebook
5.26.8.23 ttk.Notebook
......................
-- Class: tkinter.ttk.Notebook
-- Method: add (child, **kw)
Adds a new tab to the notebook.
If window is currently managed by the notebook but hidden, it
is restored to its previous position.
See *note Tab Options: 3eea. for the list of available
options.
-- Method: forget (tab_id)
Removes the tab specified by 'tab_id', unmaps and unmanages
the associated window.
-- Method: hide (tab_id)
Hides the tab specified by 'tab_id'.
The tab will not be displayed, but the associated window
remains managed by the notebook and its configuration
remembered. Hidden tabs may be restored with the *note add():
3ef0. command.
-- Method: identify (x, y)
Returns the name of the tab element at position 'x', 'y', or
the empty string if none.
-- Method: index (tab_id)
Returns the numeric index of the tab specified by 'tab_id', or
the total number of tabs if 'tab_id' is the string “end”.
-- Method: insert (pos, child, **kw)
Inserts a pane at the specified position.
'pos' is either the string “end”, an integer index, or the
name of a managed child. If 'child' is already managed by the
notebook, moves it to the specified position.
See *note Tab Options: 3eea. for the list of available
options.
-- Method: select (tab_id=None)
Selects the specified 'tab_id'.
The associated child window will be displayed, and the
previously selected window (if different) is unmapped. If
'tab_id' is omitted, returns the widget name of the currently
selected pane.
-- Method: tab (tab_id, option=None, **kw)
Query or modify the options of the specific 'tab_id'.
If 'kw' is not given, returns a dictionary of the tab option
values. If 'option' is specified, returns the value of that
'option'. Otherwise, sets the options to the corresponding
values.
-- Method: tabs ()
Returns a list of windows managed by the notebook.
-- Method: enable_traversal ()
Enable keyboard traversal for a toplevel window containing
this notebook.
This will extend the bindings for the toplevel window
containing the notebook as follows:
* ‘Control’-‘Tab’: selects the tab following the currently
selected one.
* ‘Shift’-‘Control’-‘Tab’: selects the tab preceding the
currently selected one.
* ‘Alt’-‘K’: where 'K' is the mnemonic (underlined)
character of any tab, will select that tab.
Multiple notebooks in a single toplevel may be enabled for
traversal, including nested notebooks. However, notebook
traversal only works properly if all panes have the notebook
they are in as master.
File: python.info, Node: Progressbar, Next: Separator, Prev: Notebook, Up: tkinter ttk — Tk themed widgets
5.26.8.24 Progressbar
.....................
The ‘ttk.Progressbar’ widget shows the status of a long-running
operation. It can operate in two modes: 1) the determinate mode which
shows the amount completed relative to the total amount of work to be
done and 2) the indeterminate mode which provides an animated display to
let the user know that work is progressing.
* Menu:
* Options: Options<4>.
* ttk.Progressbar: ttk Progressbar.
File: python.info, Node: Options<4>, Next: ttk Progressbar, Up: Progressbar
5.26.8.25 Options
.................
This widget accepts the following specific options:
Option Description
-----------------------------------------------------------------------------------
orient One of “horizontal” or “vertical”. Specifies the orientation of
the progress bar.
length Specifies the length of the long axis of the progress bar (width
if horizontal, height if vertical).
mode One of “determinate” or “indeterminate”.
maximum A number specifying the maximum value. Defaults to 100.
value The current value of the progress bar. In “determinate” mode,
this represents the amount of work completed. In “indeterminate”
mode, it is interpreted as modulo 'maximum'; that is, the
progress bar completes one “cycle” when its value increases by
'maximum'.
variable A name which is linked to the option value. If specified, the
value of the progress bar is automatically set to the value of
this name whenever the latter is modified.
phase Read-only option. The widget periodically increments the value
of this option whenever its value is greater than 0 and, in
determinate mode, less than maximum. This option may be used by
the current theme to provide additional animation effects.
File: python.info, Node: ttk Progressbar, Prev: Options<4>, Up: Progressbar
5.26.8.26 ttk.Progressbar
.........................
-- Class: tkinter.ttk.Progressbar
-- Method: start (interval=None)
Begin autoincrement mode: schedules a recurring timer event
that calls *note Progressbar.step(): 3efc. every 'interval'
milliseconds. If omitted, 'interval' defaults to 50
milliseconds.
-- Method: step (amount=None)
Increments the progress bar’s value by 'amount'.
'amount' defaults to 1.0 if omitted.
-- Method: stop ()
Stop autoincrement mode: cancels any recurring timer event
initiated by *note Progressbar.start(): 3efb. for this
progress bar.
File: python.info, Node: Separator, Next: Sizegrip, Prev: Progressbar, Up: tkinter ttk — Tk themed widgets
5.26.8.27 Separator
...................
The ‘ttk.Separator’ widget displays a horizontal or vertical separator
bar.
It has no other methods besides the ones inherited from ‘ttk.Widget’.
* Menu:
* Options: Options<5>.
File: python.info, Node: Options<5>, Up: Separator
5.26.8.28 Options
.................
This widget accepts the following specific option:
Option Description
----------------------------------------------------------------------------------
orient One of “horizontal” or “vertical”. Specifies the orientation of
the separator.
File: python.info, Node: Sizegrip, Next: Treeview, Prev: Separator, Up: tkinter ttk — Tk themed widgets
5.26.8.29 Sizegrip
..................
The ‘ttk.Sizegrip’ widget (also known as a grow box) allows the user to
resize the containing toplevel window by pressing and dragging the grip.
This widget has neither specific options nor specific methods, besides
the ones inherited from ‘ttk.Widget’.
* Menu:
* Platform-specific notes::
* Bugs::
File: python.info, Node: Platform-specific notes, Next: Bugs, Up: Sizegrip
5.26.8.30 Platform-specific notes
.................................
* On macOS, toplevel windows automatically include a built-in size
grip by default. Adding a ‘Sizegrip’ is harmless, since the
built-in grip will just mask the widget.
File: python.info, Node: Bugs, Prev: Platform-specific notes, Up: Sizegrip
5.26.8.31 Bugs
..............
* If the containing toplevel’s position was specified relative to the
right or bottom of the screen (e.g. ….), the ‘Sizegrip’ widget
will not resize the window.
* This widget supports only “southeast” resizing.
File: python.info, Node: Treeview, Next: Ttk Styling, Prev: Sizegrip, Up: tkinter ttk — Tk themed widgets
5.26.8.32 Treeview
..................
The ‘ttk.Treeview’ widget displays a hierarchical collection of items.
Each item has a textual label, an optional image, and an optional list
of data values. The data values are displayed in successive columns
after the tree label.
The order in which data values are displayed may be controlled by
setting the widget option ‘displaycolumns’. The tree widget can also
display column headings. Columns may be accessed by number or symbolic
names listed in the widget option columns. See *note Column
Identifiers: 3f04.
Each item is identified by a unique name. The widget will generate item
IDs if they are not supplied by the caller. There is a distinguished
root item, named ‘{}’. The root item itself is not displayed; its
children appear at the top level of the hierarchy.
Each item also has a list of tags, which can be used to associate event
bindings with individual items and control the appearance of the item.
The Treeview widget supports horizontal and vertical scrolling,
according to the options described in *note Scrollable Widget Options:
3ed3. and the methods *note Treeview.xview(): 3f05. and *note
Treeview.yview(): 3f06.
* Menu:
* Options: Options<6>.
* Item Options::
* Tag Options::
* Column Identifiers::
* Virtual Events: Virtual Events<2>.
* ttk.Treeview: ttk Treeview.
File: python.info, Node: Options<6>, Next: Item Options, Up: Treeview
5.26.8.33 Options
.................
This widget accepts the following specific options:
Option Description
----------------------------------------------------------------------------------
columns A list of column identifiers, specifying the number of
columns and their names.
displaycolumns A list of column identifiers (either symbolic or integer
indices) specifying which data columns are displayed and
the order in which they appear, or the string “#all”.
height Specifies the number of rows which should be visible.
Note: the requested width is determined from the sum of
the column widths.
padding Specifies the internal padding for the widget. The
padding is a list of up to four length specifications.
selectmode Controls how the built-in class bindings manage the
selection. One of “extended”, “browse” or “none”. If set
to “extended” (the default), multiple items may be
selected. If “browse”, only a single item will be
selected at a time. If “none”, the selection will not be
changed.
Note that the application code and tag bindings can set
the selection however they wish, regardless of the value
of this option.
show A list containing zero or more of the following values,
specifying which elements of the tree to display.
* tree: display tree labels in column #0.
* headings: display the heading row.
The default is “tree headings”, i.e., show all elements.
'Note': Column #0 always refers to the tree column, even
if show=”tree” is not specified.
File: python.info, Node: Item Options, Next: Tag Options, Prev: Options<6>, Up: Treeview
5.26.8.34 Item Options
......................
The following item options may be specified for items in the insert and
item widget commands.
Option Description
---------------------------------------------------------------------------------
text The textual label to display for the item.
image A Tk Image, displayed to the left of the label.
values The list of values associated with the item.
Each item should have the same number of values as the widget
option columns. If there are fewer values than columns, the
remaining values are assumed empty. If there are more values
than columns, the extra values are ignored.
open ‘True’/‘False’ value indicating whether the item’s children
should be displayed or hidden.
tags A list of tags associated with this item.
File: python.info, Node: Tag Options, Next: Column Identifiers, Prev: Item Options, Up: Treeview
5.26.8.35 Tag Options
.....................
The following options may be specified on tags:
Option Description
---------------------------------------------------------------------------------
foreground Specifies the text foreground color.
background Specifies the cell or item background color.
font Specifies the font to use when drawing text.
image Specifies the item image, in case the item’s image option is
empty.
File: python.info, Node: Column Identifiers, Next: Virtual Events<2>, Prev: Tag Options, Up: Treeview
5.26.8.36 Column Identifiers
............................
Column identifiers take any of the following forms:
* A symbolic name from the list of columns option.
* An integer n, specifying the nth data column.
* A string of the form #n, where n is an integer, specifying the nth
display column.
Notes:
* Item’s option values may be displayed in a different order than the
order in which they are stored.
* Column #0 always refers to the tree column, even if show=”tree” is
not specified.
A data column number is an index into an item’s option values list; a
display column number is the column number in the tree where the values
are displayed. Tree labels are displayed in column #0. If option
displaycolumns is not set, then data column n is displayed in column
#n+1. Again, 'column #0 always refers to the tree column'.
File: python.info, Node: Virtual Events<2>, Next: ttk Treeview, Prev: Column Identifiers, Up: Treeview
5.26.8.37 Virtual Events
........................
The Treeview widget generates the following virtual events.
Event Description
--------------------------------------------------------------------------------
<> Generated whenever the selection changes.
<> Generated just before settings the focus item to
open=True.
<> Generated just after setting the focus item to
open=False.
The *note Treeview.focus(): 3f0b. and *note Treeview.selection(): a93.
methods can be used to determine the affected item or items.
File: python.info, Node: ttk Treeview, Prev: Virtual Events<2>, Up: Treeview
5.26.8.38 ttk.Treeview
......................
-- Class: tkinter.ttk.Treeview
-- Method: bbox (item, column=None)
Returns the bounding box (relative to the treeview widget’s
window) of the specified 'item' in the form (x, y, width,
height).
If 'column' is specified, returns the bounding box of that
cell. If the 'item' is not visible (i.e., if it is a
descendant of a closed item or is scrolled offscreen), returns
an empty string.
-- Method: get_children (item=None)
Returns the list of children belonging to 'item'.
If 'item' is not specified, returns root children.
-- Method: set_children (item, *newchildren)
Replaces 'item'’s child with 'newchildren'.
Children present in 'item' that are not present in
'newchildren' are detached from the tree. No items in
'newchildren' may be an ancestor of 'item'. Note that not
specifying 'newchildren' results in detaching 'item'’s
children.
-- Method: column (column, option=None, **kw)
Query or modify the options for the specified 'column'.
If 'kw' is not given, returns a dict of the column option
values. If 'option' is specified then the value for that
'option' is returned. Otherwise, sets the options to the
corresponding values.
The valid options/values are:
'id'
Returns the column name. This is a read-only option.
'anchor': One of the standard Tk anchor values.
Specifies how the text in this column should be aligned
with respect to the cell.
'minwidth': width
The minimum width of the column in pixels. The treeview
widget will not make the column any smaller than
specified by this option when the widget is resized or
the user drags a column.
'stretch': ‘True’/‘False’
Specifies whether the column’s width should be adjusted
when the widget is resized.
'width': width
The width of the column in pixels.
To configure the tree column, call this with column = “#0”
-- Method: delete (*items)
Delete all specified 'items' and all their descendants.
The root item may not be deleted.
-- Method: detach (*items)
Unlinks all of the specified 'items' from the tree.
The items and all of their descendants are still present, and
may be reinserted at another point in the tree, but will not
be displayed.
The root item may not be detached.
-- Method: exists (item)
Returns ‘True’ if the specified 'item' is present in the tree.
-- Method: focus (item=None)
If 'item' is specified, sets the focus item to 'item'.
Otherwise, returns the current focus item, or ‘’ if there is
none.
-- Method: heading (column, option=None, **kw)
Query or modify the heading options for the specified
'column'.
If 'kw' is not given, returns a dict of the heading option
values. If 'option' is specified then the value for that
'option' is returned. Otherwise, sets the options to the
corresponding values.
The valid options/values are:
'text': text
The text to display in the column heading.
'image': imageName
Specifies an image to display to the right of the column
heading.
'anchor': anchor
Specifies how the heading text should be aligned. One of
the standard Tk anchor values.
'command': callback
A callback to be invoked when the heading label is
pressed.
To configure the tree column heading, call this with column =
“#0”.
-- Method: identify (component, x, y)
Returns a description of the specified 'component' under the
point given by 'x' and 'y', or the empty string if no such
'component' is present at that position.
-- Method: identify_row (y)
Returns the item ID of the item at position 'y'.
-- Method: identify_column (x)
Returns the data column identifier of the cell at position
'x'.
The tree column has ID #0.
-- Method: identify_region (x, y)
Returns one of:
region meaning
-----------------------------------------------------------
heading Tree heading area.
separator Space between two columns headings.
tree The tree area.
cell A data cell.
Availability: Tk 8.6.
-- Method: identify_element (x, y)
Returns the element at position 'x', 'y'.
Availability: Tk 8.6.
-- Method: index (item)
Returns the integer index of 'item' within its parent’s list
of children.
-- Method: insert (parent, index, iid=None, **kw)
Creates a new item and returns the item identifier of the
newly created item.
'parent' is the item ID of the parent item, or the empty
string to create a new top-level item. 'index' is an integer,
or the value “end”, specifying where in the list of parent’s
children to insert the new item. If 'index' is less than or
equal to zero, the new node is inserted at the beginning; if
'index' is greater than or equal to the current number of
children, it is inserted at the end. If 'iid' is specified,
it is used as the item identifier; 'iid' must not already
exist in the tree. Otherwise, a new unique identifier is
generated.
See *note Item Options: 3f08. for the list of available
options.
-- Method: item (item, option=None, **kw)
Query or modify the options for the specified 'item'.
If no options are given, a dict with options/values for the
item is returned. If 'option' is specified then the value for
that option is returned. Otherwise, sets the options to the
corresponding values as given by 'kw'.
-- Method: move (item, parent, index)
Moves 'item' to position 'index' in 'parent'’s list of
children.
It is illegal to move an item under one of its descendants.
If 'index' is less than or equal to zero, 'item' is moved to
the beginning; if greater than or equal to the number of
children, it is moved to the end. If 'item' was detached it
is reattached.
-- Method: next (item)
Returns the identifier of 'item'’s next sibling, or ‘’ if
'item' is the last child of its parent.
-- Method: parent (item)
Returns the ID of the parent of 'item', or ‘’ if 'item' is at
the top level of the hierarchy.
-- Method: prev (item)
Returns the identifier of 'item'’s previous sibling, or ‘’ if
'item' is the first child of its parent.
-- Method: reattach (item, parent, index)
An alias for *note Treeview.move(): 3f1d.
-- Method: see (item)
Ensure that 'item' is visible.
Sets all of 'item'’s ancestors open option to ‘True’, and
scrolls the widget if necessary so that 'item' is within the
visible portion of the tree.
-- Method: selection ()
Returns a tuple of selected items.
Changed in version 3.8: ‘selection()’ no longer takes
arguments. For changing the selection state use the following
selection methods.
-- Method: selection_set (*items)
'items' becomes the new selection.
Changed in version 3.6: 'items' can be passed as separate
arguments, not just as a single tuple.
-- Method: selection_add (*items)
Add 'items' to the selection.
Changed in version 3.6: 'items' can be passed as separate
arguments, not just as a single tuple.
-- Method: selection_remove (*items)
Remove 'items' from the selection.
Changed in version 3.6: 'items' can be passed as separate
arguments, not just as a single tuple.
-- Method: selection_toggle (*items)
Toggle the selection state of each item in 'items'.
Changed in version 3.6: 'items' can be passed as separate
arguments, not just as a single tuple.
-- Method: set (item, column=None, value=None)
With one argument, returns a dictionary of column/value pairs
for the specified 'item'. With two arguments, returns the
current value of the specified 'column'. With three
arguments, sets the value of given 'column' in given 'item' to
the specified 'value'.
-- Method: tag_bind (tagname, sequence=None, callback=None)
Bind a callback for the given event 'sequence' to the tag
'tagname'. When an event is delivered to an item, the
callbacks for each of the item’s tags option are called.
-- Method: tag_configure (tagname, option=None, **kw)
Query or modify the options for the specified 'tagname'.
If 'kw' is not given, returns a dict of the option settings
for 'tagname'. If 'option' is specified, returns the value
for that 'option' for the specified 'tagname'. Otherwise,
sets the options to the corresponding values for the given
'tagname'.
-- Method: tag_has (tagname, item=None)
If 'item' is specified, returns 1 or 0 depending on whether
the specified 'item' has the given 'tagname'. Otherwise,
returns a list of all items that have the specified tag.
Availability: Tk 8.6
-- Method: xview (*args)
Query or modify horizontal position of the treeview.
-- Method: yview (*args)
Query or modify vertical position of the treeview.
File: python.info, Node: Ttk Styling, Prev: Treeview, Up: tkinter ttk — Tk themed widgets
5.26.8.39 Ttk Styling
.....................
Each widget in ‘ttk’ is assigned a style, which specifies the set of
elements making up the widget and how they are arranged, along with
dynamic and default settings for element options. By default the style
name is the same as the widget’s class name, but it may be overridden by
the widget’s style option. If you don’t know the class name of a
widget, use the method ‘Misc.winfo_class()’ (somewidget.winfo_class()).
See also
........
Tcl'2004 conference presentation(1)
This document explains how the theme engine works
-- Class: tkinter.ttk.Style
This class is used to manipulate the style database.
-- Method: configure (style, query_opt=None, **kw)
Query or set the default value of the specified option(s) in
'style'.
Each key in 'kw' is an option and each value is a string
identifying the value for that option.
For example, to change every default button to be a flat
button with some padding and a different background color:
from tkinter import ttk
import tkinter
root = tkinter.Tk()
ttk.Style().configure("TButton", padding=6, relief="flat",
background="#ccc")
btn = ttk.Button(text="Sample")
btn.pack()
root.mainloop()
-- Method: map (style, query_opt=None, **kw)
Query or sets dynamic values of the specified option(s) in
'style'.
Each key in 'kw' is an option and each value should be a list
or a tuple (usually) containing statespecs grouped in tuples,
lists, or some other preference. A statespec is a compound of
one or more states and then a value.
An example may make it more understandable:
import tkinter
from tkinter import ttk
root = tkinter.Tk()
style = ttk.Style()
style.map("C.TButton",
foreground=[('pressed', 'red'), ('active', 'blue')],
background=[('pressed', '!disabled', 'black'), ('active', 'white')]
)
colored_btn = ttk.Button(text="Test", style="C.TButton").pack()
root.mainloop()
Note that the order of the (states, value) sequences for an
option does matter, if the order is changed to ‘[('active',
'blue'), ('pressed', 'red')]’ in the foreground option, for
example, the result would be a blue foreground when the widget
were in active or pressed states.
-- Method: lookup (style, option, state=None, default=None)
Returns the value specified for 'option' in 'style'.
If 'state' is specified, it is expected to be a sequence of
one or more states. If the 'default' argument is set, it is
used as a fallback value in case no specification for option
is found.
To check what font a Button uses by default:
from tkinter import ttk
print(ttk.Style().lookup("TButton", "font"))
-- Method: layout (style, layoutspec=None)
Define the widget layout for given 'style'. If 'layoutspec'
is omitted, return the layout specification for given style.
'layoutspec', if specified, is expected to be a list or some
other sequence type (excluding strings), where each item
should be a tuple and the first item is the layout name and
the second item should have the format described in *note
Layouts: 3f2e.
To understand the format, see the following example (it is not
intended to do anything useful):
from tkinter import ttk
import tkinter
root = tkinter.Tk()
style = ttk.Style()
style.layout("TMenubutton", [
("Menubutton.background", None),
("Menubutton.button", {"children":
[("Menubutton.focus", {"children":
[("Menubutton.padding", {"children":
[("Menubutton.label", {"side": "left", "expand": 1})]
})]
})]
}),
])
mbtn = ttk.Menubutton(text='Text')
mbtn.pack()
root.mainloop()
-- Method: element_create (elementname, etype, *args, **kw)
Create a new element in the current theme, of the given
'etype' which is expected to be either “image”, “from” or
“vsapi”. The latter is only available in Tk 8.6 on Windows.
If “image” is used, 'args' should contain the default image
name followed by statespec/value pairs (this is the
imagespec), and 'kw' may have the following options:
border=padding
padding is a list of up to four integers, specifying the
left, top, right, and bottom borders, respectively.
height=height
Specifies a minimum height for the element. If less than
zero, the base image’s height is used as a default.
padding=padding
Specifies the element’s interior padding. Defaults to
border’s value if not specified.
sticky=spec
Specifies how the image is placed within the final
parcel. spec contains zero or more characters “n”, “s”,
“w”, or “e”.
width=width
Specifies a minimum width for the element. If less than
zero, the base image’s width is used as a default.
Example:
img1 = tkinter.PhotoImage(master=root, file='button.png')
img1 = tkinter.PhotoImage(master=root, file='button-pressed.png')
img1 = tkinter.PhotoImage(master=root, file='button-active.png')
style = ttk.Style(root)
style.element_create('Button.button', 'image',
img1, ('pressed', img2), ('active', img3),
border=(2, 4), sticky='we')
If “from” is used as the value of 'etype', *note
element_create(): 25a. will clone an existing element. 'args'
is expected to contain a themename, from which the element
will be cloned, and optionally an element to clone from. If
this element to clone from is not specified, an empty element
will be used. 'kw' is discarded.
Example:
style = ttk.Style(root)
style.element_create('plain.background', 'from', 'default')
If “vsapi” is used as the value of 'etype', *note
element_create(): 25a. will create a new element in the
current theme whose visual appearance is drawn using the
Microsoft Visual Styles API which is responsible for the
themed styles on Windows XP and Vista. 'args' is expected to
contain the Visual Styles class and part as given in the
Microsoft documentation followed by an optional sequence of
tuples of ttk states and the corresponding Visual Styles API
state value. 'kw' may have the following options:
padding=padding
Specify the element’s interior padding. 'padding' is a
list of up to four integers specifying the left, top,
right and bottom padding quantities respectively. If
fewer than four elements are specified, bottom defaults
to top, right defaults to left, and top defaults to left.
In other words, a list of three numbers specify the left,
vertical, and right padding; a list of two numbers
specify the horizontal and the vertical padding; a single
number specifies the same padding all the way around the
widget. This option may not be mixed with any other
options.
margins=padding
Specifies the elements exterior padding. 'padding' is a
list of up to four integers specifying the left, top,
right and bottom padding quantities respectively. This
option may not be mixed with any other options.
width=width
Specifies the width for the element. If this option is
set then the Visual Styles API will not be queried for
the recommended size or the part. If this option is set
then 'height' should also be set. The 'width' and
'height' options cannot be mixed with the 'padding' or
'margins' options.
height=height
Specifies the height of the element. See the comments
for 'width'.
Example:
style = ttk.Style(root)
style.element_create('pin', 'vsapi', 'EXPLORERBAR', 3, [
('pressed', '!selected', 3),
('active', '!selected', 2),
('pressed', 'selected', 6),
('active', 'selected', 5),
('selected', 4),
('', 1)])
style.layout('Explorer.Pin',
[('Explorer.Pin.pin', {'sticky': 'news'})])
pin = ttk.Checkbutton(style='Explorer.Pin')
pin.pack(expand=True, fill='both')
Changed in version 3.13: Added support of the “vsapi” element
factory.
-- Method: element_names ()
Returns the list of elements defined in the current theme.
-- Method: element_options (elementname)
Returns the list of 'elementname'’s options.
-- Method: theme_create (themename, parent=None, settings=None)
Create a new theme.
It is an error if 'themename' already exists. If 'parent' is
specified, the new theme will inherit styles, elements and
layouts from the parent theme. If 'settings' are present they
are expected to have the same syntax used for *note
theme_settings(): 3f32.
-- Method: theme_settings (themename, settings)
Temporarily sets the current theme to 'themename', apply
specified 'settings' and then restore the previous theme.
Each key in 'settings' is a style and each value may contain
the keys ‘configure’, ‘map’, ‘layout’ and ‘element create’ and
they are expected to have the same format as specified by the
methods *note Style.configure(): 3f2b, *note Style.map():
191d, *note Style.layout(): 3f2d. and *note
Style.element_create(): 25a. respectively.
As an example, let’s change the Combobox for the default theme
a bit:
from tkinter import ttk
import tkinter
root = tkinter.Tk()
style = ttk.Style()
style.theme_settings("default", {
"TCombobox": {
"configure": {"padding": 5},
"map": {
"background": [("active", "green2"),
("!disabled", "green4")],
"fieldbackground": [("!disabled", "green3")],
"foreground": [("focus", "OliveDrab1"),
("!disabled", "OliveDrab2")]
}
}
})
combo = ttk.Combobox().pack()
root.mainloop()
-- Method: theme_names ()
Returns a list of all known themes.
-- Method: theme_use (themename=None)
If 'themename' is not given, returns the theme in use.
Otherwise, sets the current theme to 'themename', refreshes
all widgets and emits a <> event.
* Menu:
* Layouts::
---------- Footnotes ----------
(1) https://tktable.sourceforge.net/tile/tile-tcl2004.pdf
File: python.info, Node: Layouts, Up: Ttk Styling
5.26.8.40 Layouts
.................
A layout can be just ‘None’, if it takes no options, or a dict of
options specifying how to arrange the element. The layout mechanism
uses a simplified version of the pack geometry manager: given an initial
cavity, each element is allocated a parcel.
The valid options/values are:
'side': whichside
Specifies which side of the cavity to place the element; one of
top, right, bottom or left. If omitted, the element occupies the
entire cavity.
'sticky': nswe
Specifies where the element is placed inside its allocated parcel.
'unit': 0 or 1
If set to 1, causes the element and all of its descendants to be
treated as a single element for the purposes of *note
Widget.identify(): 3ed9. et al. It’s used for things like
scrollbar thumbs with grips.
'children': [sublayout… ]
Specifies a list of elements to place inside the element. Each
element is a tuple (or other sequence type) where the first item is
the layout name, and the other is a *note Layout: 3f2e.
File: python.info, Node: IDLE — Python editor and shell<2>, Prev: tkinter ttk — Tk themed widgets, Up: Graphical User Interfaces with Tk
5.26.9 IDLE — Python editor and shell
-------------------------------------
'Source code:' Lib/idlelib/(1)
__________________________________________________________________
IDLE is Python’s Integrated Development and Learning Environment.
IDLE has the following features:
* cross-platform: works mostly the same on Windows, Unix, and macOS
* Python shell window (interactive interpreter) with colorizing of
code input, output, and error messages
* multi-window text editor with multiple undo, Python colorizing,
smart indent, call tips, auto completion, and other features
* search within any window, replace within editor windows, and search
through multiple files (grep)
* debugger with persistent breakpoints, stepping, and viewing of
global and local namespaces
* configuration, browsers, and other dialogs
* Menu:
* Menus::
* Editing and Navigation::
* Startup and Code Execution::
* Help and Preferences::
* idlelib — implementation of IDLE application::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/idlelib/
File: python.info, Node: Menus, Next: Editing and Navigation, Up: IDLE — Python editor and shell<2>
5.26.9.1 Menus
..............
IDLE has two main window types, the Shell window and the Editor window.
It is possible to have multiple editor windows simultaneously. On
Windows and Linux, each has its own top menu. Each menu documented
below indicates which window type it is associated with.
Output windows, such as used for Edit => Find in Files, are a subtype of
editor window. They currently have the same top menu but a different
default title and context menu.
On macOS, there is one application menu. It dynamically changes
according to the window currently selected. It has an IDLE menu, and
some entries described below are moved around to conform to Apple
guidelines.
* Menu:
* File menu (Shell and Editor): File menu Shell and Editor.
* Edit menu (Shell and Editor): Edit menu Shell and Editor.
* Format menu (Editor window only): Format menu Editor window only.
* Run menu (Editor window only): Run menu Editor window only.
* Shell menu (Shell window only): Shell menu Shell window only.
* Debug menu (Shell window only): Debug menu Shell window only.
* Options menu (Shell and Editor): Options menu Shell and Editor.
* Window menu (Shell and Editor): Window menu Shell and Editor.
* Help menu (Shell and Editor): Help menu Shell and Editor.
* Context menus::
File: python.info, Node: File menu Shell and Editor, Next: Edit menu Shell and Editor, Up: Menus
5.26.9.2 File menu (Shell and Editor)
.....................................
New File
Create a new file editing window.
Open…
Open an existing file with an Open dialog.
Open Module…
Open an existing module (searches sys.path).
Recent Files
Open a list of recent files. Click one to open it.
Module Browser
Show functions, classes, and methods in the current Editor file in
a tree structure. In the shell, open a module first.
Path Browser
Show sys.path directories, modules, functions, classes and methods
in a tree structure.
Save
Save the current window to the associated file, if there is one.
Windows that have been changed since being opened or last saved
have a * before and after the window title. If there is no
associated file, do Save As instead.
Save As…
Save the current window with a Save As dialog. The file saved
becomes the new associated file for the window. (If your file
namager is set to hide extensions, the current extension will be
omitted in the file name box. If the new filename has no ‘.’,
‘.py’ and ‘.txt’ will be added for Python and text files, except
that on macOS Aqua,’.py’ is added for all files.)
Save Copy As…
Save the current window to different file without changing the
associated file. (See Save As note above about filename
extensions.)
Print Window
Print the current window to the default printer.
Close Window
Close the current window (if an unsaved editor, ask to save; if an
unsaved Shell, ask to quit execution). Calling ‘exit()’ or
‘close()’ in the Shell window also closes Shell. If this is the
only window, also exit IDLE.
Exit IDLE
Close all windows and quit IDLE (ask to save unsaved edit windows).
File: python.info, Node: Edit menu Shell and Editor, Next: Format menu Editor window only, Prev: File menu Shell and Editor, Up: Menus
5.26.9.3 Edit menu (Shell and Editor)
.....................................
Undo
Undo the last change to the current window. A maximum of 1000
changes may be undone.
Redo
Redo the last undone change to the current window.
Select All
Select the entire contents of the current window.
Cut
Copy selection into the system-wide clipboard; then delete the
selection.
Copy
Copy selection into the system-wide clipboard.
Paste
Insert contents of the system-wide clipboard into the current
window.
The clipboard functions are also available in context menus.
Find…
Open a search dialog with many options
Find Again
Repeat the last search, if there is one.
Find Selection
Search for the currently selected string, if there is one.
Find in Files…
Open a file search dialog. Put results in a new output window.
Replace…
Open a search-and-replace dialog.
Go to Line
Move the cursor to the beginning of the line requested and make
that line visible. A request past the end of the file goes to the
end. Clear any selection and update the line and column status.
Show Completions
Open a scrollable list allowing selection of existing names. See
*note Completions: 3f3b. in the Editing and navigation section
below.
Expand Word
Expand a prefix you have typed to match a full word in the same
window; repeat to get a different expansion.
Show Call Tip
After an unclosed parenthesis for a function, open a small window
with function parameter hints. See *note Calltips: 3f3c. in the
Editing and navigation section below.
Show Surrounding Parens
Highlight the surrounding parenthesis.
File: python.info, Node: Format menu Editor window only, Next: Run menu Editor window only, Prev: Edit menu Shell and Editor, Up: Menus
5.26.9.4 Format menu (Editor window only)
.........................................
Format Paragraph
Reformat the current blank-line-delimited paragraph in comment
block or multiline string or selected line in a string. All lines
in the paragraph will be formatted to less than N columns, where N
defaults to 72.
Indent Region
Shift selected lines right by the indent width (default 4 spaces).
Dedent Region
Shift selected lines left by the indent width (default 4 spaces).
Comment Out Region
Insert ## in front of selected lines.
Uncomment Region
Remove leading # or ## from selected lines.
Tabify Region
Turn 'leading' stretches of spaces into tabs. (Note: We recommend
using 4 space blocks to indent Python code.)
Untabify Region
Turn 'all' tabs into the correct number of spaces.
Toggle Tabs
Open a dialog to switch between indenting with spaces and tabs.
New Indent Width
Open a dialog to change indent width. The accepted default by the
Python community is 4 spaces.
Strip Trailing Chitespace
Remove trailing space and other whitespace characters after the
last non-whitespace character of a line by applying str.rstrip to
each line, including lines within multiline strings. Except for
Shell windows, remove extra newlines at the end of the file.
File: python.info, Node: Run menu Editor window only, Next: Shell menu Shell window only, Prev: Format menu Editor window only, Up: Menus
5.26.9.5 Run menu (Editor window only)
......................................
Run Module
Do *note Check Module: 3f41. If no error, restart the shell to
clean the environment, then execute the module. Output is
displayed in the Shell window. Note that output requires use of
‘print’ or ‘write’. When execution is complete, the Shell retains
focus and displays a prompt. At this point, one may interactively
explore the result of execution. This is similar to executing a
file with ‘python -i file’ at a command line.
Run… Customized
Same as *note Run Module: 3f40, but run the module with customized
settings. 'Command Line Arguments' extend *note sys.argv: 1258. as
if passed on a command line. The module can be run in the Shell
without restarting.
Check Module
Check the syntax of the module currently open in the Editor window.
If the module has not been saved IDLE will either prompt the user
to save or autosave, as selected in the General tab of the Idle
Settings dialog. If there is a syntax error, the approximate
location is indicated in the Editor window.
Python Shell
Open or wake up the Python Shell window.
File: python.info, Node: Shell menu Shell window only, Next: Debug menu Shell window only, Prev: Run menu Editor window only, Up: Menus
5.26.9.6 Shell menu (Shell window only)
.......................................
View Last Restart
Scroll the shell window to the last Shell restart.
Restart Shell
Restart the shell to clean the environment and reset display and
exception handling.
Previous History
Cycle through earlier commands in history which match the current
entry.
Next History
Cycle through later commands in history which match the current
entry.
Interrupt Execution
Stop a running program.
File: python.info, Node: Debug menu Shell window only, Next: Options menu Shell and Editor, Prev: Shell menu Shell window only, Up: Menus
5.26.9.7 Debug menu (Shell window only)
.......................................
Go to File/Line
Look on the current line. with the cursor, and the line above for
a filename and line number. If found, open the file if not already
open, and show the line. Use this to view source lines referenced
in an exception traceback and lines found by Find in Files. Also
available in the context menu of the Shell window and Output
windows.
Debugger (toggle)
When activated, code entered in the Shell or run from an Editor
will run under the debugger. In the Editor, breakpoints can be set
with the context menu. This feature is still incomplete and
somewhat experimental.
Stack Viewer
Show the stack traceback of the last exception in a tree widget,
with access to locals and globals.
Auto-open Stack Viewer
Toggle automatically opening the stack viewer on an unhandled
exception.
File: python.info, Node: Options menu Shell and Editor, Next: Window menu Shell and Editor, Prev: Debug menu Shell window only, Up: Menus
5.26.9.8 Options menu (Shell and Editor)
........................................
Configure IDLE
Open a configuration dialog and change preferences for the
following: fonts, indentation, keybindings, text color themes,
startup windows and size, additional help sources, and extensions.
On macOS, open the configuration dialog by selecting Preferences in
the application menu. For more details, see *note Setting
preferences: 3f47. under Help and preferences.
Most configuration options apply to all windows or all future windows.
The option items below only apply to the active window.
Show/Hide Code Context (Editor Window only)
Open a pane at the top of the edit window which shows the block
context of the code which has scrolled above the top of the window.
See *note Code Context: 3f48. in the Editing and Navigation section
below.
Show/Hide Line Numbers (Editor Window only)
Open a column to the left of the edit window which shows the number
of each line of text. The default is off, which may be changed in
the preferences (see *note Setting preferences: 3f47.).
Zoom/Restore Height
Toggles the window between normal size and maximum height. The
initial size defaults to 40 lines by 80 chars unless changed on the
General tab of the Configure IDLE dialog. The maximum height for a
screen is determined by momentarily maximizing a window the first
time one is zoomed on the screen. Changing screen settings may
invalidate the saved height. This toggle has no effect when a
window is maximized.
File: python.info, Node: Window menu Shell and Editor, Next: Help menu Shell and Editor, Prev: Options menu Shell and Editor, Up: Menus
5.26.9.9 Window menu (Shell and Editor)
.......................................
Lists the names of all open windows; select one to bring it to the
foreground (deiconifying it if necessary).
File: python.info, Node: Help menu Shell and Editor, Next: Context menus, Prev: Window menu Shell and Editor, Up: Menus
5.26.9.10 Help menu (Shell and Editor)
......................................
About IDLE
Display version, copyright, license, credits, and more.
IDLE Help
Display this IDLE document, detailing the menu options, basic
editing and navigation, and other tips.
Python Docs
Access local Python documentation, if installed, or start a web
browser and open docs.python.org showing the latest Python
documentation.
Turtle Demo
Run the turtledemo module with example Python code and turtle
drawings.
Additional help sources may be added here with the Configure IDLE dialog
under the General tab. See the *note Help sources: 3f4b. subsection
below for more on Help menu choices.
File: python.info, Node: Context menus, Prev: Help menu Shell and Editor, Up: Menus
5.26.9.11 Context menus
.......................
Open a context menu by right-clicking in a window (Control-click on
macOS). Context menus have the standard clipboard functions also on the
Edit menu.
Cut
Copy selection into the system-wide clipboard; then delete the
selection.
Copy
Copy selection into the system-wide clipboard.
Paste
Insert contents of the system-wide clipboard into the current
window.
Editor windows also have breakpoint functions. Lines with a breakpoint
set are specially marked. Breakpoints only have an effect when running
under the debugger. Breakpoints for a file are saved in the user’s
‘.idlerc’ directory.
Set Breakpoint
Set a breakpoint on the current line.
Clear Breakpoint
Clear the breakpoint on that line.
Shell and Output windows also have the following.
Go to file/line
Same as in Debug menu.
The Shell window also has an output squeezing facility explained in the
'Python Shell window' subsection below.
Squeeze
If the cursor is over an output line, squeeze all the output
between the code above and the prompt below down to a ‘Squeezed
text’ label.
File: python.info, Node: Editing and Navigation, Next: Startup and Code Execution, Prev: Menus, Up: IDLE — Python editor and shell<2>
5.26.9.12 Editing and Navigation
................................
* Menu:
* Editor windows::
* Key bindings::
* Automatic indentation::
* Search and Replace::
* Completions::
* Calltips::
* Code Context::
* Shell window::
* Text colors::
File: python.info, Node: Editor windows, Next: Key bindings, Up: Editing and Navigation
5.26.9.13 Editor windows
........................
IDLE may open editor windows when it starts, depending on settings and
how you start IDLE. Thereafter, use the File menu. There can be only
one open editor window for a given file.
The title bar contains the name of the file, the full path, and the
version of Python and IDLE running the window. The status bar contains
the line number (‘Ln’) and column number (‘Col’). Line numbers start
with 1; column numbers with 0.
IDLE assumes that files with a known .py* extension contain Python code
and that other files do not. Run Python code with the Run menu.
File: python.info, Node: Key bindings, Next: Automatic indentation, Prev: Editor windows, Up: Editing and Navigation
5.26.9.14 Key bindings
......................
The IDLE insertion cursor is a thin vertical bar between character
positions. When characters are entered, the insertion cursor and
everything to its right moves right one character and the new character
is entered in the new space.
Several non-character keys move the cursor and possibly delete
characters. Deletion does not puts text on the clipboard, but IDLE has
an undo list. Wherever this doc discusses keys, ‘C’ refers to the
‘Control’ key on Windows and Unix and the ‘Command’ key on macOS. (And
all such discussions assume that the keys have not been re-bound to
something else.)
* Arrow keys move the cursor one character or line.
* ‘C’-‘LeftArrow’ and ‘C’-‘RightArrow’ moves left or right one word.
* ‘Home’ and ‘End’ go to the beginning or end of the line.
* ‘Page Up’ and ‘Page Down’ go up or down one screen.
* ‘C’-‘Home’ and ‘C’-‘End’ go to beginning or end of the file.
* ‘Backspace’ and ‘Del’ (or ‘C’-‘d’) delete the previous or next
character.
* ‘C’-‘Backspace’ and ‘C’-‘Del’ delete one word left or right.
* ‘C’-‘k’ deletes (‘kills’) everything to the right.
Standard keybindings (like ‘C’-‘c’ to copy and ‘C’-‘v’ to paste) may
work. Keybindings are selected in the Configure IDLE dialog.
File: python.info, Node: Automatic indentation, Next: Search and Replace, Prev: Key bindings, Up: Editing and Navigation
5.26.9.15 Automatic indentation
...............................
After a block-opening statement, the next line is indented by 4 spaces
(in the Python Shell window by one tab). After certain keywords (break,
return etc.) the next line is dedented. In leading indentation,
‘Backspace’ deletes up to 4 spaces if they are there. ‘Tab’ inserts
spaces (in the Python Shell window one tab), number depends on Indent
width. Currently, tabs are restricted to four spaces due to Tcl/Tk
limitations.
See also the indent/dedent region commands on the *note Format menu:
3f3d.
File: python.info, Node: Search and Replace, Next: Completions, Prev: Automatic indentation, Up: Editing and Navigation
5.26.9.16 Search and Replace
............................
Any selection becomes a search target. However, only selections within
a line work because searches are only performed within lines with the
terminal newline removed. If ‘[x] Regular expression’ is checked, the
target is interpreted according to the Python re module.
File: python.info, Node: Completions, Next: Calltips, Prev: Search and Replace, Up: Editing and Navigation
5.26.9.17 Completions
.....................
Completions are supplied, when requested and available, for module
names, attributes of classes or functions, or filenames. Each request
method displays a completion box with existing names. (See tab
completions below for an exception.) For any box, change the name being
completed and the item highlighted in the box by typing and deleting
characters; by hitting ‘Up’, ‘Down’, ‘PageUp’, ‘PageDown’, ‘Home’, and
‘End’ keys; and by a single click within the box. Close the box with
‘Escape’, ‘Enter’, and double ‘Tab’ keys or clicks outside the box. A
double click within the box selects and closes.
One way to open a box is to type a key character and wait for a
predefined interval. This defaults to 2 seconds; customize it in the
settings dialog. (To prevent auto popups, set the delay to a large
number of milliseconds, such as 100000000.) For imported module names
or class or function attributes, type ‘.’. For filenames in the root
directory, type *note os.sep: 666. or *note os.altsep: 667. immediately
after an opening quote. (On Windows, one can specify a drive first.)
Move into subdirectories by typing a directory name and a separator.
Instead of waiting, or after a box is closed, open a completion box
immediately with Show Completions on the Edit menu. The default hot key
is ‘C’-‘space’. If one types a prefix for the desired name before
opening the box, the first match or near miss is made visible. The
result is the same as if one enters a prefix after the box is displayed.
Show Completions after a quote completes filenames in the current
directory instead of a root directory.
Hitting ‘Tab’ after a prefix usually has the same effect as Show
Completions. (With no prefix, it indents.) However, if there is only
one match to the prefix, that match is immediately added to the editor
text without opening a box.
Invoking ‘Show Completions’, or hitting ‘Tab’ after a prefix, outside of
a string and without a preceding ‘.’ opens a box with keywords, builtin
names, and available module-level names.
When editing code in an editor (as oppose to Shell), increase the
available module-level names by running your code and not restarting the
Shell thereafter. This is especially useful after adding imports at the
top of a file. This also increases possible attribute completions.
Completion boxes initially exclude names beginning with ‘_’ or, for
modules, not included in ‘__all__’. The hidden names can be accessed by
typing ‘_’ after ‘.’, either before or after the box is opened.
File: python.info, Node: Calltips, Next: Code Context, Prev: Completions, Up: Editing and Navigation
5.26.9.18 Calltips
..................
A calltip is shown automatically when one types ‘(’ after the name of an
'accessible' function. A function name expression may include dots and
subscripts. A calltip remains until it is clicked, the cursor is moved
out of the argument area, or ‘)’ is typed. Whenever the cursor is in
the argument part of a definition, select Edit and “Show Call Tip” on
the menu or enter its shortcut to display a calltip.
The calltip consists of the function’s signature and docstring up to the
latter’s first blank line or the fifth non-blank line. (Some builtin
functions lack an accessible signature.) A ‘/’ or ‘*’ in the signature
indicates that the preceding or following arguments are passed by
position or name (keyword) only. Details are subject to change.
In Shell, the accessible functions depends on what modules have been
imported into the user process, including those imported by Idle itself,
and which definitions have been run, all since the last restart.
For example, restart the Shell and enter ‘itertools.count(’. A calltip
appears because Idle imports itertools into the user process for its own
use. (This could change.) Enter ‘turtle.write(’ and nothing appears.
Idle does not itself import turtle. The menu entry and shortcut also do
nothing. Enter ‘import turtle’. Thereafter, ‘turtle.write(’ will
display a calltip.
In an editor, import statements have no effect until one runs the file.
One might want to run a file after writing import statements, after
adding function definitions, or after opening an existing file.
File: python.info, Node: Code Context, Next: Shell window, Prev: Calltips, Up: Editing and Navigation
5.26.9.19 Code Context
......................
Within an editor window containing Python code, code context can be
toggled in order to show or hide a pane at the top of the window. When
shown, this pane freezes the opening lines for block code, such as those
beginning with ‘class’, ‘def’, or ‘if’ keywords, that would have
otherwise scrolled out of view. The size of the pane will be expanded
and contracted as needed to show the all current levels of context, up
to the maximum number of lines defined in the Configure IDLE dialog
(which defaults to 15). If there are no current context lines and the
feature is toggled on, a single blank line will display. Clicking on a
line in the context pane will move that line to the top of the editor.
The text and background colors for the context pane can be configured
under the Highlights tab in the Configure IDLE dialog.
File: python.info, Node: Shell window, Next: Text colors, Prev: Code Context, Up: Editing and Navigation
5.26.9.20 Shell window
......................
In IDLE’s Shell, enter, edit, and recall complete statements. (Most
consoles and terminals only work with a single physical line at a time).
Submit a single-line statement for execution by hitting ‘Return’ with
the cursor anywhere on the line. If a line is extended with Backslash
(‘\’), the cursor must be on the last physical line. Submit a
multi-line compound statement by entering a blank line after the
statement.
When one pastes code into Shell, it is not compiled and possibly
executed until one hits ‘Return’, as specified above. One may edit
pasted code first. If one pastes more than one statement into Shell,
the result will be a *note SyntaxError: 18d. when multiple statements
are compiled as if they were one.
Lines containing ‘RESTART’ mean that the user execution process has been
re-started. This occurs when the user execution process has crashed,
when one requests a restart on the Shell menu, or when one runs code in
an editor window.
The editing features described in previous subsections work when
entering code interactively. IDLE’s Shell window also responds to the
following:
* ‘C’-‘c’ attempts to interrupt statement execution (but may fail).
* ‘C’-‘d’ closes Shell if typed at a ‘>>>’ prompt.
* ‘Alt’-‘p’ and ‘Alt’-‘n’ (‘C’-‘p’ and ‘C’-‘n’ on macOS) retrieve to
the current prompt the previous or next previously entered
statement that matches anything already typed.
* ‘Return’ while the cursor is on any previous statement appends the
latter to anything already typed at the prompt.
File: python.info, Node: Text colors, Prev: Shell window, Up: Editing and Navigation
5.26.9.21 Text colors
.....................
Idle defaults to black on white text, but colors text with special
meanings. For the shell, these are shell output, shell error, user
output, and user error. For Python code, at the shell prompt or in an
editor, these are keywords, builtin class and function names, names
following ‘class’ and ‘def’, strings, and comments. For any text
window, these are the cursor (when present), found text (when possible),
and selected text.
IDLE also highlights the *note soft keywords: 808. *note match: 809,
*note case: 809, and *note _: 80a. in pattern-matching statements.
However, this highlighting is not perfect and will be incorrect in some
rare cases, including some ‘_’-s in ‘case’ patterns.
Text coloring is done in the background, so uncolorized text is
occasionally visible. To change the color scheme, use the Configure
IDLE dialog Highlighting tab. The marking of debugger breakpoint lines
in the editor and text in popups and dialogs is not user-configurable.
File: python.info, Node: Startup and Code Execution, Next: Help and Preferences, Prev: Editing and Navigation, Up: IDLE — Python editor and shell<2>
5.26.9.22 Startup and Code Execution
....................................
Upon startup with the ‘-s’ option, IDLE will execute the file referenced
by the environment variables ‘IDLESTARTUP’ or *note PYTHONSTARTUP: fc3.
IDLE first checks for ‘IDLESTARTUP’; if ‘IDLESTARTUP’ is present the
file referenced is run. If ‘IDLESTARTUP’ is not present, IDLE checks
for ‘PYTHONSTARTUP’. Files referenced by these environment variables
are convenient places to store functions that are used frequently from
the IDLE shell, or for executing import statements to import common
modules.
In addition, ‘Tk’ also loads a startup file if it is present. Note that
the Tk file is loaded unconditionally. This additional file is
‘.Idle.py’ and is looked for in the user’s home directory. Statements
in this file will be executed in the Tk namespace, so this file is not
useful for importing functions to be used from IDLE’s Python shell.
* Menu:
* Command line usage::
* Startup failure::
* Running user code::
* User output in Shell::
* Developing tkinter applications::
* Running without a subprocess::
File: python.info, Node: Command line usage, Next: Startup failure, Up: Startup and Code Execution
5.26.9.23 Command line usage
............................
IDLE can be invoked from the command line with various options. The
general syntax is:
python -m idlelib [options] [file ...]
The following options are available:
-- Option: -c
Run the specified Python command in the shell window. For example,
pass ‘-c "print('Hello, World!')"’. On Windows, the outer quotes
must be double quotes as shown.
-- Option: -d
Enable the debugger and open the shell window.
-- Option: -e
Open an editor window.
-- Option: -h
Print a help message with legal combinations of options and exit.
-- Option: -i
Open a shell window.
-- Option: -r
Run the specified file in the shell window.
-- Option: -s
Run the startup file (as defined by the environment variables
‘IDLESTARTUP’ or *note PYTHONSTARTUP: fc3.) before opening the
shell window.
-- Option: -t
Set the title of the shell window.
-- Option: -
Read and execute standard input in the shell window. This option
must be the last one before any arguments.
If arguments are provided:
- If ‘-’, ‘-c’, or ‘-r’ is used, all arguments are placed in
‘sys.argv[1:]’, and ‘sys.argv[0]’ is set to ‘''’, ‘'-c'’, or ‘'-r'’
respectively. No editor window is opened, even if that is the
default set in the 'Options' dialog.
- Otherwise, arguments are treated as files to be opened for editing,
and ‘sys.argv’ reflects the arguments passed to IDLE itself.
File: python.info, Node: Startup failure, Next: Running user code, Prev: Command line usage, Up: Startup and Code Execution
5.26.9.24 Startup failure
.........................
IDLE uses a socket to communicate between the IDLE GUI process and the
user code execution process. A connection must be established whenever
the Shell starts or restarts. (The latter is indicated by a divider
line that says ‘RESTART’). If the user process fails to connect to the
GUI process, it usually displays a ‘Tk’ error box with a ‘cannot
connect’ message that directs the user here. It then exits.
One specific connection failure on Unix systems results from
misconfigured masquerading rules somewhere in a system’s network setup.
When IDLE is started from a terminal, one will see a message starting
with ‘** Invalid host:’. The valid value is ‘127.0.0.1
(idlelib.rpc.LOCALHOST)’. One can diagnose with ‘tcpconnect -irv
127.0.0.1 6543’ in one terminal window and ‘tcplisten ’ in
another.
A common cause of failure is a user-written file with the same name as a
standard library module, such as 'random.py' and 'tkinter.py'. When
such a file is located in the same directory as a file that is about to
be run, IDLE cannot import the stdlib file. The current fix is to
rename the user file.
Though less common than in the past, an antivirus or firewall program
may stop the connection. If the program cannot be taught to allow the
connection, then it must be turned off for IDLE to work. It is safe to
allow this internal connection because no data is visible on external
ports. A similar problem is a network mis-configuration that blocks
connections.
Python installation issues occasionally stop IDLE: multiple versions can
clash, or a single installation might need admin access. If one undo
the clash, or cannot or does not want to run as admin, it might be
easiest to completely remove Python and start over.
A zombie pythonw.exe process could be a problem. On Windows, use Task
Manager to check for one and stop it if there is. Sometimes a restart
initiated by a program crash or Keyboard Interrupt (control-C) may fail
to connect. Dismissing the error box or using Restart Shell on the
Shell menu may fix a temporary problem.
When IDLE first starts, it attempts to read user configuration files in
‘~/.idlerc/’ (~ is one’s home directory). If there is a problem, an
error message should be displayed. Leaving aside random disk glitches,
this can be prevented by never editing the files by hand. Instead, use
the configuration dialog, under Options. Once there is an error in a
user configuration file, the best solution may be to delete it and start
over with the settings dialog.
If IDLE quits with no message, and it was not started from a console,
try starting it from a console or terminal (‘python -m idlelib’) and see
if this results in an error message.
On Unix-based systems with tcl/tk older than ‘8.6.11’ (see ‘About IDLE’)
certain characters of certain fonts can cause a tk failure with a
message to the terminal. This can happen either if one starts IDLE to
edit a file with such a character or later when entering such a
character. If one cannot upgrade tcl/tk, then re-configure IDLE to use
a font that works better.
File: python.info, Node: Running user code, Next: User output in Shell, Prev: Startup failure, Up: Startup and Code Execution
5.26.9.25 Running user code
...........................
With rare exceptions, the result of executing Python code with IDLE is
intended to be the same as executing the same code by the default
method, directly with Python in a text-mode system console or terminal
window. However, the different interface and operation occasionally
affect visible results. For instance, ‘sys.modules’ starts with more
entries, and ‘threading.active_count()’ returns 2 instead of 1.
By default, IDLE runs user code in a separate OS process rather than in
the user interface process that runs the shell and editor. In the
execution process, it replaces ‘sys.stdin’, ‘sys.stdout’, and
‘sys.stderr’ with objects that get input from and send output to the
Shell window. The original values stored in ‘sys.__stdin__’,
‘sys.__stdout__’, and ‘sys.__stderr__’ are not touched, but may be
‘None’.
Sending print output from one process to a text widget in another is
slower than printing to a system terminal in the same process. This has
the most effect when printing multiple arguments, as the string for each
argument, each separator, the newline are sent separately. For
development, this is usually not a problem, but if one wants to print
faster in IDLE, format and join together everything one wants displayed
together and then print a single string. Both format strings and *note
str.join(): 21ab. can help combine fields and lines.
IDLE’s standard stream replacements are not inherited by subprocesses
created in the execution process, whether directly by user code or by
modules such as multiprocessing. If such subprocess use ‘input’ from
sys.stdin or ‘print’ or ‘write’ to sys.stdout or sys.stderr, IDLE should
be started in a command line window. (On Windows, use ‘python’ or ‘py’
rather than ‘pythonw’ or ‘pyw’.) The secondary subprocess will then be
attached to that window for input and output.
If ‘sys’ is reset by user code, such as with ‘importlib.reload(sys)’,
IDLE’s changes are lost and input from the keyboard and output to the
screen will not work correctly.
When Shell has the focus, it controls the keyboard and screen. This is
normally transparent, but functions that directly access the keyboard
and screen will not work. These include system-specific functions that
determine whether a key has been pressed and if so, which.
The IDLE code running in the execution process adds frames to the call
stack that would not be there otherwise. IDLE wraps
‘sys.getrecursionlimit’ and ‘sys.setrecursionlimit’ to reduce the effect
of the additional stack frames.
When user code raises SystemExit either directly or by calling sys.exit,
IDLE returns to a Shell prompt instead of exiting.
File: python.info, Node: User output in Shell, Next: Developing tkinter applications, Prev: Running user code, Up: Startup and Code Execution
5.26.9.26 User output in Shell
..............................
When a program outputs text, the result is determined by the
corresponding output device. When IDLE executes user code, ‘sys.stdout’
and ‘sys.stderr’ are connected to the display area of IDLE’s Shell.
Some of its features are inherited from the underlying Tk Text widget.
Others are programmed additions. Where it matters, Shell is designed
for development rather than production runs.
For instance, Shell never throws away output. A program that sends
unlimited output to Shell will eventually fill memory, resulting in a
memory error. In contrast, some system text windows only keep the last
n lines of output. A Windows console, for instance, keeps a
user-settable 1 to 9999 lines, with 300 the default.
A Tk Text widget, and hence IDLE’s Shell, displays characters
(codepoints) in the BMP (Basic Multilingual Plane) subset of Unicode.
Which characters are displayed with a proper glyph and which with a
replacement box depends on the operating system and installed fonts.
Tab characters cause the following text to begin after the next tab
stop. (They occur every 8 ‘characters’). Newline characters cause
following text to appear on a new line. Other control characters are
ignored or displayed as a space, box, or something else, depending on
the operating system and font. (Moving the text cursor through such
output with arrow keys may exhibit some surprising spacing behavior.)
>>> s = 'a\tb\a<\x02><\r>\bc\nd' # Enter 22 chars.
>>> len(s)
14
>>> s # Display repr(s)
'a\tb\x07<\x02><\r>\x08c\nd'
>>> print(s, end='') # Display s as is.
# Result varies by OS and font. Try it.
The ‘repr’ function is used for interactive echo of expression values.
It returns an altered version of the input string in which control
codes, some BMP codepoints, and all non-BMP codepoints are replaced with
escape codes. As demonstrated above, it allows one to identify the
characters in a string, regardless of how they are displayed.
Normal and error output are generally kept separate (on separate lines)
from code input and each other. They each get different highlight
colors.
For SyntaxError tracebacks, the normal ‘^’ marking where the error was
detected is replaced by coloring the text with an error highlight. When
code run from a file causes other exceptions, one may right click on a
traceback line to jump to the corresponding line in an IDLE editor. The
file will be opened if necessary.
Shell has a special facility for squeezing output lines down to a
‘Squeezed text’ label. This is done automatically for output over N
lines (N = 50 by default). N can be changed in the PyShell section of
the General page of the Settings dialog. Output with fewer lines can be
squeezed by right clicking on the output. This can be useful lines long
enough to slow down scrolling.
Squeezed output is expanded in place by double-clicking the label. It
can also be sent to the clipboard or a separate view window by
right-clicking the label.
File: python.info, Node: Developing tkinter applications, Next: Running without a subprocess, Prev: User output in Shell, Up: Startup and Code Execution
5.26.9.27 Developing tkinter applications
.........................................
IDLE is intentionally different from standard Python in order to
facilitate development of tkinter programs. Enter ‘import tkinter as
tk; root = tk.Tk()’ in standard Python and nothing appears. Enter the
same in IDLE and a tk window appears. In standard Python, one must also
enter ‘root.update()’ to see the window. IDLE does the equivalent in
the background, about 20 times a second, which is about every 50
milliseconds. Next enter ‘b = tk.Button(root, text='button');
b.pack()’. Again, nothing visibly changes in standard Python until one
enters ‘root.update()’.
Most tkinter programs run ‘root.mainloop()’, which usually does not
return until the tk app is destroyed. If the program is run with
‘python -i’ or from an IDLE editor, a ‘>>>’ shell prompt does not appear
until ‘mainloop()’ returns, at which time there is nothing left to
interact with.
When running a tkinter program from an IDLE editor, one can comment out
the mainloop call. One then gets a shell prompt immediately and can
interact with the live application. One just has to remember to
re-enable the mainloop call when running in standard Python.
File: python.info, Node: Running without a subprocess, Prev: Developing tkinter applications, Up: Startup and Code Execution
5.26.9.28 Running without a subprocess
......................................
By default, IDLE executes user code in a separate subprocess via a
socket, which uses the internal loopback interface. This connection is
not externally visible and no data is sent to or received from the
internet. If firewall software complains anyway, you can ignore it.
If the attempt to make the socket connection fails, Idle will notify
you. Such failures are sometimes transient, but if persistent, the
problem may be either a firewall blocking the connection or
misconfiguration of a particular system. Until the problem is fixed,
one can run Idle with the -n command line switch.
If IDLE is started with the -n command line switch it will run in a
single process and will not create the subprocess which runs the RPC
Python execution server. This can be useful if Python cannot create the
subprocess or the RPC socket interface on your platform. However, in
this mode user code is not isolated from IDLE itself. Also, the
environment is not restarted when Run/Run Module (F5) is selected. If
your code has been modified, you must reload() the affected modules and
re-import any specific items (e.g. from foo import baz) if the changes
are to take effect. For these reasons, it is preferable to run IDLE
with the default subprocess if at all possible.
Deprecated since version 3.4.
File: python.info, Node: Help and Preferences, Next: idlelib — implementation of IDLE application, Prev: Startup and Code Execution, Up: IDLE — Python editor and shell<2>
5.26.9.29 Help and Preferences
..............................
* Menu:
* Help sources::
* Setting preferences::
* IDLE on macOS::
* Extensions::
File: python.info, Node: Help sources, Next: Setting preferences, Up: Help and Preferences
5.26.9.30 Help sources
......................
Help menu entry “IDLE Help” displays a formatted html version of the
IDLE chapter of the Library Reference. The result, in a read-only
tkinter text window, is close to what one sees in a web browser.
Navigate through the text with a mousewheel, the scrollbar, or up and
down arrow keys held down. Or click the TOC (Table of Contents) button
and select a section header in the opened box.
Help menu entry “Python Docs” opens the extensive sources of help,
including tutorials, available at ‘docs.python.org/x.y’, where ‘x.y’ is
the currently running Python version. If your system has an off-line
copy of the docs (this may be an installation option), that will be
opened instead.
Selected URLs can be added or removed from the help menu at any time
using the General tab of the Configure IDLE dialog.
File: python.info, Node: Setting preferences, Next: IDLE on macOS, Prev: Help sources, Up: Help and Preferences
5.26.9.31 Setting preferences
.............................
The font preferences, highlighting, keys, and general preferences can be
changed via Configure IDLE on the Option menu. Non-default user
settings are saved in a ‘.idlerc’ directory in the user’s home
directory. Problems caused by bad user configuration files are solved
by editing or deleting one or more of the files in ‘.idlerc’.
On the Font tab, see the text sample for the effect of font face and
size on multiple characters in multiple languages. Edit the sample to
add other characters of personal interest. Use the sample to select
monospaced fonts. If particular characters have problems in Shell or an
editor, add them to the top of the sample and try changing first size
and then font.
On the Highlights and Keys tab, select a built-in or custom color theme
and key set. To use a newer built-in color theme or key set with older
IDLEs, save it as a new custom theme or key set and it well be
accessible to older IDLEs.
File: python.info, Node: IDLE on macOS, Next: Extensions, Prev: Setting preferences, Up: Help and Preferences
5.26.9.32 IDLE on macOS
.......................
Under System Preferences: Dock, one can set “Prefer tabs when opening
documents” to “Always”. This setting is not compatible with the
tk/tkinter GUI framework used by IDLE, and it breaks a few IDLE
features.
File: python.info, Node: Extensions, Prev: IDLE on macOS, Up: Help and Preferences
5.26.9.33 Extensions
....................
IDLE contains an extension facility. Preferences for extensions can be
changed with the Extensions tab of the preferences dialog. See the
beginning of config-extensions.def in the idlelib directory for further
information. The only current default extension is zzdummy, an example
also used for testing.
File: python.info, Node: idlelib — implementation of IDLE application, Prev: Help and Preferences, Up: IDLE — Python editor and shell<2>
5.26.9.34 idlelib — implementation of IDLE application
......................................................
'Source code:' Lib/idlelib(1)
__________________________________________________________________
The Lib/idlelib package implements the IDLE application. See the rest
of this page for how to use IDLE.
The files in idlelib are described in idlelib/README.txt. Access it
either in idlelib or click Help => About IDLE on the IDLE menu. This
file also maps IDLE menu items to the code that implements the item.
Except for files listed under ‘Startup’, the idlelib code is ‘private’
in sense that feature changes can be backported (see PEP 434(2)).
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/idlelib
(2) https://peps.python.org/pep-0434/
File: python.info, Node: Development Tools, Next: Debugging and Profiling, Prev: Graphical User Interfaces with Tk, Up: The Python Standard Library
5.27 Development Tools
======================
The modules described in this chapter help you write software. For
example, the *note pydoc: b5. module takes a module and generates
documentation based on the module’s contents. The *note doctest: 3a.
and *note unittest: 106. modules contains frameworks for writing unit
tests that automatically exercise code and verify that the expected
output is produced.
The list of modules described in this chapter is:
* Menu:
* typing — Support for type hints::
* pydoc — Documentation generator and online help system::
* Python Development Mode::
* doctest — Test interactive Python examples::
* unittest — Unit testing framework::
* unittest.mock — mock object library: unittest mock — mock object library.
* unittest.mock — getting started: unittest mock — getting started.
* test — Regression tests package for Python::
* test.support — Utilities for the Python test suite: test support — Utilities for the Python test suite.
* test.support.socket_helper — Utilities for socket tests: test support socket_helper — Utilities for socket tests.
* test.support.script_helper — Utilities for the Python execution tests: test support script_helper — Utilities for the Python execution tests.
* test.support.bytecode_helper — Support tools for testing correct bytecode generation: test support bytecode_helper — Support tools for testing correct bytecode generation.
* test.support.threading_helper — Utilities for threading tests: test support threading_helper — Utilities for threading tests.
* test.support.os_helper — Utilities for os tests: test support os_helper — Utilities for os tests.
* test.support.import_helper — Utilities for import tests: test support import_helper — Utilities for import tests.
* test.support.warnings_helper — Utilities for warnings tests: test support warnings_helper — Utilities for warnings tests.
File: python.info, Node: typing — Support for type hints, Next: pydoc — Documentation generator and online help system, Up: Development Tools
5.27.1 ‘typing’ — Support for type hints
----------------------------------------
Added in version 3.5.
'Source code:' Lib/typing.py(1)
Note: The Python runtime does not enforce function and variable
type annotations. They can be used by third party tools such as
*note type checkers: 26f, IDEs, linters, etc.
__________________________________________________________________
This module provides runtime support for type hints.
Consider the function below:
def surface_area_of_cube(edge_length: float) -> str:
return f"The surface area of the cube is {6 * edge_length ** 2}."
The function ‘surface_area_of_cube’ takes an argument expected to be an
instance of *note float: 2f1, as indicated by the *note type hint: 1f81.
‘edge_length: float’. The function is expected to return an instance of
*note str: 447, as indicated by the ‘-> str’ hint.
While type hints can be simple classes like *note float: 2f1. or *note
str: 447, they can also be more complex. The *note typing: 104. module
provides a vocabulary of more advanced type hints.
New features are frequently added to the ‘typing’ module. The
typing_extensions(2) package provides backports of these new features to
older versions of Python.
See also
........
Typing cheat sheet(3)
A quick overview of type hints (hosted at the mypy docs)
Type System Reference section of the mypy docs(4)
The Python typing system is standardised via PEPs, so this
reference should broadly apply to most Python type checkers. (Some
parts may still be specific to mypy.)
Static Typing with Python(5)
Type-checker-agnostic documentation written by the community
detailing type system features, useful typing related tools and
typing best practices.
* Menu:
* Specification for the Python Type System::
* Type aliases::
* NewType::
* Annotating callable objects::
* Generics::
* Annotating tuples::
* The type of class objects::
* Annotating generators and coroutines::
* User-defined generic types::
* The Any type::
* Nominal vs structural subtyping::
* Module contents: Module contents<3>.
* Deprecation Timeline of Major Features::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/typing.py
(2) https://pypi.org/project/typing_extensions/
(3) https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html
(4) https://mypy.readthedocs.io/en/stable/index.html
(5) https://typing.python.org/en/latest/
File: python.info, Node: Specification for the Python Type System, Next: Type aliases, Up: typing — Support for type hints
5.27.1.1 Specification for the Python Type System
.................................................
The canonical, up-to-date specification of the Python type system can be
found at Specification for the Python type system(1).
---------- Footnotes ----------
(1) https://typing.python.org/en/latest/spec/index.html
File: python.info, Node: Type aliases, Next: NewType, Prev: Specification for the Python Type System, Up: typing — Support for type hints
5.27.1.2 Type aliases
.....................
A type alias is defined using the *note type: 433. statement, which
creates an instance of *note TypeAliasType: 450. In this example,
‘Vector’ and ‘list[float]’ will be treated equivalently by static type
checkers:
type Vector = list[float]
def scale(scalar: float, vector: Vector) -> Vector:
return [scalar * num for num in vector]
# passes type checking; a list of floats qualifies as a Vector.
new_vector = scale(2.0, [1.0, -4.2, 5.4])
Type aliases are useful for simplifying complex type signatures. For
example:
from collections.abc import Sequence
type ConnectionOptions = dict[str, str]
type Address = tuple[str, int]
type Server = tuple[Address, ConnectionOptions]
def broadcast_message(message: str, servers: Sequence[Server]) -> None:
...
# The static type checker will treat the previous type signature as
# being exactly equivalent to this one.
def broadcast_message(
message: str,
servers: Sequence[tuple[tuple[str, int], dict[str, str]]]
) -> None:
...
The *note type: 433. statement is new in Python 3.12. For backwards
compatibility, type aliases can also be created through simple
assignment:
Vector = list[float]
Or marked with *note TypeAlias: 7c4. to make it explicit that this is a
type alias, not a normal variable assignment:
from typing import TypeAlias
Vector: TypeAlias = list[float]
File: python.info, Node: NewType, Next: Annotating callable objects, Prev: Type aliases, Up: typing — Support for type hints
5.27.1.3 NewType
................
Use the *note NewType: cf5. helper to create distinct types:
from typing import NewType
UserId = NewType('UserId', int)
some_id = UserId(524313)
The static type checker will treat the new type as if it were a subclass
of the original type. This is useful in helping catch logical errors:
def get_user_name(user_id: UserId) -> str:
...
# passes type checking
user_a = get_user_name(UserId(42351))
# fails type checking; an int is not a UserId
user_b = get_user_name(-1)
You may still perform all ‘int’ operations on a variable of type
‘UserId’, but the result will always be of type ‘int’. This lets you
pass in a ‘UserId’ wherever an ‘int’ might be expected, but will prevent
you from accidentally creating a ‘UserId’ in an invalid way:
# 'output' is of type 'int', not 'UserId'
output = UserId(23413) + UserId(54341)
Note that these checks are enforced only by the static type checker. At
runtime, the statement ‘Derived = NewType('Derived', Base)’ will make
‘Derived’ a callable that immediately returns whatever parameter you
pass it. That means the expression ‘Derived(some_value)’ does not
create a new class or introduce much overhead beyond that of a regular
function call.
More precisely, the expression ‘some_value is Derived(some_value)’ is
always true at runtime.
It is invalid to create a subtype of ‘Derived’:
from typing import NewType
UserId = NewType('UserId', int)
# Fails at runtime and does not pass type checking
class AdminUserId(UserId): pass
However, it is possible to create a *note NewType: cf5. based on a
‘derived’ ‘NewType’:
from typing import NewType
UserId = NewType('UserId', int)
ProUserId = NewType('ProUserId', UserId)
and typechecking for ‘ProUserId’ will work as expected.
See PEP 484(1) for more details.
Note: Recall that the use of a type alias declares two types to be
'equivalent' to one another. Doing ‘type Alias = Original’ will
make the static type checker treat ‘Alias’ as being 'exactly
equivalent' to ‘Original’ in all cases. This is useful when you
want to simplify complex type signatures.
In contrast, ‘NewType’ declares one type to be a 'subtype' of
another. Doing ‘Derived = NewType('Derived', Original)’ will make
the static type checker treat ‘Derived’ as a 'subclass' of
‘Original’, which means a value of type ‘Original’ cannot be used
in places where a value of type ‘Derived’ is expected. This is
useful when you want to prevent logic errors with minimal runtime
cost.
Added in version 3.5.2.
Changed in version 3.10: ‘NewType’ is now a class rather than a
function. As a result, there is some additional runtime cost when
calling ‘NewType’ over a regular function.
Changed in version 3.11: The performance of calling ‘NewType’ has been
restored to its level in Python 3.9.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
File: python.info, Node: Annotating callable objects, Next: Generics, Prev: NewType, Up: typing — Support for type hints
5.27.1.4 Annotating callable objects
....................................
Functions – or other *note callable: 28f7. objects – can be annotated
using *note collections.abc.Callable: 7e5. or deprecated *note
typing.Callable: 7c0. ‘Callable[[int], str]’ signifies a function that
takes a single parameter of type *note int: 259. and returns a *note
str: 447.
For example:
from collections.abc import Callable, Awaitable
def feeder(get_next_item: Callable[[], str]) -> None:
... # Body
def async_query(on_success: Callable[[int], None],
on_error: Callable[[int, Exception], None]) -> None:
... # Body
async def on_update(value: str) -> None:
... # Body
callback: Callable[[str], Awaitable[None]] = on_update
The subscription syntax must always be used with exactly two values: the
argument list and the return type. The argument list must be a list of
types, a *note ParamSpec: 15e, *note Concatenate: 7bf, or an ellipsis.
The return type must be a single type.
If a literal ellipsis ‘...’ is given as the argument list, it indicates
that a callable with any arbitrary parameter list would be acceptable:
def concat(x: str, y: str) -> str:
return x + y
x: Callable[..., str]
x = str # OK
x = concat # Also OK
‘Callable’ cannot express complex signatures such as functions that take
a variadic number of arguments, *note overloaded functions: 3f79, or
functions that have keyword-only parameters. However, these signatures
can be expressed by defining a *note Protocol: 266. class with a *note
__call__(): 555. method:
from collections.abc import Iterable
from typing import Protocol
class Combiner(Protocol):
def __call__(self, *vals: bytes, maxlen: int | None = None) -> list[bytes]: ...
def batch_proc(data: Iterable[bytes], cb_results: Combiner) -> bytes:
for item in data:
...
def good_cb(*vals: bytes, maxlen: int | None = None) -> list[bytes]:
...
def bad_cb(*vals: bytes, maxitems: int | None) -> list[bytes]:
...
batch_proc([], good_cb) # OK
batch_proc([], bad_cb) # Error! Argument 2 has incompatible type because of
# different name and kind in the callback
Callables which take other callables as arguments may indicate that
their parameter types are dependent on each other using *note ParamSpec:
15e. Additionally, if that callable adds or removes arguments from
other callables, the *note Concatenate: 7bf. operator may be used. They
take the form ‘Callable[ParamSpecVariable, ReturnType]’ and
‘Callable[Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable],
ReturnType]’ respectively.
Changed in version 3.10: ‘Callable’ now supports *note ParamSpec: 15e.
and *note Concatenate: 7bf. See PEP 612(1) for more details.
See also
........
The documentation for *note ParamSpec: 15e. and *note Concatenate: 7bf.
provides examples of usage in ‘Callable’.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0612/
File: python.info, Node: Generics, Next: Annotating tuples, Prev: Annotating callable objects, Up: typing — Support for type hints
5.27.1.5 Generics
.................
Since type information about objects kept in containers cannot be
statically inferred in a generic way, many container classes in the
standard library support subscription to denote the expected types of
container elements.
from collections.abc import Mapping, Sequence
class Employee: ...
# Sequence[Employee] indicates that all elements in the sequence
# must be instances of "Employee".
# Mapping[str, str] indicates that all keys and all values in the mapping
# must be strings.
def notify_by_email(employees: Sequence[Employee],
overrides: Mapping[str, str]) -> None: ...
Generic functions and classes can be parameterized by using *note type
parameter syntax: 2b5.:
from collections.abc import Sequence
def first[T](l: Sequence[T]) -> T: # Function is generic over the TypeVar "T"
return l[0]
Or by using the *note TypeVar: 15d. factory directly:
from collections.abc import Sequence
from typing import TypeVar
U = TypeVar('U') # Declare type variable "U"
def second(l: Sequence[U]) -> U: # Function is generic over the TypeVar "U"
return l[1]
Changed in version 3.12: Syntactic support for generics is new in Python
3.12.
File: python.info, Node: Annotating tuples, Next: The type of class objects, Prev: Generics, Up: typing — Support for type hints
5.27.1.6 Annotating tuples
..........................
For most containers in Python, the typing system assumes that all
elements in the container will be of the same type. For example:
from collections.abc import Mapping
# Type checker will infer that all elements in ``x`` are meant to be ints
x: list[int] = []
# Type checker error: ``list`` only accepts a single type argument:
y: list[int, str] = [1, 'foo']
# Type checker will infer that all keys in ``z`` are meant to be strings,
# and that all values in ``z`` are meant to be either strings or ints
z: Mapping[str, str | int] = {}
*note list: 60d. only accepts one type argument, so a type checker would
emit an error on the ‘y’ assignment above. Similarly, *note Mapping:
8c9. only accepts two type arguments: the first indicates the type of
the keys, and the second indicates the type of the values.
Unlike most other Python containers, however, it is common in idiomatic
Python code for tuples to have elements which are not all of the same
type. For this reason, tuples are special-cased in Python’s typing
system. *note tuple: 36b. accepts 'any number' of type arguments:
# OK: ``x`` is assigned to a tuple of length 1 where the sole element is an int
x: tuple[int] = (5,)
# OK: ``y`` is assigned to a tuple of length 2;
# element 1 is an int, element 2 is a str
y: tuple[int, str] = (5, "foo")
# Error: the type annotation indicates a tuple of length 1,
# but ``z`` has been assigned to a tuple of length 3
z: tuple[int] = (1, 2, 3)
To denote a tuple which could be of 'any' length, and in which all
elements are of the same type ‘T’, use ‘tuple[T, ...]’. To denote an
empty tuple, use ‘tuple[()]’. Using plain ‘tuple’ as an annotation is
equivalent to using ‘tuple[Any, ...]’:
x: tuple[int, ...] = (1, 2)
# These reassignments are OK: ``tuple[int, ...]`` indicates x can be of any length
x = (1, 2, 3)
x = ()
# This reassignment is an error: all elements in ``x`` must be ints
x = ("foo", "bar")
# ``y`` can only ever be assigned to an empty tuple
y: tuple[()] = ()
z: tuple = ("foo", "bar")
# These reassignments are OK: plain ``tuple`` is equivalent to ``tuple[Any, ...]``
z = (1, 2, 3)
z = ()
File: python.info, Node: The type of class objects, Next: Annotating generators and coroutines, Prev: Annotating tuples, Up: typing — Support for type hints
5.27.1.7 The type of class objects
..................................
A variable annotated with ‘C’ may accept a value of type ‘C’. In
contrast, a variable annotated with ‘type[C]’ (or deprecated *note
typing.Type[C]: 3f7f.) may accept values that are classes themselves –
specifically, it will accept the 'class object' of ‘C’. For example:
a = 3 # Has type ``int``
b = int # Has type ``type[int]``
c = type(a) # Also has type ``type[int]``
Note that ‘type[C]’ is covariant:
class User: ...
class ProUser(User): ...
class TeamUser(User): ...
def make_new_user(user_class: type[User]) -> User:
# ...
return user_class()
make_new_user(User) # OK
make_new_user(ProUser) # Also OK: ``type[ProUser]`` is a subtype of ``type[User]``
make_new_user(TeamUser) # Still fine
make_new_user(User()) # Error: expected ``type[User]`` but got ``User``
make_new_user(int) # Error: ``type[int]`` is not a subtype of ``type[User]``
The only legal parameters for *note type: d48. are classes, *note Any:
6a8, *note type variables: 1f7f, and unions of any of these types. For
example:
def new_non_team_user(user_class: type[BasicUser | ProUser]): ...
new_non_team_user(BasicUser) # OK
new_non_team_user(ProUser) # OK
new_non_team_user(TeamUser) # Error: ``type[TeamUser]`` is not a subtype
# of ``type[BasicUser | ProUser]``
new_non_team_user(User) # Also an error
‘type[Any]’ is equivalent to *note type: d48, which is the root of
Python’s *note metaclass hierarchy: 1f73.
File: python.info, Node: Annotating generators and coroutines, Next: User-defined generic types, Prev: The type of class objects, Up: typing — Support for type hints
5.27.1.8 Annotating generators and coroutines
.............................................
A generator can be annotated using the generic type *note
Generator[YieldType, SendType, ReturnType]: dda. For example:
def echo_round() -> Generator[int, float, str]:
sent = yield 0
while sent >= 0:
sent = yield round(sent)
return 'Done'
Note that unlike many other generic classes in the standard library, the
‘SendType’ of *note Generator: dda. behaves contravariantly, not
covariantly or invariantly.
The ‘SendType’ and ‘ReturnType’ parameters default to ‘None’:
def infinite_stream(start: int) -> Generator[int]:
while True:
yield start
start += 1
It is also possible to set these types explicitly:
def infinite_stream(start: int) -> Generator[int, None, None]:
while True:
yield start
start += 1
Simple generators that only ever yield values can also be annotated as
having a return type of either *note Iterable[YieldType]: 224d. or *note
Iterator[YieldType]: 224e.:
def infinite_stream(start: int) -> Iterator[int]:
while True:
yield start
start += 1
Async generators are handled in a similar fashion, but don’t expect a
‘ReturnType’ type argument (*note AsyncGenerator[YieldType, SendType]:
c8a.). The ‘SendType’ argument defaults to ‘None’, so the following
definitions are equivalent:
async def infinite_stream(start: int) -> AsyncGenerator[int]:
while True:
yield start
start = await increment(start)
async def infinite_stream(start: int) -> AsyncGenerator[int, None]:
while True:
yield start
start = await increment(start)
As in the synchronous case, *note AsyncIterable[YieldType]: dde. and
*note AsyncIterator[YieldType]: ddd. are available as well:
async def infinite_stream(start: int) -> AsyncIterator[int]:
while True:
yield start
start = await increment(start)
Coroutines can be annotated using *note Coroutine[YieldType, SendType,
ReturnType]: ddc. Generic arguments correspond to those of *note
Generator: dda, for example:
from collections.abc import Coroutine
c: Coroutine[list[str], str, int] # Some coroutine defined elsewhere
x = c.send('hi') # Inferred type of 'x' is list[str]
async def bar() -> None:
y = await c # Inferred type of 'y' is int
File: python.info, Node: User-defined generic types, Next: The Any type, Prev: Annotating generators and coroutines, Up: typing — Support for type hints
5.27.1.9 User-defined generic types
...................................
A user-defined class can be defined as a generic class.
from logging import Logger
class LoggedVar[T]:
def __init__(self, value: T, name: str, logger: Logger) -> None:
self.name = name
self.logger = logger
self.value = value
def set(self, new: T) -> None:
self.log('Set ' + repr(self.value))
self.value = new
def get(self) -> T:
self.log('Get ' + repr(self.value))
return self.value
def log(self, message: str) -> None:
self.logger.info('%s: %s', self.name, message)
This syntax indicates that the class ‘LoggedVar’ is parameterised around
a single *note type variable: 3f82. ‘T’ . This also makes ‘T’ valid as
a type within the class body.
Generic classes implicitly inherit from *note Generic: 1619. For
compatibility with Python 3.11 and lower, it is also possible to inherit
explicitly from *note Generic: 1619. to indicate a generic class:
from typing import TypeVar, Generic
T = TypeVar('T')
class LoggedVar(Generic[T]):
...
Generic classes have *note __class_getitem__(): 736. methods, meaning
they can be parameterised at runtime (e.g. ‘LoggedVar[int]’ below):
from collections.abc import Iterable
def zero_all_vars(vars: Iterable[LoggedVar[int]]) -> None:
for var in vars:
var.set(0)
A generic type can have any number of type variables. All varieties of
*note TypeVar: 15d. are permissible as parameters for a generic type:
from typing import TypeVar, Generic, Sequence
class WeirdTrio[T, B: Sequence[bytes], S: (int, str)]:
...
OldT = TypeVar('OldT', contravariant=True)
OldB = TypeVar('OldB', bound=Sequence[bytes], covariant=True)
OldS = TypeVar('OldS', int, str)
class OldWeirdTrio(Generic[OldT, OldB, OldS]):
...
Each type variable argument to *note Generic: 1619. must be distinct.
This is thus invalid:
from typing import TypeVar, Generic
...
class Pair[M, M]: # SyntaxError
...
T = TypeVar('T')
class Pair(Generic[T, T]): # INVALID
...
Generic classes can also inherit from other classes:
from collections.abc import Sized
class LinkedList[T](Sized):
...
When inheriting from generic classes, some type parameters could be
fixed:
from collections.abc import Mapping
class MyDict[T](Mapping[str, T]):
...
In this case ‘MyDict’ has a single parameter, ‘T’.
Using a generic class without specifying type parameters assumes *note
Any: 6a8. for each position. In the following example, ‘MyIterable’ is
not generic but implicitly inherits from ‘Iterable[Any]’:
from collections.abc import Iterable
class MyIterable(Iterable): # Same as Iterable[Any]
...
User-defined generic type aliases are also supported. Examples:
from collections.abc import Iterable
type Response[S] = Iterable[S] | int
# Return type here is same as Iterable[str] | int
def response(query: str) -> Response[str]:
...
type Vec[T] = Iterable[tuple[T, T]]
def inproduct[T: (int, float, complex)](v: Vec[T]) -> T: # Same as Iterable[tuple[T, T]]
return sum(x*y for x, y in v)
For backward compatibility, generic type aliases can also be created
through a simple assignment:
from collections.abc import Iterable
from typing import TypeVar
S = TypeVar("S")
Response = Iterable[S] | int
Changed in version 3.7: *note Generic: 1619. no longer has a custom
metaclass.
Changed in version 3.12: Syntactic support for generics and type aliases
is new in version 3.12. Previously, generic classes had to explicitly
inherit from *note Generic: 1619. or contain a type variable in one of
their bases.
User-defined generics for parameter expressions are also supported via
parameter specification variables in the form ‘[**P]’. The behavior is
consistent with type variables’ described above as parameter
specification variables are treated by the ‘typing’ module as a
specialized type variable. The one exception to this is that a list of
types can be used to substitute a *note ParamSpec: 15e.:
>>> class Z[T, **P]: ... # T is a TypeVar; P is a ParamSpec
...
>>> Z[int, [dict, float]]
__main__.Z[int, [dict, float]]
Classes generic over a *note ParamSpec: 15e. can also be created using
explicit inheritance from *note Generic: 1619. In this case, ‘**’ is
not used:
from typing import ParamSpec, Generic
P = ParamSpec('P')
class Z(Generic[P]):
...
Another difference between *note TypeVar: 15d. and *note ParamSpec: 15e.
is that a generic with only one parameter specification variable will
accept parameter lists in the forms ‘X[[Type1, Type2, ...]]’ and also
‘X[Type1, Type2, ...]’ for aesthetic reasons. Internally, the latter is
converted to the former, so the following are equivalent:
>>> class X[**P]: ...
...
>>> X[int, str]
__main__.X[[int, str]]
>>> X[[int, str]]
__main__.X[[int, str]]
Note that generics with *note ParamSpec: 15e. may not have correct
‘__parameters__’ after substitution in some cases because they are
intended primarily for static type checking.
Changed in version 3.10: *note Generic: 1619. can now be parameterized
over parameter expressions. See *note ParamSpec: 15e. and PEP 612(1)
for more details.
A user-defined generic class can have ABCs as base classes without a
metaclass conflict. Generic metaclasses are not supported. The outcome
of parameterizing generics is cached, and most types in the ‘typing’
module are *note hashable: 60c. and comparable for equality.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0612/
File: python.info, Node: The Any type, Next: Nominal vs structural subtyping, Prev: User-defined generic types, Up: typing — Support for type hints
5.27.1.10 The ‘Any’ type
........................
A special kind of type is *note Any: 6a8. A static type checker will
treat every type as being compatible with *note Any: 6a8. and *note Any:
6a8. as being compatible with every type.
This means that it is possible to perform any operation or method call
on a value of type *note Any: 6a8. and assign it to any variable:
from typing import Any
a: Any = None
a = [] # OK
a = 2 # OK
s: str = ''
s = a # OK
def foo(item: Any) -> int:
# Passes type checking; 'item' could be any type,
# and that type might have a 'bar' method
item.bar()
...
Notice that no type checking is performed when assigning a value of type
*note Any: 6a8. to a more precise type. For example, the static type
checker did not report an error when assigning ‘a’ to ‘s’ even though
‘s’ was declared to be of type *note str: 447. and receives an *note
int: 259. value at runtime!
Furthermore, all functions without a return type or parameter types will
implicitly default to using *note Any: 6a8.:
def legacy_parser(text):
...
return data
# A static type checker will treat the above
# as having the same signature as:
def legacy_parser(text: Any) -> Any:
...
return data
This behavior allows *note Any: 6a8. to be used as an 'escape hatch'
when you need to mix dynamically and statically typed code.
Contrast the behavior of *note Any: 6a8. with the behavior of *note
object: a8c. Similar to *note Any: 6a8, every type is a subtype of
*note object: a8c. However, unlike *note Any: 6a8, the reverse is not
true: *note object: a8c. is 'not' a subtype of every other type.
That means when the type of a value is *note object: a8c, a type checker
will reject almost all operations on it, and assigning it to a variable
(or using it as a return value) of a more specialized type is a type
error. For example:
def hash_a(item: object) -> int:
# Fails type checking; an object does not have a 'magic' method.
item.magic()
...
def hash_b(item: Any) -> int:
# Passes type checking
item.magic()
...
# Passes type checking, since ints and strs are subclasses of object
hash_a(42)
hash_a("foo")
# Passes type checking, since Any is compatible with all types
hash_b(42)
hash_b("foo")
Use *note object: a8c. to indicate that a value could be any type in a
typesafe manner. Use *note Any: 6a8. to indicate that a value is
dynamically typed.
File: python.info, Node: Nominal vs structural subtyping, Next: Module contents<3>, Prev: The Any type, Up: typing — Support for type hints
5.27.1.11 Nominal vs structural subtyping
.........................................
Initially PEP 484(1) defined the Python static type system as using
'nominal subtyping'. This means that a class ‘A’ is allowed where a
class ‘B’ is expected if and only if ‘A’ is a subclass of ‘B’.
This requirement previously also applied to abstract base classes, such
as *note Iterable: 224d. The problem with this approach is that a class
had to be explicitly marked to support them, which is unpythonic and
unlike what one would normally do in idiomatic dynamically typed Python
code. For example, this conforms to PEP 484(2):
from collections.abc import Sized, Iterable, Iterator
class Bucket(Sized, Iterable[int]):
...
def __len__(self) -> int: ...
def __iter__(self) -> Iterator[int]: ...
PEP 544(3) allows to solve this problem by allowing users to write the
above code without explicit base classes in the class definition,
allowing ‘Bucket’ to be implicitly considered a subtype of both ‘Sized’
and ‘Iterable[int]’ by static type checkers. This is known as
'structural subtyping' (or static duck-typing):
from collections.abc import Iterator, Iterable
class Bucket: # Note: no base classes
...
def __len__(self) -> int: ...
def __iter__(self) -> Iterator[int]: ...
def collect(items: Iterable[int]) -> int: ...
result = collect(Bucket()) # Passes type check
Moreover, by subclassing a special class *note Protocol: 266, a user can
define new custom protocols to fully enjoy structural subtyping (see
examples below).
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0484/
(3) https://peps.python.org/pep-0544/
File: python.info, Node: Module contents<3>, Next: Deprecation Timeline of Major Features, Prev: Nominal vs structural subtyping, Up: typing — Support for type hints
5.27.1.12 Module contents
.........................
The ‘typing’ module defines the following classes, functions and
decorators.
* Menu:
* Special typing primitives::
* Protocols: Protocols<3>.
* ABCs for working with IO::
* Functions and decorators::
* Introspection helpers::
* Constant::
* Deprecated aliases::
File: python.info, Node: Special typing primitives, Next: Protocols<3>, Up: Module contents<3>
5.27.1.13 Special typing primitives
...................................
* Menu:
* Special types::
* Special forms::
* Building generic types and type aliases::
* Other special directives::
File: python.info, Node: Special types, Next: Special forms, Up: Special typing primitives
5.27.1.14 Special types
.......................
These can be used as types in annotations. They do not support
subscription using ‘[]’.
-- Data: typing.Any
Special type indicating an unconstrained type.
* Every type is compatible with *note Any: 6a8.
* *note Any: 6a8. is compatible with every type.
Changed in version 3.11: *note Any: 6a8. can now be used as a base
class. This can be useful for avoiding type checker errors with
classes that can duck type anywhere or are highly dynamic.
-- Data: typing.AnyStr
A *note constrained type variable: 3f88.
Definition:
AnyStr = TypeVar('AnyStr', str, bytes)
‘AnyStr’ is meant to be used for functions that may accept *note
str: 447. or *note bytes: 1c2. arguments but cannot allow the two
to mix.
For example:
def concat(a: AnyStr, b: AnyStr) -> AnyStr:
return a + b
concat("foo", "bar") # OK, output has type 'str'
concat(b"foo", b"bar") # OK, output has type 'bytes'
concat("foo", b"bar") # Error, cannot mix str and bytes
Note that, despite its name, ‘AnyStr’ has nothing to do with the
*note Any: 6a8. type, nor does it mean “any string”. In
particular, ‘AnyStr’ and ‘str | bytes’ are different from each
other and have different use cases:
# Invalid use of AnyStr:
# The type variable is used only once in the function signature,
# so cannot be "solved" by the type checker
def greet_bad(cond: bool) -> AnyStr:
return "hi there!" if cond else b"greetings!"
# The better way of annotating this function:
def greet_proper(cond: bool) -> str | bytes:
return "hi there!" if cond else b"greetings!"
Deprecated since version 3.13, will be removed in version 3.18:
Deprecated in favor of the new *note type parameter syntax: 2b5.
Use ‘class A[T: (str, bytes)]: ...’ instead of importing ‘AnyStr’.
See PEP 695(1) for more details.
In Python 3.16, ‘AnyStr’ will be removed from ‘typing.__all__’, and
deprecation warnings will be emitted at runtime when it is accessed
or imported from ‘typing’. ‘AnyStr’ will be removed from ‘typing’
in Python 3.18.
-- Data: typing.LiteralString
Special type that includes only literal strings.
Any string literal is compatible with ‘LiteralString’, as is
another ‘LiteralString’. However, an object typed as just ‘str’ is
not. A string created by composing ‘LiteralString’-typed objects
is also acceptable as a ‘LiteralString’.
Example:
def run_query(sql: LiteralString) -> None:
...
def caller(arbitrary_string: str, literal_string: LiteralString) -> None:
run_query("SELECT * FROM students") # OK
run_query(literal_string) # OK
run_query("SELECT * FROM " + literal_string) # OK
run_query(arbitrary_string) # type checker error
run_query( # type checker error
f"SELECT * FROM students WHERE name = {arbitrary_string}"
)
‘LiteralString’ is useful for sensitive APIs where arbitrary
user-generated strings could generate problems. For example, the
two cases above that generate type checker errors could be
vulnerable to an SQL injection attack.
See PEP 675(2) for more details.
Added in version 3.11.
-- Data: typing.Never
-- Data: typing.NoReturn
‘Never’ and ‘NoReturn’ represent the bottom type(3), a type that
has no members.
They can be used to indicate that a function never returns, such as
*note sys.exit(): 133c.:
from typing import Never # or NoReturn
def stop() -> Never:
raise RuntimeError('no way')
Or to define a function that should never be called, as there are
no valid arguments, such as *note assert_never(): 6a3.:
from typing import Never # or NoReturn
def never_call_me(arg: Never) -> None:
pass
def int_or_str(arg: int | str) -> None:
never_call_me(arg) # type checker error
match arg:
case int():
print("It's an int")
case str():
print("It's a str")
case _:
never_call_me(arg) # OK, arg is of type Never (or NoReturn)
‘Never’ and ‘NoReturn’ have the same meaning in the type system and
static type checkers treat both equivalently.
Added in version 3.6.2: Added *note NoReturn: 3f89.
Added in version 3.11: Added *note Never: 6a4.
-- Data: typing.Self
Special type to represent the current enclosed class.
For example:
from typing import Self, reveal_type
class Foo:
def return_self(self) -> Self:
...
return self
class SubclassOfFoo(Foo): pass
reveal_type(Foo().return_self()) # Revealed type is "Foo"
reveal_type(SubclassOfFoo().return_self()) # Revealed type is "SubclassOfFoo"
This annotation is semantically equivalent to the following, albeit
in a more succinct fashion:
from typing import TypeVar
Self = TypeVar("Self", bound="Foo")
class Foo:
def return_self(self: Self) -> Self:
...
return self
In general, if something returns ‘self’, as in the above examples,
you should use ‘Self’ as the return annotation. If
‘Foo.return_self’ was annotated as returning ‘"Foo"’, then the type
checker would infer the object returned from
‘SubclassOfFoo.return_self’ as being of type ‘Foo’ rather than
‘SubclassOfFoo’.
Other common use cases include:
- *note classmethod: 166.s that are used as alternative
constructors and return instances of the ‘cls’ parameter.
- Annotating an *note __enter__(): 5c4. method which returns
self.
You should not use ‘Self’ as the return annotation if the method is
not guaranteed to return an instance of a subclass when the class
is subclassed:
class Eggs:
# Self would be an incorrect return annotation here,
# as the object returned is always an instance of Eggs,
# even in subclasses
def returns_eggs(self) -> "Eggs":
return Eggs()
See PEP 673(4) for more details.
Added in version 3.11.
-- Data: typing.TypeAlias
Special annotation for explicitly declaring a *note type alias:
44f.
For example:
from typing import TypeAlias
Factors: TypeAlias = list[int]
‘TypeAlias’ is particularly useful on older Python versions for
annotating aliases that make use of forward references, as it can
be hard for type checkers to distinguish these from normal variable
assignments:
from typing import Generic, TypeAlias, TypeVar
T = TypeVar("T")
# "Box" does not exist yet,
# so we have to use quotes for the forward reference on Python <3.12.
# Using ``TypeAlias`` tells the type checker that this is a type alias declaration,
# not a variable assignment to a string.
BoxOfStrings: TypeAlias = "Box[str]"
class Box(Generic[T]):
@classmethod
def make_box_of_strings(cls) -> BoxOfStrings: ...
See PEP 613(5) for more details.
Added in version 3.10.
Deprecated since version 3.12: *note TypeAlias: 7c4. is deprecated
in favor of the *note type: 433. statement, which creates instances
of *note TypeAliasType: 450. and which natively supports forward
references. Note that while *note TypeAlias: 7c4. and *note
TypeAliasType: 450. serve similar purposes and have similar names,
they are distinct and the latter is not the type of the former.
Removal of *note TypeAlias: 7c4. is not currently planned, but
users are encouraged to migrate to *note type: 433. statements.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0695/
(2) https://peps.python.org/pep-0675/
(3) https://en.wikipedia.org/wiki/Bottom_type
(4) https://peps.python.org/pep-0673/
(5) https://peps.python.org/pep-0613/
File: python.info, Node: Special forms, Next: Building generic types and type aliases, Prev: Special types, Up: Special typing primitives
5.27.1.15 Special forms
.......................
These can be used as types in annotations. They all support
subscription using ‘[]’, but each has a unique syntax.
-- Data: typing.Union
Union type; ‘Union[X, Y]’ is equivalent to ‘X | Y’ and means either
X or Y.
To define a union, use e.g. ‘Union[int, str]’ or the shorthand
‘int | str’. Using that shorthand is recommended. Details:
* The arguments must be types and there must be at least one.
* Unions of unions are flattened, e.g.:
Union[Union[int, str], float] == Union[int, str, float]
However, this does not apply to unions referenced through a
type alias, to avoid forcing evaluation of the underlying
*note TypeAliasType: 450.:
type A = Union[int, str]
Union[A, float] != Union[int, str, float]
* Unions of a single argument vanish, e.g.:
Union[int] == int # The constructor actually returns int
* Redundant arguments are skipped, e.g.:
Union[int, str, int] == Union[int, str] == int | str
* When comparing unions, the argument order is ignored, e.g.:
Union[int, str] == Union[str, int]
* You cannot subclass or instantiate a ‘Union’.
* You cannot write ‘Union[X][Y]’.
Changed in version 3.7: Don’t remove explicit subclasses from
unions at runtime.
Changed in version 3.10: Unions can now be written as ‘X | Y’. See
*note union type expressions: 7bd.
-- Data: typing.Optional
‘Optional[X]’ is equivalent to ‘X | None’ (or ‘Union[X, None]’).
Note that this is not the same concept as an optional argument,
which is one that has a default. An optional argument with a
default does not require the ‘Optional’ qualifier on its type
annotation just because it is optional. For example:
def foo(arg: int = 0) -> None:
...
On the other hand, if an explicit value of ‘None’ is allowed, the
use of ‘Optional’ is appropriate, whether the argument is optional
or not. For example:
def foo(arg: Optional[int] = None) -> None:
...
Changed in version 3.10: Optional can now be written as ‘X | None’.
See *note union type expressions: 7bd.
-- Data: typing.Concatenate
Special form for annotating higher-order functions.
‘Concatenate’ can be used in conjunction with *note Callable: 2533.
and *note ParamSpec: 15e. to annotate a higher-order callable which
adds, removes, or transforms parameters of another callable. Usage
is in the form ‘Concatenate[Arg1Type, Arg2Type, ...,
ParamSpecVariable]’. ‘Concatenate’ is currently only valid when
used as the first argument to a *note Callable: 2533. The last
parameter to ‘Concatenate’ must be a *note ParamSpec: 15e. or
ellipsis (‘...’).
For example, to annotate a decorator ‘with_lock’ which provides a
*note threading.Lock: 1677. to the decorated function,
‘Concatenate’ can be used to indicate that ‘with_lock’ expects a
callable which takes in a ‘Lock’ as the first argument, and returns
a callable with a different type signature. In this case, the
*note ParamSpec: 15e. indicates that the returned callable’s
parameter types are dependent on the parameter types of the
callable being passed in:
from collections.abc import Callable
from threading import Lock
from typing import Concatenate
# Use this lock to ensure that only one thread is executing a function
# at any time.
my_lock = Lock()
def with_lock[**P, R](f: Callable[Concatenate[Lock, P], R]) -> Callable[P, R]:
'''A type-safe decorator which provides a lock.'''
def inner(*args: P.args, **kwargs: P.kwargs) -> R:
# Provide the lock as the first argument.
return f(my_lock, *args, **kwargs)
return inner
@with_lock
def sum_threadsafe(lock: Lock, numbers: list[float]) -> float:
'''Add a list of numbers together in a thread-safe manner.'''
with lock:
return sum(numbers)
# We don't need to pass in the lock ourselves thanks to the decorator.
sum_threadsafe([1.1, 2.2, 3.3])
Added in version 3.10.
See also
........
* PEP 612(1) – Parameter Specification Variables (the PEP which
introduced ‘ParamSpec’ and ‘Concatenate’)
* *note ParamSpec: 15e.
* *note Annotating callable objects: 2533.
-- Data: typing.Literal
Special typing form to define “literal types”.
‘Literal’ can be used to indicate to type checkers that the
annotated object has a value equivalent to one of the provided
literals.
For example:
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
type Mode = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: Mode) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
‘Literal[...]’ cannot be subclassed. At runtime, an arbitrary
value is allowed as type argument to ‘Literal[...]’, but type
checkers may impose restrictions. See PEP 586(2) for more details
about literal types.
Additional details:
* The arguments must be literal values and there must be at
least one.
* Nested ‘Literal’ types are flattened, e.g.:
assert Literal[Literal[1, 2], 3] == Literal[1, 2, 3]
However, this does not apply to ‘Literal’ types referenced
through a type alias, to avoid forcing evaluation of the
underlying *note TypeAliasType: 450.:
type A = Literal[1, 2]
assert Literal[A, 3] != Literal[1, 2, 3]
* Redundant arguments are skipped, e.g.:
assert Literal[1, 2, 1] == Literal[1, 2]
* When comparing literals, the argument order is ignored, e.g.:
assert Literal[1, 2] == Literal[2, 1]
* You cannot subclass or instantiate a ‘Literal’.
* You cannot write ‘Literal[X][Y]’.
Added in version 3.8.
Changed in version 3.9.1: ‘Literal’ now de-duplicates parameters.
Equality comparisons of ‘Literal’ objects are no longer order
dependent. ‘Literal’ objects will now raise a *note TypeError:
534. exception during equality comparisons if one of their
parameters are not *note hashable: 60c.
-- Data: typing.ClassVar
Special type construct to mark class variables.
As introduced in PEP 526(3), a variable annotation wrapped in
ClassVar indicates that a given attribute is intended to be used as
a class variable and should not be set on instances of that class.
Usage:
class Starship:
stats: ClassVar[dict[str, int]] = {} # class variable
damage: int = 10 # instance variable
*note ClassVar: 268. accepts only types and cannot be further
subscribed.
*note ClassVar: 268. is not a class itself, and should not be used
with *note isinstance(): 43d. or *note issubclass(): 7bc. *note
ClassVar: 268. does not change Python runtime behavior, but it can
be used by third-party type checkers. For example, a type checker
might flag the following code as an error:
enterprise_d = Starship(3000)
enterprise_d.stats = {} # Error, setting class variable on instance
Starship.stats = {} # This is OK
Added in version 3.5.3.
Changed in version 3.13: *note ClassVar: 268. can now be nested in
*note Final: 269. and vice versa.
-- Data: typing.Final
Special typing construct to indicate final names to type checkers.
Final names cannot be reassigned in any scope. Final names
declared in class scopes cannot be overridden in subclasses.
For example:
MAX_SIZE: Final = 9000
MAX_SIZE += 1 # Error reported by type checker
class Connection:
TIMEOUT: Final[int] = 10
class FastConnector(Connection):
TIMEOUT = 1 # Error reported by type checker
There is no runtime checking of these properties. See PEP 591(4)
for more details.
Added in version 3.8.
Changed in version 3.13: *note Final: 269. can now be nested in
*note ClassVar: 268. and vice versa.
-- Data: typing.Required
Special typing construct to mark a *note TypedDict: 162. key as
required.
This is mainly useful for ‘total=False’ TypedDicts. See *note
TypedDict: 162. and PEP 655(5) for more details.
Added in version 3.11.
-- Data: typing.NotRequired
Special typing construct to mark a *note TypedDict: 162. key as
potentially missing.
See *note TypedDict: 162. and PEP 655(6) for more details.
Added in version 3.11.
-- Data: typing.ReadOnly
A special typing construct to mark an item of a *note TypedDict:
162. as read-only.
For example:
class Movie(TypedDict):
title: ReadOnly[str]
year: int
def mutate_movie(m: Movie) -> None:
m["year"] = 1999 # allowed
m["title"] = "The Matrix" # typechecker error
There is no runtime checking for this property.
See *note TypedDict: 162. and PEP 705(7) for more details.
Added in version 3.13.
-- Data: typing.Annotated
Special typing form to add context-specific metadata to an
annotation.
Add metadata ‘x’ to a given type ‘T’ by using the annotation
‘Annotated[T, x]’. Metadata added using ‘Annotated’ can be used by
static analysis tools or at runtime. At runtime, the metadata is
stored in a ‘__metadata__’ attribute.
If a library or tool encounters an annotation ‘Annotated[T, x]’ and
has no special logic for the metadata, it should ignore the
metadata and simply treat the annotation as ‘T’. As such,
‘Annotated’ can be useful for code that wants to use annotations
for purposes outside Python’s static typing system.
Using ‘Annotated[T, x]’ as an annotation still allows for static
typechecking of ‘T’, as type checkers will simply ignore the
metadata ‘x’. In this way, ‘Annotated’ differs from the *note
@no_type_check: 6b0. decorator, which can also be used for adding
annotations outside the scope of the typing system, but completely
disables typechecking for a function or class.
The responsibility of how to interpret the metadata lies with the
tool or library encountering an ‘Annotated’ annotation. A tool or
library encountering an ‘Annotated’ type can scan through the
metadata elements to determine if they are of interest (e.g., using
*note isinstance(): 43d.).
-- Describe: Annotated[, ]
Here is an example of how you might use ‘Annotated’ to add metadata
to type annotations if you were doing range analysis:
@dataclass
class ValueRange:
lo: int
hi: int
T1 = Annotated[int, ValueRange(-10, 5)]
T2 = Annotated[T1, ValueRange(-20, 3)]
The first argument to ‘Annotated’ must be a valid type. Multiple
metadata elements can be supplied as ‘Annotated’ supports variadic
arguments. The order of the metadata elements is preserved and
matters for equality checks:
@dataclass
class ctype:
kind: str
a1 = Annotated[int, ValueRange(3, 10), ctype("char")]
a2 = Annotated[int, ctype("char"), ValueRange(3, 10)]
assert a1 != a2 # Order matters
It is up to the tool consuming the annotations to decide whether
the client is allowed to add multiple metadata elements to one
annotation and how to merge those annotations.
Nested ‘Annotated’ types are flattened. The order of the metadata
elements starts with the innermost annotation:
assert Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] == Annotated[
int, ValueRange(3, 10), ctype("char")
]
However, this does not apply to ‘Annotated’ types referenced
through a type alias, to avoid forcing evaluation of the underlying
*note TypeAliasType: 450.:
type From3To10[T] = Annotated[T, ValueRange(3, 10)]
assert Annotated[From3To10[int], ctype("char")] != Annotated[
int, ValueRange(3, 10), ctype("char")
]
Duplicated metadata elements are not removed:
assert Annotated[int, ValueRange(3, 10)] != Annotated[
int, ValueRange(3, 10), ValueRange(3, 10)
]
‘Annotated’ can be used with nested and generic aliases:
@dataclass
class MaxLen:
value: int
type Vec[T] = Annotated[list[tuple[T, T]], MaxLen(10)]
# When used in a type annotation, a type checker will treat "V" the same as
# ``Annotated[list[tuple[int, int]], MaxLen(10)]``:
type V = Vec[int]
‘Annotated’ cannot be used with an unpacked *note TypeVarTuple:
15f.:
type Variadic[*Ts] = Annotated[*Ts, Ann1] = Annotated[T1, T2, T3, ..., Ann1] # NOT valid
where ‘T1’, ‘T2’, … are *note TypeVars: 15d. This is invalid as
only one type should be passed to Annotated.
By default, *note get_type_hints(): 6ad. strips the metadata from
annotations. Pass ‘include_extras=True’ to have the metadata
preserved:
>>> from typing import Annotated, get_type_hints
>>> def func(x: Annotated[int, "metadata"]) -> None: pass
...
>>> get_type_hints(func)
{'x': , 'return': }
>>> get_type_hints(func, include_extras=True)
{'x': typing.Annotated[int, 'metadata'], 'return': }
At runtime, the metadata associated with an ‘Annotated’ type can be
retrieved via the ‘__metadata__’ attribute:
>>> from typing import Annotated
>>> X = Annotated[int, "very", "important", "metadata"]
>>> X
typing.Annotated[int, 'very', 'important', 'metadata']
>>> X.__metadata__
('very', 'important', 'metadata')
If you want to retrieve the original type wrapped by ‘Annotated’,
use the ‘__origin__’ attribute:
>>> from typing import Annotated, get_origin
>>> Password = Annotated[str, "secret"]
>>> Password.__origin__
Note that using *note get_origin(): a46. will return ‘Annotated’
itself:
>>> get_origin(Password)
typing.Annotated
See also
........
PEP 593(8) - Flexible function and variable annotations
The PEP introducing ‘Annotated’ to the standard library.
Added in version 3.9.
-- Data: typing.TypeIs
Special typing construct for marking user-defined type predicate
functions.
‘TypeIs’ can be used to annotate the return type of a user-defined
type predicate function. ‘TypeIs’ only accepts a single type
argument. At runtime, functions marked this way should return a
boolean and take at least one positional argument.
‘TypeIs’ aims to benefit 'type narrowing' – a technique used by
static type checkers to determine a more precise type of an
expression within a program’s code flow. Usually type narrowing is
done by analyzing conditional code flow and applying the narrowing
to a block of code. The conditional expression here is sometimes
referred to as a “type predicate”:
def is_str(val: str | float):
# "isinstance" type predicate
if isinstance(val, str):
# Type of ``val`` is narrowed to ``str``
...
else:
# Else, type of ``val`` is narrowed to ``float``.
...
Sometimes it would be convenient to use a user-defined boolean
function as a type predicate. Such a function should use
‘TypeIs[...]’ or *note TypeGuard: 164. as its return type to alert
static type checkers to this intention. ‘TypeIs’ usually has more
intuitive behavior than ‘TypeGuard’, but it cannot be used when the
input and output types are incompatible (e.g., ‘list[object]’ to
‘list[int]’) or when the function does not return ‘True’ for all
instances of the narrowed type.
Using ‘-> TypeIs[NarrowedType]’ tells the static type checker that
for a given function:
1. The return value is a boolean.
2. If the return value is ‘True’, the type of its argument is the
intersection of the argument’s original type and
‘NarrowedType’.
3. If the return value is ‘False’, the type of its argument is
narrowed to exclude ‘NarrowedType’.
For example:
from typing import assert_type, final, TypeIs
class Parent: pass
class Child(Parent): pass
@final
class Unrelated: pass
def is_parent(val: object) -> TypeIs[Parent]:
return isinstance(val, Parent)
def run(arg: Child | Unrelated):
if is_parent(arg):
# Type of ``arg`` is narrowed to the intersection
# of ``Parent`` and ``Child``, which is equivalent to
# ``Child``.
assert_type(arg, Child)
else:
# Type of ``arg`` is narrowed to exclude ``Parent``,
# so only ``Unrelated`` is left.
assert_type(arg, Unrelated)
The type inside ‘TypeIs’ must be consistent with the type of the
function’s argument; if it is not, static type checkers will raise
an error. An incorrectly written ‘TypeIs’ function can lead to
unsound behavior in the type system; it is the user’s
responsibility to write such functions in a type-safe manner.
If a ‘TypeIs’ function is a class or instance method, then the type
in ‘TypeIs’ maps to the type of the second parameter (after ‘cls’
or ‘self’).
In short, the form ‘def foo(arg: TypeA) -> TypeIs[TypeB]: ...’,
means that if ‘foo(arg)’ returns ‘True’, then ‘arg’ is an instance
of ‘TypeB’, and if it returns ‘False’, it is not an instance of
‘TypeB’.
‘TypeIs’ also works with type variables. For more information, see
PEP 742(9) (Narrowing types with ‘TypeIs’).
Added in version 3.13.
-- Data: typing.TypeGuard
Special typing construct for marking user-defined type predicate
functions.
Type predicate functions are user-defined functions that return
whether their argument is an instance of a particular type.
‘TypeGuard’ works similarly to *note TypeIs: 163, but has subtly
different effects on type checking behavior (see below).
Using ‘-> TypeGuard’ tells the static type checker that for a given
function:
1. The return value is a boolean.
2. If the return value is ‘True’, the type of its argument is the
type inside ‘TypeGuard’.
‘TypeGuard’ also works with type variables. See PEP 647(10) for
more details.
For example:
def is_str_list(val: list[object]) -> TypeGuard[list[str]]:
'''Determines whether all objects in the list are strings'''
return all(isinstance(x, str) for x in val)
def func1(val: list[object]):
if is_str_list(val):
# Type of ``val`` is narrowed to ``list[str]``.
print(" ".join(val))
else:
# Type of ``val`` remains as ``list[object]``.
print("Not a list of strings!")
‘TypeIs’ and ‘TypeGuard’ differ in the following ways:
* ‘TypeIs’ requires the narrowed type to be a subtype of the
input type, while ‘TypeGuard’ does not. The main reason is to
allow for things like narrowing ‘list[object]’ to ‘list[str]’
even though the latter is not a subtype of the former, since
‘list’ is invariant.
* When a ‘TypeGuard’ function returns ‘True’, type checkers
narrow the type of the variable to exactly the ‘TypeGuard’
type. When a ‘TypeIs’ function returns ‘True’, type checkers
can infer a more precise type combining the previously known
type of the variable with the ‘TypeIs’ type. (Technically,
this is known as an intersection type.)
* When a ‘TypeGuard’ function returns ‘False’, type checkers
cannot narrow the type of the variable at all. When a
‘TypeIs’ function returns ‘False’, type checkers can narrow
the type of the variable to exclude the ‘TypeIs’ type.
Added in version 3.10.
-- Data: typing.Unpack
Typing operator to conceptually mark an object as having been
unpacked.
For example, using the unpack operator ‘*’ on a *note type variable
tuple: 3f8b. is equivalent to using ‘Unpack’ to mark the type
variable tuple as having been unpacked:
Ts = TypeVarTuple('Ts')
tup: tuple[*Ts]
# Effectively does:
tup: tuple[Unpack[Ts]]
In fact, ‘Unpack’ can be used interchangeably with ‘*’ in the
context of *note typing.TypeVarTuple: 15f. and *note
builtins.tuple: 36b. types. You might see ‘Unpack’ being used
explicitly in older versions of Python, where ‘*’ couldn’t be used
in certain places:
# In older versions of Python, TypeVarTuple and Unpack
# are located in the `typing_extensions` backports package.
from typing_extensions import TypeVarTuple, Unpack
Ts = TypeVarTuple('Ts')
tup: tuple[*Ts] # Syntax error on Python <= 3.10!
tup: tuple[Unpack[Ts]] # Semantically equivalent, and backwards-compatible
‘Unpack’ can also be used along with *note typing.TypedDict: 162.
for typing ‘**kwargs’ in a function signature:
from typing import TypedDict, Unpack
class Movie(TypedDict):
name: str
year: int
# This function expects two keyword arguments - `name` of type `str`
# and `year` of type `int`.
def foo(**kwargs: Unpack[Movie]): ...
See PEP 692(11) for more details on using ‘Unpack’ for ‘**kwargs’
typing.
Added in version 3.11.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0612/
(2) https://peps.python.org/pep-0586/
(3) https://peps.python.org/pep-0526/
(4) https://peps.python.org/pep-0591/
(5) https://peps.python.org/pep-0655/
(6) https://peps.python.org/pep-0655/
(7) https://peps.python.org/pep-0705/
(8) https://peps.python.org/pep-0593/
(9) https://peps.python.org/pep-0742/
(10) https://peps.python.org/pep-0647/
(11) https://peps.python.org/pep-0692/
File: python.info, Node: Building generic types and type aliases, Next: Other special directives, Prev: Special forms, Up: Special typing primitives
5.27.1.16 Building generic types and type aliases
.................................................
The following classes should not be used directly as annotations. Their
intended purpose is to be building blocks for creating generic types and
type aliases.
These objects can be created through special syntax (*note type
parameter lists: 2b5. and the *note type: 433. statement). For
compatibility with Python 3.11 and earlier, they can also be created
without the dedicated syntax, as documented below.
-- Class: typing.Generic
Abstract base class for generic types.
A generic type is typically declared by adding a list of type
parameters after the class name:
class Mapping[KT, VT]:
def __getitem__(self, key: KT) -> VT:
...
# Etc.
Such a class implicitly inherits from ‘Generic’. The runtime
semantics of this syntax are discussed in the *note Language
Reference: 44d.
This class can then be used as follows:
def lookup_name[X, Y](mapping: Mapping[X, Y], key: X, default: Y) -> Y:
try:
return mapping[key]
except KeyError:
return default
Here the brackets after the function name indicate a *note generic
function: 44e.
For backwards compatibility, generic classes can also be declared
by explicitly inheriting from ‘Generic’. In this case, the type
parameters must be declared separately:
KT = TypeVar('KT')
VT = TypeVar('VT')
class Mapping(Generic[KT, VT]):
def __getitem__(self, key: KT) -> VT:
...
# Etc.
-- Class: typing.TypeVar (name, *constraints, bound=None,
covariant=False, contravariant=False, infer_variance=False,
default=typing.NoDefault)
Type variable.
The preferred way to construct a type variable is via the dedicated
syntax for *note generic functions: 44e, *note generic classes:
44d, and *note generic type aliases: 451.:
class Sequence[T]: # T is a TypeVar
...
This syntax can also be used to create bounded and constrained type
variables:
class StrSequence[S: str]: # S is a TypeVar with a `str` upper bound;
... # we can say that S is "bounded by `str`"
class StrOrBytesSequence[A: (str, bytes)]: # A is a TypeVar constrained to str or bytes
...
However, if desired, reusable type variables can also be
constructed manually, like so:
T = TypeVar('T') # Can be anything
S = TypeVar('S', bound=str) # Can be any subtype of str
A = TypeVar('A', str, bytes) # Must be exactly str or bytes
Type variables exist primarily for the benefit of static type
checkers. They serve as the parameters for generic types as well
as for generic function and type alias definitions. See *note
Generic: 1619. for more information on generic types. Generic
functions work as follows:
def repeat[T](x: T, n: int) -> Sequence[T]:
"""Return a list containing n references to x."""
return [x]*n
def print_capitalized[S: str](x: S) -> S:
"""Print x capitalized, and return x."""
print(x.capitalize())
return x
def concatenate[A: (str, bytes)](x: A, y: A) -> A:
"""Add two strings or bytes objects together."""
return x + y
Note that type variables can be 'bounded', 'constrained', or
neither, but cannot be both bounded 'and' constrained.
The variance of type variables is inferred by type checkers when
they are created through the *note type parameter syntax: 2b5. or
when ‘infer_variance=True’ is passed. Manually created type
variables may be explicitly marked covariant or contravariant by
passing ‘covariant=True’ or ‘contravariant=True’. By default,
manually created type variables are invariant. See PEP 484(1) and
PEP 695(2) for more details.
Bounded type variables and constrained type variables have
different semantics in several important ways. Using a 'bounded'
type variable means that the ‘TypeVar’ will be solved using the
most specific type possible:
x = print_capitalized('a string')
reveal_type(x) # revealed type is str
class StringSubclass(str):
pass
y = print_capitalized(StringSubclass('another string'))
reveal_type(y) # revealed type is StringSubclass
z = print_capitalized(45) # error: int is not a subtype of str
The upper bound of a type variable can be a concrete type, abstract
type (ABC or Protocol), or even a union of types:
# Can be anything with an __abs__ method
def print_abs[T: SupportsAbs](arg: T) -> None:
print("Absolute value:", abs(arg))
U = TypeVar('U', bound=str|bytes) # Can be any subtype of the union str|bytes
V = TypeVar('V', bound=SupportsAbs) # Can be anything with an __abs__ method
Using a 'constrained' type variable, however, means that the
‘TypeVar’ can only ever be solved as being exactly one of the
constraints given:
a = concatenate('one', 'two')
reveal_type(a) # revealed type is str
b = concatenate(StringSubclass('one'), StringSubclass('two'))
reveal_type(b) # revealed type is str, despite StringSubclass being passed in
c = concatenate('one', b'two') # error: type variable 'A' can be either str or bytes in a function call, but not both
At runtime, ‘isinstance(x, T)’ will raise *note TypeError: 534.
-- Attribute: __name__
The name of the type variable.
-- Attribute: __covariant__
Whether the type var has been explicitly marked as covariant.
-- Attribute: __contravariant__
Whether the type var has been explicitly marked as
contravariant.
-- Attribute: __infer_variance__
Whether the type variable’s variance should be inferred by
type checkers.
Added in version 3.12.
-- Attribute: __bound__
The upper bound of the type variable, if any.
Changed in version 3.12: For type variables created through
*note type parameter syntax: 2b5, the bound is evaluated only
when the attribute is accessed, not when the type variable is
created (see *note Lazy evaluation: 452.).
-- Attribute: __constraints__
A tuple containing the constraints of the type variable, if
any.
Changed in version 3.12: For type variables created through
*note type parameter syntax: 2b5, the constraints are
evaluated only when the attribute is accessed, not when the
type variable is created (see *note Lazy evaluation: 452.).
-- Attribute: __default__
The default value of the type variable, or *note
typing.NoDefault: 264. if it has no default.
Added in version 3.13.
-- Method: has_default ()
Return whether or not the type variable has a default value.
This is equivalent to checking whether *note __default__:
3f93. is not the *note typing.NoDefault: 264. singleton,
except that it does not force evaluation of the *note lazily
evaluated: 452. default value.
Added in version 3.13.
Changed in version 3.12: Type variables can now be declared using
the *note type parameter: 2b5. syntax introduced by PEP 695(3).
The ‘infer_variance’ parameter was added.
Changed in version 3.13: Support for default values was added.
-- Class: typing.TypeVarTuple (name, *, default=typing.NoDefault)
Type variable tuple. A specialized form of *note type variable:
3f82. that enables 'variadic' generics.
Type variable tuples can be declared in *note type parameter lists:
2b5. using a single asterisk (‘*’) before the name:
def move_first_element_to_last[T, *Ts](tup: tuple[T, *Ts]) -> tuple[*Ts, T]:
return (*tup[1:], tup[0])
Or by explicitly invoking the ‘TypeVarTuple’ constructor:
T = TypeVar("T")
Ts = TypeVarTuple("Ts")
def move_first_element_to_last(tup: tuple[T, *Ts]) -> tuple[*Ts, T]:
return (*tup[1:], tup[0])
A normal type variable enables parameterization with a single type.
A type variable tuple, in contrast, allows parameterization with an
'arbitrary' number of types by acting like an 'arbitrary' number of
type variables wrapped in a tuple. For example:
# T is bound to int, Ts is bound to ()
# Return value is (1,), which has type tuple[int]
move_first_element_to_last(tup=(1,))
# T is bound to int, Ts is bound to (str,)
# Return value is ('spam', 1), which has type tuple[str, int]
move_first_element_to_last(tup=(1, 'spam'))
# T is bound to int, Ts is bound to (str, float)
# Return value is ('spam', 3.0, 1), which has type tuple[str, float, int]
move_first_element_to_last(tup=(1, 'spam', 3.0))
# This fails to type check (and fails at runtime)
# because tuple[()] is not compatible with tuple[T, *Ts]
# (at least one element is required)
move_first_element_to_last(tup=())
Note the use of the unpacking operator ‘*’ in ‘tuple[T, *Ts]’.
Conceptually, you can think of ‘Ts’ as a tuple of type variables
‘(T1, T2, ...)’. ‘tuple[T, *Ts]’ would then become ‘tuple[T, *(T1,
T2, ...)]’, which is equivalent to ‘tuple[T, T1, T2, ...]’. (Note
that in older versions of Python, you might see this written using
*note Unpack: 162f. instead, as ‘Unpack[Ts]’.)
Type variable tuples must 'always' be unpacked. This helps
distinguish type variable tuples from normal type variables:
x: Ts # Not valid
x: tuple[Ts] # Not valid
x: tuple[*Ts] # The correct way to do it
Type variable tuples can be used in the same contexts as normal
type variables. For example, in class definitions, arguments, and
return types:
class Array[*Shape]:
def __getitem__(self, key: tuple[*Shape]) -> float: ...
def __abs__(self) -> "Array[*Shape]": ...
def get_shape(self) -> tuple[*Shape]: ...
Type variable tuples can be happily combined with normal type
variables:
class Array[DType, *Shape]: # This is fine
pass
class Array2[*Shape, DType]: # This would also be fine
pass
class Height: ...
class Width: ...
float_array_1d: Array[float, Height] = Array() # Totally fine
int_array_2d: Array[int, Height, Width] = Array() # Yup, fine too
However, note that at most one type variable tuple may appear in a
single list of type arguments or type parameters:
x: tuple[*Ts, *Ts] # Not valid
class Array[*Shape, *Shape]: # Not valid
pass
Finally, an unpacked type variable tuple can be used as the type
annotation of ‘*args’:
def call_soon[*Ts](
callback: Callable[[*Ts], None],
*args: *Ts
) -> None:
...
callback(*args)
In contrast to non-unpacked annotations of ‘*args’ - e.g. ‘*args:
int’, which would specify that 'all' arguments are ‘int’ - ‘*args:
*Ts’ enables reference to the types of the 'individual' arguments
in ‘*args’. Here, this allows us to ensure the types of the
‘*args’ passed to ‘call_soon’ match the types of the (positional)
arguments of ‘callback’.
See PEP 646(4) for more details on type variable tuples.
-- Attribute: __name__
The name of the type variable tuple.
-- Attribute: __default__
The default value of the type variable tuple, or *note
typing.NoDefault: 264. if it has no default.
Added in version 3.13.
-- Method: has_default ()
Return whether or not the type variable tuple has a default
value. This is equivalent to checking whether *note
__default__: 3f96. is not the *note typing.NoDefault: 264.
singleton, except that it does not force evaluation of the
*note lazily evaluated: 452. default value.
Added in version 3.13.
Added in version 3.11.
Changed in version 3.12: Type variable tuples can now be declared
using the *note type parameter: 2b5. syntax introduced by PEP
695(5).
Changed in version 3.13: Support for default values was added.
-- Class: typing.ParamSpec (name, *, bound=None, covariant=False,
contravariant=False, default=typing.NoDefault)
Parameter specification variable. A specialized version of *note
type variables: 3f82.
In *note type parameter lists: 2b5, parameter specifications can be
declared with two asterisks (‘**’):
type IntFunc[**P] = Callable[P, int]
For compatibility with Python 3.11 and earlier, ‘ParamSpec’ objects
can also be created as follows:
P = ParamSpec('P')
Parameter specification variables exist primarily for the benefit
of static type checkers. They are used to forward the parameter
types of one callable to another callable – a pattern commonly
found in higher order functions and decorators. They are only
valid when used in ‘Concatenate’, or as the first argument to
‘Callable’, or as parameters for user-defined Generics. See *note
Generic: 1619. for more information on generic types.
For example, to add basic logging to a function, one can create a
decorator ‘add_logging’ to log function calls. The parameter
specification variable tells the type checker that the callable
passed into the decorator and the new callable returned by it have
inter-dependent type parameters:
from collections.abc import Callable
import logging
def add_logging[T, **P](f: Callable[P, T]) -> Callable[P, T]:
'''A type-safe decorator to add logging to a function.'''
def inner(*args: P.args, **kwargs: P.kwargs) -> T:
logging.info(f'{f.__name__} was called')
return f(*args, **kwargs)
return inner
@add_logging
def add_two(x: float, y: float) -> float:
'''Add two numbers together.'''
return x + y
Without ‘ParamSpec’, the simplest way to annotate this previously
was to use a *note TypeVar: 15d. with upper bound ‘Callable[...,
Any]’. However this causes two problems:
1. The type checker can’t type check the ‘inner’ function because
‘*args’ and ‘**kwargs’ have to be typed *note Any: 6a8.
2. *note cast(): 3f98. may be required in the body of the
‘add_logging’ decorator when returning the ‘inner’ function,
or the static type checker must be told to ignore the ‘return
inner’.
-- Attribute: args
-- Attribute: kwargs
Since ‘ParamSpec’ captures both positional and keyword
parameters, ‘P.args’ and ‘P.kwargs’ can be used to split a
‘ParamSpec’ into its components. ‘P.args’ represents the
tuple of positional parameters in a given call and should only
be used to annotate ‘*args’. ‘P.kwargs’ represents the
mapping of keyword parameters to their values in a given call,
and should be only be used to annotate ‘**kwargs’. Both
attributes require the annotated parameter to be in scope. At
runtime, ‘P.args’ and ‘P.kwargs’ are instances respectively of
*note ParamSpecArgs: 7c1. and *note ParamSpecKwargs: 7c2.
-- Attribute: __name__
The name of the parameter specification.
-- Attribute: __default__
The default value of the parameter specification, or *note
typing.NoDefault: 264. if it has no default.
Added in version 3.13.
-- Method: has_default ()
Return whether or not the parameter specification has a
default value. This is equivalent to checking whether *note
__default__: 3f9c. is not the *note typing.NoDefault: 264.
singleton, except that it does not force evaluation of the
*note lazily evaluated: 452. default value.
Added in version 3.13.
Parameter specification variables created with ‘covariant=True’ or
‘contravariant=True’ can be used to declare covariant or
contravariant generic types. The ‘bound’ argument is also
accepted, similar to *note TypeVar: 15d. However the actual
semantics of these keywords are yet to be decided.
Added in version 3.10.
Changed in version 3.12: Parameter specifications can now be
declared using the *note type parameter: 2b5. syntax introduced by
PEP 695(6).
Changed in version 3.13: Support for default values was added.
Note: Only parameter specification variables defined in global
scope can be pickled.
See also
........
* PEP 612(7) – Parameter Specification Variables (the PEP which
introduced ‘ParamSpec’ and ‘Concatenate’)
* *note Concatenate: 7bf.
* *note Annotating callable objects: 2533.
-- Data: typing.ParamSpecArgs
-- Data: typing.ParamSpecKwargs
Arguments and keyword arguments attributes of a *note ParamSpec:
15e. The ‘P.args’ attribute of a ‘ParamSpec’ is an instance of
‘ParamSpecArgs’, and ‘P.kwargs’ is an instance of
‘ParamSpecKwargs’. They are intended for runtime introspection and
have no special meaning to static type checkers.
Calling *note get_origin(): a46. on either of these objects will
return the original ‘ParamSpec’:
>>> from typing import ParamSpec, get_origin
>>> P = ParamSpec("P")
>>> get_origin(P.args) is P
True
>>> get_origin(P.kwargs) is P
True
Added in version 3.10.
-- Class: typing.TypeAliasType (name, value, *, type_params=())
The type of type aliases created through the *note type: 433.
statement.
Example:
>>> type Alias = int
>>> type(Alias)
Added in version 3.12.
-- Attribute: __name__
The name of the type alias:
>>> type Alias = int
>>> Alias.__name__
'Alias'
-- Attribute: __module__
The module in which the type alias was defined:
>>> type Alias = int
>>> Alias.__module__
'__main__'
-- Attribute: __type_params__
The type parameters of the type alias, or an empty tuple if
the alias is not generic:
>>> type ListOrSet[T] = list[T] | set[T]
>>> ListOrSet.__type_params__
(T,)
>>> type NotGeneric = int
>>> NotGeneric.__type_params__
()
-- Attribute: __value__
The type alias’s value. This is *note lazily evaluated: 452,
so names used in the definition of the alias are not resolved
until the ‘__value__’ attribute is accessed:
>>> type Mutually = Recursive
>>> type Recursive = Mutually
>>> Mutually
Mutually
>>> Recursive
Recursive
>>> Mutually.__value__
Recursive
>>> Recursive.__value__
Mutually
---------- Footnotes ----------
(1) https://peps.python.org/pep-0484/
(2) https://peps.python.org/pep-0695/
(3) https://peps.python.org/pep-0695/
(4) https://peps.python.org/pep-0646/
(5) https://peps.python.org/pep-0695/
(6) https://peps.python.org/pep-0695/
(7) https://peps.python.org/pep-0612/
File: python.info, Node: Other special directives, Prev: Building generic types and type aliases, Up: Special typing primitives
5.27.1.17 Other special directives
..................................
These functions and classes should not be used directly as annotations.
Their intended purpose is to be building blocks for creating and
declaring types.
-- Class: typing.NamedTuple
Typed version of *note collections.namedtuple(): 1ca.
Usage:
class Employee(NamedTuple):
name: str
id: int
This is equivalent to:
Employee = collections.namedtuple('Employee', ['name', 'id'])
To give a field a default value, you can assign to it in the class
body:
class Employee(NamedTuple):
name: str
id: int = 3
employee = Employee('Guido')
assert employee.id == 3
Fields with a default value must come after any fields without a
default.
The resulting class has an extra attribute ‘__annotations__’ giving
a dict that maps the field names to the field types. (The field
names are in the ‘_fields’ attribute and the default values are in
the ‘_field_defaults’ attribute, both of which are part of the
*note namedtuple(): 1ca. API.)
‘NamedTuple’ subclasses can also have docstrings and methods:
class Employee(NamedTuple):
"""Represents an employee."""
name: str
id: int = 3
def __repr__(self) -> str:
return f''
‘NamedTuple’ subclasses can be generic:
class Group[T](NamedTuple):
key: T
group: list[T]
Backward-compatible usage:
# For creating a generic NamedTuple on Python 3.11
T = TypeVar("T")
class Group(NamedTuple, Generic[T]):
key: T
group: list[T]
# A functional syntax is also supported
Employee = NamedTuple('Employee', [('name', str), ('id', int)])
Changed in version 3.6: Added support for PEP 526(1) variable
annotation syntax.
Changed in version 3.6.1: Added support for default values,
methods, and docstrings.
Changed in version 3.8: The ‘_field_types’ and ‘__annotations__’
attributes are now regular dictionaries instead of instances of
‘OrderedDict’.
Changed in version 3.9: Removed the ‘_field_types’ attribute in
favor of the more standard ‘__annotations__’ attribute which has
the same information.
Changed in version 3.11: Added support for generic namedtuples.
Deprecated since version 3.13, will be removed in version 3.15: The
undocumented keyword argument syntax for creating NamedTuple
classes (‘NT = NamedTuple("NT", x=int)’) is deprecated, and will be
disallowed in 3.15. Use the class-based syntax or the functional
syntax instead.
Deprecated since version 3.13, will be removed in version 3.15:
When using the functional syntax to create a NamedTuple class,
failing to pass a value to the ‘fields’ parameter (‘NT =
NamedTuple("NT")’) is deprecated. Passing ‘None’ to the ‘fields’
parameter (‘NT = NamedTuple("NT", None)’) is also deprecated. Both
will be disallowed in Python 3.15. To create a NamedTuple class
with 0 fields, use ‘class NT(NamedTuple): pass’ or ‘NT =
NamedTuple("NT", [])’.
-- Class: typing.NewType (name, tp)
Helper class to create low-overhead *note distinct types: 3f76.
A ‘NewType’ is considered a distinct type by a typechecker. At
runtime, however, calling a ‘NewType’ returns its argument
unchanged.
Usage:
UserId = NewType('UserId', int) # Declare the NewType "UserId"
first_user = UserId(1) # "UserId" returns the argument unchanged at runtime
-- Attribute: __module__
The module in which the new type is defined.
-- Attribute: __name__
The name of the new type.
-- Attribute: __supertype__
The type that the new type is based on.
Added in version 3.5.2.
Changed in version 3.10: ‘NewType’ is now a class rather than a
function.
-- Class: typing.Protocol (Generic)
Base class for protocol classes.
Protocol classes are defined like this:
class Proto(Protocol):
def meth(self) -> int:
...
Such classes are primarily used with static type checkers that
recognize structural subtyping (static duck-typing), for example:
class C:
def meth(self) -> int:
return 0
def func(x: Proto) -> int:
return x.meth()
func(C()) # Passes static type check
See PEP 544(2) for more details. Protocol classes decorated with
*note runtime_checkable(): 43e. (described later) act as
simple-minded runtime protocols that check only the presence of
given attributes, ignoring their type signatures. Protocol classes
without this decorator cannot be used as the second argument to
*note isinstance(): 43d. or *note issubclass(): 7bc.
Protocol classes can be generic, for example:
class GenProto[T](Protocol):
def meth(self) -> T:
...
In code that needs to be compatible with Python 3.11 or older,
generic Protocols can be written as follows:
T = TypeVar("T")
class GenProto(Protocol[T]):
def meth(self) -> T:
...
Added in version 3.8.
-- Function: @typing.runtime_checkable
Mark a protocol class as a runtime protocol.
Such a protocol can be used with *note isinstance(): 43d. and *note
issubclass(): 7bc. This allows a simple-minded structural check,
very similar to “one trick ponies” in *note collections.abc: 1e.
such as *note Iterable: 224d. For example:
@runtime_checkable
class Closable(Protocol):
def close(self): ...
assert isinstance(open('/some/file'), Closable)
@runtime_checkable
class Named(Protocol):
name: str
import threading
assert isinstance(threading.Thread(name='Bob'), Named)
This decorator raises *note TypeError: 534. when applied to a
non-protocol class.
Note: ‘runtime_checkable()’ will check only the presence of
the required methods or attributes, not their type signatures
or types. For example, *note ssl.SSLObject: b81. is a class,
therefore it passes an *note issubclass(): 7bc. check against
*note Callable: 2533. However, the ‘ssl.SSLObject.__init__’
method exists only to raise a *note TypeError: 534. with a
more informative message, therefore making it impossible to
call (instantiate) *note ssl.SSLObject: b81.
Note: An *note isinstance(): 43d. check against a
runtime-checkable protocol can be surprisingly slow compared
to an ‘isinstance()’ check against a non-protocol class.
Consider using alternative idioms such as *note hasattr():
4ce. calls for structural checks in performance-sensitive
code.
Added in version 3.8.
Changed in version 3.12: The internal implementation of *note
isinstance(): 43d. checks against runtime-checkable protocols now
uses *note inspect.getattr_static(): 490. to look up attributes
(previously, *note hasattr(): 4ce. was used). As a result, some
objects which used to be considered instances of a
runtime-checkable protocol may no longer be considered instances of
that protocol on Python 3.12+, and vice versa. Most users are
unlikely to be affected by this change.
Changed in version 3.12: The members of a runtime-checkable
protocol are now considered “frozen” at runtime as soon as the
class has been created. Monkey-patching attributes onto a
runtime-checkable protocol will still work, but will have no impact
on *note isinstance(): 43d. checks comparing objects to the
protocol. See *note What’s new in Python 3.12: 4cd. for more
details.
-- Class: typing.TypedDict (dict)
Special construct to add type hints to a dictionary. At runtime it
is a plain *note dict: 258.
‘TypedDict’ declares a dictionary type that expects all of its
instances to have a certain set of keys, where each key is
associated with a value of a consistent type. This expectation is
not checked at runtime but is only enforced by type checkers.
Usage:
class Point2D(TypedDict):
x: int
y: int
label: str
a: Point2D = {'x': 1, 'y': 2, 'label': 'good'} # OK
b: Point2D = {'z': 3, 'label': 'bad'} # Fails type check
assert Point2D(x=1, y=2, label='first') == dict(x=1, y=2, label='first')
An alternative way to create a ‘TypedDict’ is by using
function-call syntax. The second argument must be a literal *note
dict: 258.:
Point2D = TypedDict('Point2D', {'x': int, 'y': int, 'label': str})
This functional syntax allows defining keys which are not valid
*note identifiers: 1e94, for example because they are keywords or
contain hyphens, or when key names must not be *note mangled: 1cba.
like regular private names:
# raises SyntaxError
class Point2D(TypedDict):
in: int # 'in' is a keyword
x-y: int # name with hyphens
class Definition(TypedDict):
__schema: str # mangled to `_Definition__schema`
# OK, functional syntax
Point2D = TypedDict('Point2D', {'in': int, 'x-y': int})
Definition = TypedDict('Definition', {'__schema': str}) # not mangled
By default, all keys must be present in a ‘TypedDict’. It is
possible to mark individual keys as non-required using *note
NotRequired: 5c1.:
class Point2D(TypedDict):
x: int
y: int
label: NotRequired[str]
# Alternative syntax
Point2D = TypedDict('Point2D', {'x': int, 'y': int, 'label': NotRequired[str]})
This means that a ‘Point2D’ ‘TypedDict’ can have the ‘label’ key
omitted.
It is also possible to mark all keys as non-required by default by
specifying a totality of ‘False’:
class Point2D(TypedDict, total=False):
x: int
y: int
# Alternative syntax
Point2D = TypedDict('Point2D', {'x': int, 'y': int}, total=False)
This means that a ‘Point2D’ ‘TypedDict’ can have any of the keys
omitted. A type checker is only expected to support a literal
‘False’ or ‘True’ as the value of the ‘total’ argument. ‘True’ is
the default, and makes all items defined in the class body
required.
Individual keys of a ‘total=False’ ‘TypedDict’ can be marked as
required using *note Required: 5c0.:
class Point2D(TypedDict, total=False):
x: Required[int]
y: Required[int]
label: str
# Alternative syntax
Point2D = TypedDict('Point2D', {
'x': Required[int],
'y': Required[int],
'label': str
}, total=False)
It is possible for a ‘TypedDict’ type to inherit from one or more
other ‘TypedDict’ types using the class-based syntax. Usage:
class Point3D(Point2D):
z: int
‘Point3D’ has three items: ‘x’, ‘y’ and ‘z’. It is equivalent to
this definition:
class Point3D(TypedDict):
x: int
y: int
z: int
A ‘TypedDict’ cannot inherit from a non-‘TypedDict’ class, except
for *note Generic: 1619. For example:
class X(TypedDict):
x: int
class Y(TypedDict):
y: int
class Z(object): pass # A non-TypedDict class
class XY(X, Y): pass # OK
class XZ(X, Z): pass # raises TypeError
A ‘TypedDict’ can be generic:
class Group[T](TypedDict):
key: T
group: list[T]
To create a generic ‘TypedDict’ that is compatible with Python 3.11
or lower, inherit from *note Generic: 1619. explicitly:
T = TypeVar("T")
class Group(TypedDict, Generic[T]):
key: T
group: list[T]
A ‘TypedDict’ can be introspected via annotations dicts (see *note
Annotations Best Practices: 7d4. for more information on
annotations best practices), *note __total__: 3fa6, *note
__required_keys__: 3fa7, and *note __optional_keys__: 3fa8.
-- Attribute: __total__
‘Point2D.__total__’ gives the value of the ‘total’ argument.
Example:
>>> from typing import TypedDict
>>> class Point2D(TypedDict): pass
>>> Point2D.__total__
True
>>> class Point2D(TypedDict, total=False): pass
>>> Point2D.__total__
False
>>> class Point3D(Point2D): pass
>>> Point3D.__total__
True
This attribute reflects 'only' the value of the ‘total’
argument to the current ‘TypedDict’ class, not whether the
class is semantically total. For example, a ‘TypedDict’ with
‘__total__’ set to ‘True’ may have keys marked with *note
NotRequired: 5c1, or it may inherit from another ‘TypedDict’
with ‘total=False’. Therefore, it is generally better to use
*note __required_keys__: 3fa7. and *note __optional_keys__:
3fa8. for introspection.
-- Attribute: __required_keys__
Added in version 3.9.
-- Attribute: __optional_keys__
‘Point2D.__required_keys__’ and ‘Point2D.__optional_keys__’
return *note frozenset: 5d6. objects containing required and
non-required keys, respectively.
Keys marked with *note Required: 5c0. will always appear in
‘__required_keys__’ and keys marked with *note NotRequired:
5c1. will always appear in ‘__optional_keys__’.
For backwards compatibility with Python 3.10 and below, it is
also possible to use inheritance to declare both required and
non-required keys in the same ‘TypedDict’ . This is done by
declaring a ‘TypedDict’ with one value for the ‘total’
argument and then inheriting from it in another ‘TypedDict’
with a different value for ‘total’:
>>> class Point2D(TypedDict, total=False):
... x: int
... y: int
...
>>> class Point3D(Point2D):
... z: int
...
>>> Point3D.__required_keys__ == frozenset({'z'})
True
>>> Point3D.__optional_keys__ == frozenset({'x', 'y'})
True
Added in version 3.9.
Note: If ‘from __future__ import annotations’ is used or
if annotations are given as strings, annotations are not
evaluated when the ‘TypedDict’ is defined. Therefore,
the runtime introspection that ‘__required_keys__’ and
‘__optional_keys__’ rely on may not work properly, and
the values of the attributes may be incorrect.
Support for *note ReadOnly: 161. is reflected in the following
attributes:
-- Attribute: __readonly_keys__
A *note frozenset: 5d6. containing the names of all read-only
keys. Keys are read-only if they carry the *note ReadOnly:
161. qualifier.
Added in version 3.13.
-- Attribute: __mutable_keys__
A *note frozenset: 5d6. containing the names of all mutable
keys. Keys are mutable if they do not carry the *note
ReadOnly: 161. qualifier.
Added in version 3.13.
See PEP 589(3) for more examples and detailed rules of using
‘TypedDict’.
Added in version 3.8.
Changed in version 3.11: Added support for marking individual keys
as *note Required: 5c0. or *note NotRequired: 5c1. See PEP 655(4).
Changed in version 3.11: Added support for generic ‘TypedDict’s.
Changed in version 3.13: Removed support for the keyword-argument
method of creating ‘TypedDict’s.
Changed in version 3.13: Support for the *note ReadOnly: 161.
qualifier was added.
Deprecated since version 3.13, will be removed in version 3.15:
When using the functional syntax to create a TypedDict class,
failing to pass a value to the ‘fields’ parameter (‘TD =
TypedDict("TD")’) is deprecated. Passing ‘None’ to the ‘fields’
parameter (‘TD = TypedDict("TD", None)’) is also deprecated. Both
will be disallowed in Python 3.15. To create a TypedDict class
with 0 fields, use ‘class TD(TypedDict): pass’ or ‘TD =
TypedDict("TD", {})’.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0526/
(2) https://peps.python.org/pep-0544/
(3) https://peps.python.org/pep-0589/
(4) https://peps.python.org/pep-0655/
File: python.info, Node: Protocols<3>, Next: ABCs for working with IO, Prev: Special typing primitives, Up: Module contents<3>
5.27.1.18 Protocols
...................
The following protocols are provided by the ‘typing’ module. All are
decorated with *note @runtime_checkable: 43e.
-- Class: typing.SupportsAbs
An ABC with one abstract method ‘__abs__’ that is covariant in its
return type.
-- Class: typing.SupportsBytes
An ABC with one abstract method ‘__bytes__’.
-- Class: typing.SupportsComplex
An ABC with one abstract method ‘__complex__’.
-- Class: typing.SupportsFloat
An ABC with one abstract method ‘__float__’.
-- Class: typing.SupportsIndex
An ABC with one abstract method ‘__index__’.
Added in version 3.8.
-- Class: typing.SupportsInt
An ABC with one abstract method ‘__int__’.
-- Class: typing.SupportsRound
An ABC with one abstract method ‘__round__’ that is covariant in
its return type.
File: python.info, Node: ABCs for working with IO, Next: Functions and decorators, Prev: Protocols<3>, Up: Module contents<3>
5.27.1.19 ABCs for working with IO
..................................
-- Class: typing.IO
-- Class: typing.TextIO
-- Class: typing.BinaryIO
Generic type ‘IO[AnyStr]’ and its subclasses ‘TextIO(IO[str])’ and
‘BinaryIO(IO[bytes])’ represent the types of I/O streams such as
returned by *note open(): 517.
File: python.info, Node: Functions and decorators, Next: Introspection helpers, Prev: ABCs for working with IO, Up: Module contents<3>
5.27.1.20 Functions and decorators
..................................
-- Function: typing.cast (typ, val)
Cast a value to a type.
This returns the value unchanged. To the type checker this signals
that the return value has the designated type, but at runtime we
intentionally don’t check anything (we want this to be as fast as
possible).
-- Function: typing.assert_type (val, typ, /)
Ask a static type checker to confirm that 'val' has an inferred
type of 'typ'.
At runtime this does nothing: it returns the first argument
unchanged with no checks or side effects, no matter the actual type
of the argument.
When a static type checker encounters a call to ‘assert_type()’, it
emits an error if the value is not of the specified type:
def greet(name: str) -> None:
assert_type(name, str) # OK, inferred type of `name` is `str`
assert_type(name, int) # type checker error
This function is useful for ensuring the type checker’s
understanding of a script is in line with the developer’s
intentions:
def complex_function(arg: object):
# Do some complex type-narrowing logic,
# after which we hope the inferred type will be `int`
...
# Test whether the type checker correctly understands our function
assert_type(arg, int)
Added in version 3.11.
-- Function: typing.assert_never (arg, /)
Ask a static type checker to confirm that a line of code is
unreachable.
Example:
def int_or_str(arg: int | str) -> None:
match arg:
case int():
print("It's an int")
case str():
print("It's a str")
case _ as unreachable:
assert_never(unreachable)
Here, the annotations allow the type checker to infer that the last
case can never execute, because ‘arg’ is either an *note int: 259.
or a *note str: 447, and both options are covered by earlier cases.
If a type checker finds that a call to ‘assert_never()’ is
reachable, it will emit an error. For example, if the type
annotation for ‘arg’ was instead ‘int | str | float’, the type
checker would emit an error pointing out that ‘unreachable’ is of
type *note float: 2f1. For a call to ‘assert_never’ to pass type
checking, the inferred type of the argument passed in must be the
bottom type, *note Never: 6a4, and nothing else.
At runtime, this throws an exception when called.
See also
........
Unreachable Code and Exhaustiveness Checking(1) has more
information about exhaustiveness checking with static typing.
Added in version 3.11.
-- Function: typing.reveal_type (obj, /)
Ask a static type checker to reveal the inferred type of an
expression.
When a static type checker encounters a call to this function, it
emits a diagnostic with the inferred type of the argument. For
example:
x: int = 1
reveal_type(x) # Revealed type is "builtins.int"
This can be useful when you want to debug how your type checker
handles a particular piece of code.
At runtime, this function prints the runtime type of its argument
to *note sys.stderr: 939. and returns the argument unchanged
(allowing the call to be used within an expression):
x = reveal_type(1) # prints "Runtime type is int"
print(x) # prints "1"
Note that the runtime type may be different from (more or less
specific than) the type statically inferred by a type checker.
Most type checkers support ‘reveal_type()’ anywhere, even if the
name is not imported from ‘typing’. Importing the name from
‘typing’, however, allows your code to run without runtime errors
and communicates intent more clearly.
Added in version 3.11.
-- Function: @typing.dataclass_transform (*, eq_default=True,
order_default=False, kw_only_default=False,
frozen_default=False, field_specifiers=(), **kwargs)
Decorator to mark an object as providing *note dataclass: 1cb.-like
behavior.
‘dataclass_transform’ may be used to decorate a class, metaclass,
or a function that is itself a decorator. The presence of
‘@dataclass_transform()’ tells a static type checker that the
decorated object performs runtime “magic” that transforms a class
in a similar way to *note @dataclasses.dataclass: 1cb.
Example usage with a decorator function:
@dataclass_transform()
def create_model[T](cls: type[T]) -> type[T]:
...
return cls
@create_model
class CustomerModel:
id: int
name: str
On a base class:
@dataclass_transform()
class ModelBase: ...
class CustomerModel(ModelBase):
id: int
name: str
On a metaclass:
@dataclass_transform()
class ModelMeta(type): ...
class ModelBase(metaclass=ModelMeta): ...
class CustomerModel(ModelBase):
id: int
name: str
The ‘CustomerModel’ classes defined above will be treated by type
checkers similarly to classes created with *note
@dataclasses.dataclass: 1cb. For example, type checkers will
assume these classes have ‘__init__’ methods that accept ‘id’ and
‘name’.
The decorated class, metaclass, or function may accept the
following bool arguments which type checkers will assume have the
same effect as they would have on the *note @dataclasses.dataclass:
1cb. decorator: ‘init’, ‘eq’, ‘order’, ‘unsafe_hash’, ‘frozen’,
‘match_args’, ‘kw_only’, and ‘slots’. It must be possible for the
value of these arguments (‘True’ or ‘False’) to be statically
evaluated.
The arguments to the ‘dataclass_transform’ decorator can be used to
customize the default behaviors of the decorated class, metaclass,
or function:
Parameters:
* ‘eq_default’ (*note bool: 463.) – Indicates whether the ‘eq’
parameter is assumed to be ‘True’ or ‘False’ if it is omitted
by the caller. Defaults to ‘True’.
* ‘order_default’ (*note bool: 463.) – Indicates whether the
‘order’ parameter is assumed to be ‘True’ or ‘False’ if it is
omitted by the caller. Defaults to ‘False’.
* ‘kw_only_default’ (*note bool: 463.) – Indicates whether the
‘kw_only’ parameter is assumed to be ‘True’ or ‘False’ if it
is omitted by the caller. Defaults to ‘False’.
* ‘frozen_default’ (*note bool: 463.) – Indicates whether the
‘frozen’ parameter is assumed to be ‘True’ or ‘False’ if it is
omitted by the caller. Defaults to ‘False’.
Added in version 3.12.
* ‘field_specifiers’ (*note tuple: 36b.‘[’*note Callable:
7e5.‘[’‘...’‘, ’‘Any’‘]’‘, ’‘...’‘]’) – Specifies a static
list of supported classes or functions that describe fields,
similar to *note dataclasses.field(): 1a1f. Defaults to ‘()’.
* ‘**kwargs’ (‘Any’) – Arbitrary other keyword arguments are
accepted in order to allow for possible future extensions.
Type checkers recognize the following optional parameters on field
specifiers: 'Recognised parameters for field specifiers'
Parameter name Description
--------------------------------------------------------------------------------------------------------------
‘init’ Indicates whether the field should be included in the synthesized ‘__init__’
method. If unspecified, ‘init’ defaults to ‘True’.
‘default’ Provides the default value for the field.
‘default_factory’ Provides a runtime callback that returns the default value for the field. If
neither ‘default’ nor ‘default_factory’ are specified, the field is assumed to
have no default value and must be provided a value when the class is instantiated.
‘factory’ An alias for the ‘default_factory’ parameter on field specifiers.
‘kw_only’ Indicates whether the field should be marked as keyword-only. If ‘True’, the
field will be keyword-only. If ‘False’, it will not be keyword-only. If
unspecified, the value of the ‘kw_only’ parameter on the object decorated with
‘dataclass_transform’ will be used, or if that is unspecified, the value of
‘kw_only_default’ on ‘dataclass_transform’ will be used.
‘alias’ Provides an alternative name for the field. This alternative name is used in the
synthesized ‘__init__’ method.
At runtime, this decorator records its arguments in the
‘__dataclass_transform__’ attribute on the decorated object. It
has no other runtime effect.
See PEP 681(2) for more details.
Added in version 3.11.
-- Function: @typing.overload
Decorator for creating overloaded functions and methods.
The ‘@overload’ decorator allows describing functions and methods
that support multiple different combinations of argument types. A
series of ‘@overload’-decorated definitions must be followed by
exactly one non-‘@overload’-decorated definition (for the same
function/method).
‘@overload’-decorated definitions are for the benefit of the type
checker only, since they will be overwritten by the
non-‘@overload’-decorated definition. The
non-‘@overload’-decorated definition, meanwhile, will be used at
runtime but should be ignored by a type checker. At runtime,
calling an ‘@overload’-decorated function directly will raise *note
NotImplementedError: 22a.
An example of overload that gives a more precise type than can be
expressed using a union or a type variable:
@overload
def process(response: None) -> None:
...
@overload
def process(response: int) -> tuple[int, str]:
...
@overload
def process(response: bytes) -> str:
...
def process(response):
... # actual implementation goes here
See PEP 484(3) for more details and comparison with other typing
semantics.
Changed in version 3.11: Overloaded functions can now be
introspected at runtime using *note get_overloads(): 6aa.
-- Function: typing.get_overloads (func)
Return a sequence of *note @overload: 3fb3.-decorated definitions
for 'func'.
'func' is the function object for the implementation of the
overloaded function. For example, given the definition of
‘process’ in the documentation for *note @overload: 3fb3,
‘get_overloads(process)’ will return a sequence of three function
objects for the three defined overloads. If called on a function
with no overloads, ‘get_overloads()’ returns an empty sequence.
‘get_overloads()’ can be used for introspecting an overloaded
function at runtime.
Added in version 3.11.
-- Function: typing.clear_overloads ()
Clear all registered overloads in the internal registry.
This can be used to reclaim the memory used by the registry.
Added in version 3.11.
-- Function: @typing.final
Decorator to indicate final methods and final classes.
Decorating a method with ‘@final’ indicates to a type checker that
the method cannot be overridden in a subclass. Decorating a class
with ‘@final’ indicates that it cannot be subclassed.
For example:
class Base:
@final
def done(self) -> None:
...
class Sub(Base):
def done(self) -> None: # Error reported by type checker
...
@final
class Leaf:
...
class Other(Leaf): # Error reported by type checker
...
There is no runtime checking of these properties. See PEP 591(4)
for more details.
Added in version 3.8.
Changed in version 3.11: The decorator will now attempt to set a
‘__final__’ attribute to ‘True’ on the decorated object. Thus, a
check like ‘if getattr(obj, "__final__", False)’ can be used at
runtime to determine whether an object ‘obj’ has been marked as
final. If the decorated object does not support setting
attributes, the decorator returns the object unchanged without
raising an exception.
-- Function: @typing.no_type_check
Decorator to indicate that annotations are not type hints.
This works as a class or function *note decorator: 72c. With a
class, it applies recursively to all methods and classes defined in
that class (but not to methods defined in its superclasses or
subclasses). Type checkers will ignore all annotations in a
function or class with this decorator.
‘@no_type_check’ mutates the decorated object in place.
-- Function: @typing.no_type_check_decorator
Decorator to give another decorator the *note no_type_check(): 6b0.
effect.
This wraps the decorator with something that wraps the decorated
function in *note no_type_check(): 6b0.
Deprecated since version 3.13, will be removed in version 3.15: No
type checker ever added support for ‘@no_type_check_decorator’. It
is therefore deprecated, and will be removed in Python 3.15.
-- Function: @typing.override
Decorator to indicate that a method in a subclass is intended to
override a method or attribute in a superclass.
Type checkers should emit an error if a method decorated with
‘@override’ does not, in fact, override anything. This helps
prevent bugs that may occur when a base class is changed without an
equivalent change to a child class.
For example:
class Base:
def log_status(self) -> None:
...
class Sub(Base):
@override
def log_status(self) -> None: # Okay: overrides Base.log_status
...
@override
def done(self) -> None: # Error reported by type checker
...
There is no runtime checking of this property.
The decorator will attempt to set an ‘__override__’ attribute to
‘True’ on the decorated object. Thus, a check like ‘if
getattr(obj, "__override__", False)’ can be used at runtime to
determine whether an object ‘obj’ has been marked as an override.
If the decorated object does not support setting attributes, the
decorator returns the object unchanged without raising an
exception.
See PEP 698(5) for more details.
Added in version 3.12.
-- Function: @typing.type_check_only
Decorator to mark a class or function as unavailable at runtime.
This decorator is itself not available at runtime. It is mainly
intended to mark classes that are defined in type stub files if an
implementation returns an instance of a private class:
@type_check_only
class Response: # private or not available at runtime
code: int
def get_header(self, name: str) -> str: ...
def fetch_response() -> Response: ...
Note that returning instances of private classes is not
recommended. It is usually preferable to make such classes public.
---------- Footnotes ----------
(1) https://typing.python.org/en/latest/guides/unreachable.html
(2) https://peps.python.org/pep-0681/
(3) https://peps.python.org/pep-0484/
(4) https://peps.python.org/pep-0591/
(5) https://peps.python.org/pep-0698/
File: python.info, Node: Introspection helpers, Next: Constant, Prev: Functions and decorators, Up: Module contents<3>
5.27.1.21 Introspection helpers
...............................
-- Function: typing.get_type_hints (obj, globalns=None, localns=None,
include_extras=False)
Return a dictionary containing type hints for a function, method,
module or class object.
This is often the same as ‘obj.__annotations__’, but this function
makes the following changes to the annotations dictionary:
* Forward references encoded as string literals or *note
ForwardRef: 1843. objects are handled by evaluating them in
'globalns', 'localns', and (where applicable) 'obj'’s *note
type parameter: 2b5. namespace. If 'globalns' or 'localns' is
not given, appropriate namespace dictionaries are inferred
from 'obj'.
* ‘None’ is replaced with *note types.NoneType: 84e.
* If *note @no_type_check: 6b0. has been applied to 'obj', an
empty dictionary is returned.
* If 'obj' is a class ‘C’, the function returns a dictionary
that merges annotations from ‘C’’s base classes with those on
‘C’ directly. This is done by traversing *note C.__mro__:
1f29. and iteratively combining ‘__annotations__’
dictionaries. Annotations on classes appearing earlier in the
*note method resolution order: 2180. always take precedence
over annotations on classes appearing later in the method
resolution order.
* The function recursively replaces all occurrences of
‘Annotated[T, ...]’ with ‘T’, unless 'include_extras' is set
to ‘True’ (see *note Annotated: 93d. for more information).
See also *note inspect.get_annotations(): 80f, a lower-level
function that returns annotations more directly.
Note: If any forward references in the annotations of 'obj'
are not resolvable or are not valid Python code, this function
will raise an exception such as *note NameError: 43a. For
example, this can happen with imported *note type aliases:
44f. that include forward references, or with names imported
under *note if TYPE_CHECKING: cf4.
Changed in version 3.9: Added ‘include_extras’ parameter as part of
PEP 593(1). See the documentation on *note Annotated: 93d. for
more information.
Changed in version 3.11: Previously, ‘Optional[t]’ was added for
function and method annotations if a default value equal to ‘None’
was set. Now the annotation is returned unchanged.
-- Function: typing.get_origin (tp)
Get the unsubscripted version of a type: for a typing object of the
form ‘X[Y, Z, ...]’ return ‘X’.
If ‘X’ is a typing-module alias for a builtin or *note collections:
1d. class, it will be normalized to the original class. If ‘X’ is
an instance of *note ParamSpecArgs: 7c1. or *note ParamSpecKwargs:
7c2, return the underlying *note ParamSpec: 15e. Return ‘None’ for
unsupported objects.
Examples:
assert get_origin(str) is None
assert get_origin(Dict[str, int]) is dict
assert get_origin(Union[int, str]) is Union
assert get_origin(Annotated[str, "metadata"]) is Annotated
P = ParamSpec('P')
assert get_origin(P.args) is P
assert get_origin(P.kwargs) is P
Added in version 3.8.
-- Function: typing.get_args (tp)
Get type arguments with all substitutions performed: for a typing
object of the form ‘X[Y, Z, ...]’ return ‘(Y, Z, ...)’.
If ‘X’ is a union or *note Literal: 851. contained in another
generic type, the order of ‘(Y, Z, ...)’ may be different from the
order of the original arguments ‘[Y, Z, ...]’ due to type caching.
Return ‘()’ for unsupported objects.
Examples:
assert get_args(int) == ()
assert get_args(Dict[int, str]) == (int, str)
assert get_args(Union[int, str]) == (int, str)
Added in version 3.8.
-- Function: typing.get_protocol_members (tp)
Return the set of members defined in a *note Protocol: 266.
>>> from typing import Protocol, get_protocol_members
>>> class P(Protocol):
... def a(self) -> str: ...
... b: int
>>> get_protocol_members(P) == frozenset({'a', 'b'})
True
Raise *note TypeError: 534. for arguments that are not Protocols.
Added in version 3.13.
-- Function: typing.is_protocol (tp)
Determine if a type is a *note Protocol: 266.
For example:
class P(Protocol):
def a(self) -> str: ...
b: int
is_protocol(P) # => True
is_protocol(int) # => False
Added in version 3.13.
-- Function: typing.is_typeddict (tp)
Check if a type is a *note TypedDict: 162.
For example:
class Film(TypedDict):
title: str
year: int
assert is_typeddict(Film)
assert not is_typeddict(list | str)
# TypedDict is a factory for creating typed dicts,
# not a typed dict itself
assert not is_typeddict(TypedDict)
Added in version 3.10.
-- Class: typing.ForwardRef
Class used for internal typing representation of string forward
references.
For example, ‘List["SomeClass"]’ is implicitly transformed into
‘List[ForwardRef("SomeClass")]’. ‘ForwardRef’ should not be
instantiated by a user, but may be used by introspection tools.
Note: PEP 585(2) generic types such as ‘list["SomeClass"]’
will not be implicitly transformed into
‘list[ForwardRef("SomeClass")]’ and thus will not
automatically resolve to ‘list[SomeClass]’.
Added in version 3.7.4.
-- Data: typing.NoDefault
A sentinel object used to indicate that a type parameter has no
default value. For example:
>>> T = TypeVar("T")
>>> T.__default__ is typing.NoDefault
True
>>> S = TypeVar("S", default=None)
>>> S.__default__ is None
True
Added in version 3.13.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0593/
(2) https://peps.python.org/pep-0585/
File: python.info, Node: Constant, Next: Deprecated aliases, Prev: Introspection helpers, Up: Module contents<3>
5.27.1.22 Constant
..................
-- Data: typing.TYPE_CHECKING
A special constant that is assumed to be ‘True’ by 3rd party static
type checkers. It is ‘False’ at runtime.
Usage:
if TYPE_CHECKING:
import expensive_mod
def fun(arg: 'expensive_mod.SomeType') -> None:
local_var: expensive_mod.AnotherType = other_fun()
The first type annotation must be enclosed in quotes, making it a
“forward reference”, to hide the ‘expensive_mod’ reference from the
interpreter runtime. Type annotations for local variables are not
evaluated, so the second annotation does not need to be enclosed in
quotes.
Note: If ‘from __future__ import annotations’ is used,
annotations are not evaluated at function definition time.
Instead, they are stored as strings in ‘__annotations__’.
This makes it unnecessary to use quotes around the annotation
(see PEP 563(1)).
Added in version 3.5.2.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0563/
File: python.info, Node: Deprecated aliases, Prev: Constant, Up: Module contents<3>
5.27.1.23 Deprecated aliases
............................
This module defines several deprecated aliases to pre-existing standard
library classes. These were originally included in the ‘typing’ module
in order to support parameterizing these generic classes using ‘[]’.
However, the aliases became redundant in Python 3.9 when the
corresponding pre-existing classes were enhanced to support ‘[]’ (see
PEP 585(1)).
The redundant types are deprecated as of Python 3.9. However, while the
aliases may be removed at some point, removal of these aliases is not
currently planned. As such, no deprecation warnings are currently
issued by the interpreter for these aliases.
If at some point it is decided to remove these deprecated aliases, a
deprecation warning will be issued by the interpreter for at least two
releases prior to removal. The aliases are guaranteed to remain in the
‘typing’ module without deprecation warnings until at least Python 3.14.
Type checkers are encouraged to flag uses of the deprecated types if the
program they are checking targets a minimum Python version of 3.9 or
newer.
* Menu:
* Aliases to built-in types::
* Aliases to types in collections::
* Aliases to other concrete types::
* Aliases to container ABCs in collections.abc: Aliases to container ABCs in collections abc.
* Aliases to asynchronous ABCs in collections.abc: Aliases to asynchronous ABCs in collections abc.
* Aliases to other ABCs in collections.abc: Aliases to other ABCs in collections abc.
* Aliases to contextlib ABCs::
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
File: python.info, Node: Aliases to built-in types, Next: Aliases to types in collections, Up: Deprecated aliases
5.27.1.24 Aliases to built-in types
...................................
-- Class: typing.Dict (dict, MutableMapping[KT, VT])
Deprecated alias to *note dict: 258.
Note that to annotate arguments, it is preferred to use an abstract
collection type such as *note Mapping: 8c9. rather than to use
*note dict: 258. or ‘typing.Dict’.
Deprecated since version 3.9: *note builtins.dict: 258. now
supports subscripting (‘[]’). See PEP 585(1) and *note Generic
Alias Type: 6ae.
-- Class: typing.List (list, MutableSequence[T])
Deprecated alias to *note list: 60d.
Note that to annotate arguments, it is preferred to use an abstract
collection type such as *note Sequence: 11b6. or *note Iterable:
224d. rather than to use *note list: 60d. or ‘typing.List’.
Deprecated since version 3.9: *note builtins.list: 60d. now
supports subscripting (‘[]’). See PEP 585(2) and *note Generic
Alias Type: 6ae.
-- Class: typing.Set (set, MutableSet[T])
Deprecated alias to *note builtins.set: 5d5.
Note that to annotate arguments, it is preferred to use an abstract
collection type such as *note collections.abc.Set: 2245. rather
than to use *note set: 5d5. or *note typing.Set: 3fbe.
Deprecated since version 3.9: *note builtins.set: 5d5. now supports
subscripting (‘[]’). See PEP 585(3) and *note Generic Alias Type:
6ae.
-- Class: typing.FrozenSet (frozenset, AbstractSet[T_co])
Deprecated alias to *note builtins.frozenset: 5d6.
Deprecated since version 3.9: *note builtins.frozenset: 5d6. now
supports subscripting (‘[]’). See PEP 585(4) and *note Generic
Alias Type: 6ae.
-- Data: typing.Tuple
Deprecated alias for *note tuple: 36b.
*note tuple: 36b. and ‘Tuple’ are special-cased in the type system;
see *note Annotating tuples: 3f7b. for more details.
Deprecated since version 3.9: *note builtins.tuple: 36b. now
supports subscripting (‘[]’). See PEP 585(5) and *note Generic
Alias Type: 6ae.
-- Class: typing.Type (Generic[CT_co])
Deprecated alias to *note type: d48.
See *note The type of class objects: 3f7e. for details on using
*note type: d48. or ‘typing.Type’ in type annotations.
Added in version 3.5.2.
Deprecated since version 3.9: *note builtins.type: d48. now
supports subscripting (‘[]’). See PEP 585(6) and *note Generic
Alias Type: 6ae.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
(2) https://peps.python.org/pep-0585/
(3) https://peps.python.org/pep-0585/
(4) https://peps.python.org/pep-0585/
(5) https://peps.python.org/pep-0585/
(6) https://peps.python.org/pep-0585/
File: python.info, Node: Aliases to types in collections, Next: Aliases to other concrete types, Prev: Aliases to built-in types, Up: Deprecated aliases
5.27.1.25 Aliases to types in ‘collections’
...........................................
-- Class: typing.DefaultDict (collections.defaultdict,
MutableMapping[KT, VT])
Deprecated alias to *note collections.defaultdict: 11af.
Added in version 3.5.2.
Deprecated since version 3.9: *note collections.defaultdict: 11af.
now supports subscripting (‘[]’). See PEP 585(1) and *note Generic
Alias Type: 6ae.
-- Class: typing.OrderedDict (collections.OrderedDict,
MutableMapping[KT, VT])
Deprecated alias to *note collections.OrderedDict: 5d7.
Added in version 3.7.2.
Deprecated since version 3.9: *note collections.OrderedDict: 5d7.
now supports subscripting (‘[]’). See PEP 585(2) and *note Generic
Alias Type: 6ae.
-- Class: typing.ChainMap (collections.ChainMap, MutableMapping[KT,
VT])
Deprecated alias to *note collections.ChainMap: c1c.
Added in version 3.6.1.
Deprecated since version 3.9: *note collections.ChainMap: c1c. now
supports subscripting (‘[]’). See PEP 585(3) and *note Generic
Alias Type: 6ae.
-- Class: typing.Counter (collections.Counter, Dict[T, int])
Deprecated alias to *note collections.Counter: 108b.
Added in version 3.6.1.
Deprecated since version 3.9: *note collections.Counter: 108b. now
supports subscripting (‘[]’). See PEP 585(4) and *note Generic
Alias Type: 6ae.
-- Class: typing.Deque (deque, MutableSequence[T])
Deprecated alias to *note collections.deque: 5d8.
Added in version 3.6.1.
Deprecated since version 3.9: *note collections.deque: 5d8. now
supports subscripting (‘[]’). See PEP 585(5) and *note Generic
Alias Type: 6ae.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
(2) https://peps.python.org/pep-0585/
(3) https://peps.python.org/pep-0585/
(4) https://peps.python.org/pep-0585/
(5) https://peps.python.org/pep-0585/
File: python.info, Node: Aliases to other concrete types, Next: Aliases to container ABCs in collections abc, Prev: Aliases to types in collections, Up: Deprecated aliases
5.27.1.26 Aliases to other concrete types
.........................................
-- Class: typing.Pattern
-- Class: typing.Match
Deprecated aliases corresponding to the return types from *note
re.compile(): bd3. and *note re.match(): 1220.
These types (and the corresponding functions) are generic over
*note AnyStr: 2b4. ‘Pattern’ can be specialised as ‘Pattern[str]’
or ‘Pattern[bytes]’; ‘Match’ can be specialised as ‘Match[str]’ or
‘Match[bytes]’.
Deprecated since version 3.9: Classes ‘Pattern’ and ‘Match’ from
*note re: b9. now support ‘[]’. See PEP 585(1) and *note Generic
Alias Type: 6ae.
-- Class: typing.Text
Deprecated alias for *note str: 447.
‘Text’ is provided to supply a forward compatible path for Python 2
code: in Python 2, ‘Text’ is an alias for ‘unicode’.
Use ‘Text’ to indicate that a value must contain a unicode string
in a manner that is compatible with both Python 2 and Python 3:
def add_unicode_checkmark(text: Text) -> Text:
return text + u' \u2713'
Added in version 3.5.2.
Deprecated since version 3.11: Python 2 is no longer supported, and
most type checkers also no longer support type checking Python 2
code. Removal of the alias is not currently planned, but users are
encouraged to use *note str: 447. instead of ‘Text’.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
File: python.info, Node: Aliases to container ABCs in collections abc, Next: Aliases to asynchronous ABCs in collections abc, Prev: Aliases to other concrete types, Up: Deprecated aliases
5.27.1.27 Aliases to container ABCs in ‘collections.abc’
........................................................
-- Class: typing.AbstractSet (Collection[T_co])
Deprecated alias to *note collections.abc.Set: 2245.
Deprecated since version 3.9: *note collections.abc.Set: 2245. now
supports subscripting (‘[]’). See PEP 585(1) and *note Generic
Alias Type: 6ae.
-- Class: typing.ByteString (Sequence[int])
This type represents the types *note bytes: 1c2, *note bytearray:
53a, and *note memoryview: 464. of byte sequences.
Deprecated since version 3.9, will be removed in version 3.14:
Prefer *note collections.abc.Buffer: 2c6, or a union like ‘bytes |
bytearray | memoryview’.
-- Class: typing.Collection (Sized, Iterable[T_co], Container[T_co])
Deprecated alias to *note collections.abc.Collection: c88.
Added in version 3.6.
Deprecated since version 3.9: *note collections.abc.Collection:
c88. now supports subscripting (‘[]’). See PEP 585(2) and *note
Generic Alias Type: 6ae.
-- Class: typing.Container (Generic[T_co])
Deprecated alias to *note collections.abc.Container: 224f.
Deprecated since version 3.9: *note collections.abc.Container:
224f. now supports subscripting (‘[]’). See PEP 585(3) and *note
Generic Alias Type: 6ae.
-- Class: typing.ItemsView (MappingView, AbstractSet[tuple[KT_co,
VT_co]])
Deprecated alias to *note collections.abc.ItemsView: 2253.
Deprecated since version 3.9: *note collections.abc.ItemsView:
2253. now supports subscripting (‘[]’). See PEP 585(4) and *note
Generic Alias Type: 6ae.
-- Class: typing.KeysView (MappingView, AbstractSet[KT_co])
Deprecated alias to *note collections.abc.KeysView: 2252.
Deprecated since version 3.9: *note collections.abc.KeysView: 2252.
now supports subscripting (‘[]’). See PEP 585(5) and *note Generic
Alias Type: 6ae.
-- Class: typing.Mapping (Collection[KT], Generic[KT, VT_co])
Deprecated alias to *note collections.abc.Mapping: 8c9.
Deprecated since version 3.9: *note collections.abc.Mapping: 8c9.
now supports subscripting (‘[]’). See PEP 585(6) and *note Generic
Alias Type: 6ae.
-- Class: typing.MappingView (Sized)
Deprecated alias to *note collections.abc.MappingView: 2251.
Deprecated since version 3.9: *note collections.abc.MappingView:
2251. now supports subscripting (‘[]’). See PEP 585(7) and *note
Generic Alias Type: 6ae.
-- Class: typing.MutableMapping (Mapping[KT, VT])
Deprecated alias to *note collections.abc.MutableMapping: 10a2.
Deprecated since version 3.9: *note collections.abc.MutableMapping:
10a2. now supports subscripting (‘[]’). See PEP 585(8) and *note
Generic Alias Type: 6ae.
-- Class: typing.MutableSequence (Sequence[T])
Deprecated alias to *note collections.abc.MutableSequence: 1a1.
Deprecated since version 3.9: *note
collections.abc.MutableSequence: 1a1. now supports subscripting
(‘[]’). See PEP 585(9) and *note Generic Alias Type: 6ae.
-- Class: typing.MutableSet (AbstractSet[T])
Deprecated alias to *note collections.abc.MutableSet: 2250.
Deprecated since version 3.9: *note collections.abc.MutableSet:
2250. now supports subscripting (‘[]’). See PEP 585(10) and *note
Generic Alias Type: 6ae.
-- Class: typing.Sequence (Reversible[T_co], Collection[T_co])
Deprecated alias to *note collections.abc.Sequence: 11b6.
Deprecated since version 3.9: *note collections.abc.Sequence: 11b6.
now supports subscripting (‘[]’). See PEP 585(11) and *note
Generic Alias Type: 6ae.
-- Class: typing.ValuesView (MappingView, Collection[_VT_co])
Deprecated alias to *note collections.abc.ValuesView: 2254.
Deprecated since version 3.9: *note collections.abc.ValuesView:
2254. now supports subscripting (‘[]’). See PEP 585(12) and *note
Generic Alias Type: 6ae.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
(2) https://peps.python.org/pep-0585/
(3) https://peps.python.org/pep-0585/
(4) https://peps.python.org/pep-0585/
(5) https://peps.python.org/pep-0585/
(6) https://peps.python.org/pep-0585/
(7) https://peps.python.org/pep-0585/
(8) https://peps.python.org/pep-0585/
(9) https://peps.python.org/pep-0585/
(10) https://peps.python.org/pep-0585/
(11) https://peps.python.org/pep-0585/
(12) https://peps.python.org/pep-0585/
File: python.info, Node: Aliases to asynchronous ABCs in collections abc, Next: Aliases to other ABCs in collections abc, Prev: Aliases to container ABCs in collections abc, Up: Deprecated aliases
5.27.1.28 Aliases to asynchronous ABCs in ‘collections.abc’
...........................................................
-- Class: typing.Coroutine (Awaitable[ReturnType], Generic[YieldType,
SendType, ReturnType])
Deprecated alias to *note collections.abc.Coroutine: ddc.
See *note Annotating generators and coroutines: 2534. for details
on using *note collections.abc.Coroutine: ddc. and
‘typing.Coroutine’ in type annotations.
Added in version 3.5.3.
Deprecated since version 3.9: *note collections.abc.Coroutine: ddc.
now supports subscripting (‘[]’). See PEP 585(1) and *note Generic
Alias Type: 6ae.
-- Class: typing.AsyncGenerator (AsyncIterator[YieldType],
Generic[YieldType, SendType])
Deprecated alias to *note collections.abc.AsyncGenerator: c8a.
See *note Annotating generators and coroutines: 2534. for details
on using *note collections.abc.AsyncGenerator: c8a. and
‘typing.AsyncGenerator’ in type annotations.
Added in version 3.6.1.
Deprecated since version 3.9: *note collections.abc.AsyncGenerator:
c8a. now supports subscripting (‘[]’). See PEP 585(2) and *note
Generic Alias Type: 6ae.
Changed in version 3.13: The ‘SendType’ parameter now has a
default.
-- Class: typing.AsyncIterable (Generic[T_co])
Deprecated alias to *note collections.abc.AsyncIterable: dde.
Added in version 3.5.2.
Deprecated since version 3.9: *note collections.abc.AsyncIterable:
dde. now supports subscripting (‘[]’). See PEP 585(3) and *note
Generic Alias Type: 6ae.
-- Class: typing.AsyncIterator (AsyncIterable[T_co])
Deprecated alias to *note collections.abc.AsyncIterator: ddd.
Added in version 3.5.2.
Deprecated since version 3.9: *note collections.abc.AsyncIterator:
ddd. now supports subscripting (‘[]’). See PEP 585(4) and *note
Generic Alias Type: 6ae.
-- Class: typing.Awaitable (Generic[T_co])
Deprecated alias to *note collections.abc.Awaitable: ddb.
Added in version 3.5.2.
Deprecated since version 3.9: *note collections.abc.Awaitable: ddb.
now supports subscripting (‘[]’). See PEP 585(5) and *note Generic
Alias Type: 6ae.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
(2) https://peps.python.org/pep-0585/
(3) https://peps.python.org/pep-0585/
(4) https://peps.python.org/pep-0585/
(5) https://peps.python.org/pep-0585/
File: python.info, Node: Aliases to other ABCs in collections abc, Next: Aliases to contextlib ABCs, Prev: Aliases to asynchronous ABCs in collections abc, Up: Deprecated aliases
5.27.1.29 Aliases to other ABCs in ‘collections.abc’
....................................................
-- Class: typing.Iterable (Generic[T_co])
Deprecated alias to *note collections.abc.Iterable: 224d.
Deprecated since version 3.9: *note collections.abc.Iterable: 224d.
now supports subscripting (‘[]’). See PEP 585(1) and *note Generic
Alias Type: 6ae.
-- Class: typing.Iterator (Iterable[T_co])
Deprecated alias to *note collections.abc.Iterator: 224e.
Deprecated since version 3.9: *note collections.abc.Iterator: 224e.
now supports subscripting (‘[]’). See PEP 585(2) and *note Generic
Alias Type: 6ae.
-- Data: typing.Callable
Deprecated alias to *note collections.abc.Callable: 7e5.
See *note Annotating callable objects: 2533. for details on how to
use *note collections.abc.Callable: 7e5. and ‘typing.Callable’ in
type annotations.
Deprecated since version 3.9: *note collections.abc.Callable: 7e5.
now supports subscripting (‘[]’). See PEP 585(3) and *note Generic
Alias Type: 6ae.
Changed in version 3.10: ‘Callable’ now supports *note ParamSpec:
15e. and *note Concatenate: 7bf. See PEP 612(4) for more details.
-- Class: typing.Generator (Iterator[YieldType], Generic[YieldType,
SendType, ReturnType])
Deprecated alias to *note collections.abc.Generator: dda.
See *note Annotating generators and coroutines: 2534. for details
on using *note collections.abc.Generator: dda. and
‘typing.Generator’ in type annotations.
Deprecated since version 3.9: *note collections.abc.Generator: dda.
now supports subscripting (‘[]’). See PEP 585(5) and *note Generic
Alias Type: 6ae.
Changed in version 3.13: Default values for the send and return
types were added.
-- Class: typing.Hashable
Deprecated alias to *note collections.abc.Hashable: 4f1.
Deprecated since version 3.12: Use *note collections.abc.Hashable:
4f1. directly instead.
-- Class: typing.Reversible (Iterable[T_co])
Deprecated alias to *note collections.abc.Reversible: c89.
Deprecated since version 3.9: *note collections.abc.Reversible:
c89. now supports subscripting (‘[]’). See PEP 585(6) and *note
Generic Alias Type: 6ae.
-- Class: typing.Sized
Deprecated alias to *note collections.abc.Sized: 4f2.
Deprecated since version 3.12: Use *note collections.abc.Sized:
4f2. directly instead.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
(2) https://peps.python.org/pep-0585/
(3) https://peps.python.org/pep-0585/
(4) https://peps.python.org/pep-0612/
(5) https://peps.python.org/pep-0585/
(6) https://peps.python.org/pep-0585/
File: python.info, Node: Aliases to contextlib ABCs, Prev: Aliases to other ABCs in collections abc, Up: Deprecated aliases
5.27.1.30 Aliases to ‘contextlib’ ABCs
......................................
-- Class: typing.ContextManager (Generic[T_co, ExitT_co])
Deprecated alias to *note contextlib.AbstractContextManager: c8d.
The first type parameter, ‘T_co’, represents the type returned by
the *note __enter__(): 5c4. method. The optional second type
parameter, ‘ExitT_co’, which defaults to ‘bool | None’, represents
the type returned by the *note __exit__(): 12f3. method.
Added in version 3.5.4.
Deprecated since version 3.9: *note
contextlib.AbstractContextManager: c8d. now supports subscripting
(‘[]’). See PEP 585(1) and *note Generic Alias Type: 6ae.
Changed in version 3.13: Added the optional second type parameter,
‘ExitT_co’.
-- Class: typing.AsyncContextManager (Generic[T_co, AExitT_co])
Deprecated alias to *note contextlib.AbstractAsyncContextManager:
b2b.
The first type parameter, ‘T_co’, represents the type returned by
the *note __aenter__(): 1fd0. method. The optional second type
parameter, ‘AExitT_co’, which defaults to ‘bool | None’, represents
the type returned by the *note __aexit__(): 162a. method.
Added in version 3.6.2.
Deprecated since version 3.9: *note
contextlib.AbstractAsyncContextManager: b2b. now supports
subscripting (‘[]’). See PEP 585(2) and *note Generic Alias Type:
6ae.
Changed in version 3.13: Added the optional second type parameter,
‘AExitT_co’.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
(2) https://peps.python.org/pep-0585/
File: python.info, Node: Deprecation Timeline of Major Features, Prev: Module contents<3>, Up: typing — Support for type hints
5.27.1.31 Deprecation Timeline of Major Features
................................................
Certain features in ‘typing’ are deprecated and may be removed in a
future version of Python. The following table summarizes major
deprecations for your convenience. This is subject to change, and not
all deprecations are listed.
Feature Deprecated in Projected removal PEP/issue
------------------------------------------------------------------------------------------------------------------------
‘typing’ versions of 3.9 Undecided (see PEP 585(1)
standard collections *note Deprecated aliases: 3fb8.
for more information)
*note typing.ByteString: 2d7. 3.9 3.14 gh-91896(2)
*note typing.Text: 305. 3.11 Undecided gh-92332(3)
*note typing.Hashable: 4ef. 3.12 Undecided gh-94309(4)
and
*note typing.Sized: 4f0.
*note typing.TypeAlias: 7c4. 3.12 Undecided PEP 695(5)
*note @typing.no_type_check_decorator: 2b3.3.13 3.15 gh-106309(6)
*note typing.AnyStr: 2b4. 3.13 3.18 gh-105578(7)
---------- Footnotes ----------
(1) https://peps.python.org/pep-0585/
(2) https://github.com/python/cpython/issues/91896
(3) https://github.com/python/cpython/issues/92332
(4) https://github.com/python/cpython/issues/94309
(5) https://peps.python.org/pep-0695/
(6) https://github.com/python/cpython/issues/106309
(7) https://github.com/python/cpython/issues/105578
File: python.info, Node: pydoc — Documentation generator and online help system, Next: Python Development Mode, Prev: typing — Support for type hints, Up: Development Tools
5.27.2 ‘pydoc’ — Documentation generator and online help system
---------------------------------------------------------------
'Source code:' Lib/pydoc.py(1)
__________________________________________________________________
The ‘pydoc’ module automatically generates documentation from Python
modules. The documentation can be presented as pages of text on the
console, served to a web browser, or saved to HTML files.
For modules, classes, functions and methods, the displayed documentation
is derived from the docstring (i.e. the *note __doc__: 927. attribute)
of the object, and recursively of its documentable members. If there is
no docstring, ‘pydoc’ tries to obtain a description from the block of
comment lines just above the definition of the class, function or method
in the source file, or at the top of the module (see *note
inspect.getcomments(): 15c7.).
The built-in function *note help(): 8d6. invokes the online help system
in the interactive interpreter, which uses ‘pydoc’ to generate its
documentation as text on the console. The same text documentation can
also be viewed from outside the Python interpreter by running ‘pydoc’ as
a script at the operating system’s command prompt. For example, running
python -m pydoc sys
at a shell prompt will display documentation on the *note sys: d9.
module, in a style similar to the manual pages shown by the Unix ‘man’
command. The argument to ‘pydoc’ can be the name of a function, module,
or package, or a dotted reference to a class, method, or function within
a module or module in a package. If the argument to ‘pydoc’ looks like
a path (that is, it contains the path separator for your operating
system, such as a slash in Unix), and refers to an existing Python
source file, then documentation is produced for that file.
Note: In order to find objects and their documentation, ‘pydoc’
imports the module(s) to be documented. Therefore, any code on
module level will be executed on that occasion. Use an ‘if
__name__ == '__main__':’ guard to only execute code when a file is
invoked as a script and not just imported.
When printing output to the console, ‘pydoc’ attempts to paginate the
output for easier reading. If either the ‘MANPAGER’ or the ‘PAGER’
environment variable is set, ‘pydoc’ will use its value as a pagination
program. When both are set, ‘MANPAGER’ is used.
Specifying a ‘-w’ flag before the argument will cause HTML documentation
to be written out to a file in the current directory, instead of
displaying text on the console.
Specifying a ‘-k’ flag before the argument will search the synopsis
lines of all available modules for the keyword given as the argument,
again in a manner similar to the Unix ‘man’ command. The synopsis line
of a module is the first line of its documentation string.
You can also use ‘pydoc’ to start an HTTP server on the local machine
that will serve documentation to visiting web browsers. ‘python -m
pydoc -p 1234’ will start a HTTP server on port 1234, allowing you to
browse the documentation at ‘http://localhost:1234/’ in your preferred
web browser. Specifying ‘0’ as the port number will select an arbitrary
unused port.
‘python -m pydoc -n ’ will start the server listening at the
given hostname. By default the hostname is ‘localhost’ but if you want
the server to be reached from other machines, you may want to change the
host name that the server responds to. During development this is
especially useful if you want to run pydoc from within a container.
‘python -m pydoc -b’ will start the server and additionally open a web
browser to a module index page. Each served page has a navigation bar
at the top where you can 'Get' help on an individual item, 'Search' all
modules with a keyword in their synopsis line, and go to the 'Module
index', 'Topics' and 'Keywords' pages.
When ‘pydoc’ generates documentation, it uses the current environment
and path to locate modules. Thus, invoking ‘pydoc spam’ documents
precisely the version of the module you would get if you started the
Python interpreter and typed ‘import spam’.
Module docs for core modules are assumed to reside in
‘https://docs.python.org/X.Y/library/’ where ‘X’ and ‘Y’ are the major
and minor version numbers of the Python interpreter. This can be
overridden by setting the ‘PYTHONDOCS’ environment variable to a
different URL or to a local directory containing the Library Reference
Manual pages.
Changed in version 3.2: Added the ‘-b’ option.
Changed in version 3.3: The ‘-g’ command line option was removed.
Changed in version 3.4: ‘pydoc’ now uses *note inspect.signature(): 733.
rather than *note inspect.getfullargspec(): 734. to extract signature
information from callables.
Changed in version 3.7: Added the ‘-n’ option.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/pydoc.py
File: python.info, Node: Python Development Mode, Next: doctest — Test interactive Python examples, Prev: pydoc — Documentation generator and online help system, Up: Development Tools
5.27.3 Python Development Mode
------------------------------
Added in version 3.7.
The Python Development Mode introduces additional runtime checks that
are too expensive to be enabled by default. It should not be more
verbose than the default if the code is correct; new warnings are only
emitted when an issue is detected.
It can be enabled using the *note -X dev: 176. command line option or by
setting the *note PYTHONDEVMODE: aed. environment variable to ‘1’.
See also *note Python debug build: 1fb.
* Menu:
* Effects of the Python Development Mode::
* ResourceWarning Example::
* Bad file descriptor error example::
File: python.info, Node: Effects of the Python Development Mode, Next: ResourceWarning Example, Up: Python Development Mode
5.27.3.1 Effects of the Python Development Mode
...............................................
Enabling the Python Development Mode is similar to the following
command, but with additional effects described below:
PYTHONMALLOC=debug PYTHONASYNCIODEBUG=1 python -W default -X faulthandler
Effects of the Python Development Mode:
* Add ‘default’ *note warning filter: 1d39. The following warnings
are shown:
* *note DeprecationWarning: 1a5.
* *note ImportWarning: 4f6.
* *note PendingDeprecationWarning: 8c7.
* *note ResourceWarning: 246.
Normally, the above warnings are filtered by the default *note
warning filters: 1d39.
It behaves as if the *note -W default: 8c6. command line option is
used.
Use the *note -W error: 8c6. command line option or set the *note
PYTHONWARNINGS: baa. environment variable to ‘error’ to treat
warnings as errors.
* Install debug hooks on memory allocators to check for:
* Buffer underflow
* Buffer overflow
* Memory allocator API violation
* Unsafe usage of the GIL
See the *note PyMem_SetupDebugHooks(): c6a. C function.
It behaves as if the *note PYTHONMALLOC: c64. environment variable
is set to ‘debug’.
To enable the Python Development Mode without installing debug
hooks on memory allocators, set the *note PYTHONMALLOC: c64.
environment variable to ‘default’.
* Call *note faulthandler.enable(): c9e. at Python startup to install
handlers for the *note SIGSEGV: 1d49, *note SIGFPE: 1d4a, *note
SIGABRT: 1d4b, *note SIGBUS: 1d4c. and *note SIGILL: 1d4d. signals
to dump the Python traceback on a crash.
It behaves as if the *note -X faulthandler: 176. command line
option is used or if the *note PYTHONFAULTHANDLER: 1079.
environment variable is set to ‘1’.
* Enable *note asyncio debug mode: 1d4f. For example, *note asyncio:
a. checks for coroutines that were not awaited and logs them.
It behaves as if the *note PYTHONASYNCIODEBUG: 1d4e. environment
variable is set to ‘1’.
* Check the 'encoding' and 'errors' arguments for string encoding and
decoding operations. Examples: *note open(): 517, *note
str.encode(): 8d4. and *note bytes.decode(): 8d5.
By default, for best performance, the 'errors' argument is only
checked at the first encoding/decoding error and the 'encoding'
argument is sometimes ignored for empty strings.
* The *note io.IOBase: 1f7. destructor logs ‘close()’ exceptions.
* Set the *note dev_mode: 1aa0. attribute of *note sys.flags: 688. to
‘True’.
The Python Development Mode does not enable the *note tracemalloc: ff.
module by default, because the overhead cost (to performance and memory)
would be too large. Enabling the *note tracemalloc: ff. module provides
additional information on the origin of some errors. For example, *note
ResourceWarning: 246. logs the traceback where the resource was
allocated, and a buffer overflow error logs the traceback where the
memory block was allocated.
The Python Development Mode does not prevent the *note -O: db4. command
line option from removing *note assert: 968. statements nor from setting
*note __debug__: 7d5. to ‘False’.
The Python Development Mode can only be enabled at the Python startup.
Its value can be read from *note sys.flags.dev_mode: 688.
Changed in version 3.8: The *note io.IOBase: 1f7. destructor now logs
‘close()’ exceptions.
Changed in version 3.9: The 'encoding' and 'errors' arguments are now
checked for string encoding and decoding operations.
File: python.info, Node: ResourceWarning Example, Next: Bad file descriptor error example, Prev: Effects of the Python Development Mode, Up: Python Development Mode
5.27.3.2 ResourceWarning Example
................................
Example of a script counting the number of lines of the text file
specified in the command line:
import sys
def main():
fp = open(sys.argv[1])
nlines = len(fp.readlines())
print(nlines)
# The file is closed implicitly
if __name__ == "__main__":
main()
The script does not close the file explicitly. By default, Python does
not emit any warning. Example using README.txt, which has 269 lines:
$ python script.py README.txt
269
Enabling the Python Development Mode displays a *note ResourceWarning:
246. warning:
$ python -X dev script.py README.txt
269
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
In addition, enabling *note tracemalloc: ff. shows the line where the
file was opened:
$ python -X dev -X tracemalloc=5 script.py README.rst
269
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
main()
Object allocated at (most recent call last):
File "script.py", lineno 10
main()
File "script.py", lineno 4
fp = open(sys.argv[1])
The fix is to close explicitly the file. Example using a context
manager:
def main():
# Close the file explicitly when exiting the with block
with open(sys.argv[1]) as fp:
nlines = len(fp.readlines())
print(nlines)
Not closing a resource explicitly can leave a resource open for way
longer than expected; it can cause severe issues upon exiting Python.
It is bad in CPython, but it is even worse in PyPy. Closing resources
explicitly makes an application more deterministic and more reliable.
File: python.info, Node: Bad file descriptor error example, Prev: ResourceWarning Example, Up: Python Development Mode
5.27.3.3 Bad file descriptor error example
..........................................
Script displaying the first line of itself:
import os
def main():
fp = open(__file__)
firstline = fp.readline()
print(firstline.rstrip())
os.close(fp.fileno())
# The file is closed implicitly
main()
By default, Python does not emit any warning:
$ python script.py
import os
The Python Development Mode shows a *note ResourceWarning: 246. and logs
a “Bad file descriptor” error when finalizing the file object:
$ python -X dev script.py
import os
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exception ignored in: <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
Traceback (most recent call last):
File "script.py", line 10, in
main()
OSError: [Errno 9] Bad file descriptor
‘os.close(fp.fileno())’ closes the file descriptor. When the file
object finalizer tries to close the file descriptor again, it fails with
the ‘Bad file descriptor’ error. A file descriptor must be closed only
once. In the worst case scenario, closing it twice can lead to a crash
(see bpo-18748(1) for an example).
The fix is to remove the ‘os.close(fp.fileno())’ line, or open the file
with ‘closefd=False’.
---------- Footnotes ----------
(1) https://bugs.python.org/issue?@action=redirect&bpo=18748
File: python.info, Node: doctest — Test interactive Python examples, Next: unittest — Unit testing framework, Prev: Python Development Mode, Up: Development Tools
5.27.4 ‘doctest’ — Test interactive Python examples
---------------------------------------------------
'Source code:' Lib/doctest.py(1)
__________________________________________________________________
The *note doctest: 3a. module searches for pieces of text that look like
interactive Python sessions, and then executes those sessions to verify
that they work exactly as shown. There are several common ways to use
doctest:
* To check that a module’s docstrings are up-to-date by verifying
that all interactive examples still work as documented.
* To perform regression testing by verifying that interactive
examples from a test file or a test object work as expected.
* To write tutorial documentation for a package, liberally
illustrated with input-output examples. Depending on whether the
examples or the expository text are emphasized, this has the flavor
of “literate testing” or “executable documentation”.
Here’s a complete but small example module:
"""
This is the "example" module.
The example module supplies one function, factorial(). For example,
>>> factorial(5)
120
"""
def factorial(n):
"""Return the factorial of n, an exact integer >= 0.
>>> [factorial(n) for n in range(6)]
[1, 1, 2, 6, 24, 120]
>>> factorial(30)
265252859812191058636308480000000
>>> factorial(-1)
Traceback (most recent call last):
...
ValueError: n must be >= 0
Factorials of floats are OK, but the float must be an exact integer:
>>> factorial(30.1)
Traceback (most recent call last):
...
ValueError: n must be exact integer
>>> factorial(30.0)
265252859812191058636308480000000
It must also not be ridiculously large:
>>> factorial(1e100)
Traceback (most recent call last):
...
OverflowError: n too large
"""
import math
if not n >= 0:
raise ValueError("n must be >= 0")
if math.floor(n) != n:
raise ValueError("n must be exact integer")
if n+1 == n: # catch a value like 1e300
raise OverflowError("n too large")
result = 1
factor = 2
while factor <= n:
result *= factor
factor += 1
return result
if __name__ == "__main__":
import doctest
doctest.testmod()
If you run ‘example.py’ directly from the command line, *note doctest:
3a. works its magic:
$ python example.py
$
There’s no output! That’s normal, and it means all the examples worked.
Pass ‘-v’ to the script, and *note doctest: 3a. prints a detailed log of
what it’s trying, and prints a summary at the end:
$ python example.py -v
Trying:
factorial(5)
Expecting:
120
ok
Trying:
[factorial(n) for n in range(6)]
Expecting:
[1, 1, 2, 6, 24, 120]
ok
And so on, eventually ending with:
Trying:
factorial(1e100)
Expecting:
Traceback (most recent call last):
...
OverflowError: n too large
ok
2 items passed all tests:
1 test in __main__
6 tests in __main__.factorial
7 tests in 2 items.
7 passed.
Test passed.
$
That’s all you need to know to start making productive use of *note
doctest: 3a.! Jump in. The following sections provide full details.
Note that there are many examples of doctests in the standard Python
test suite and libraries. Especially useful examples can be found in
the standard test file ‘Lib/test/test_doctest/test_doctest.py’.
* Menu:
* Simple Usage; Checking Examples in Docstrings: Simple Usage Checking Examples in Docstrings.
* Simple Usage; Checking Examples in a Text File: Simple Usage Checking Examples in a Text File.
* Command-line Usage::
* How It Works::
* Basic API::
* Unittest API::
* Advanced API::
* Debugging::
* Soapbox::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/doctest.py
File: python.info, Node: Simple Usage Checking Examples in Docstrings, Next: Simple Usage Checking Examples in a Text File, Up: doctest — Test interactive Python examples
5.27.4.1 Simple Usage: Checking Examples in Docstrings
......................................................
The simplest way to start using doctest (but not necessarily the way
you’ll continue to do it) is to end each module ‘M’ with:
if __name__ == "__main__":
import doctest
doctest.testmod()
‘doctest’ then examines docstrings in module ‘M’.
Running the module as a script causes the examples in the docstrings to
get executed and verified:
python M.py
This won’t display anything unless an example fails, in which case the
failing example(s) and the cause(s) of the failure(s) are printed to
stdout, and the final line of output is ‘***Test Failed*** N failures.’,
where 'N' is the number of examples that failed.
Run it with the ‘-v’ switch instead:
python M.py -v
and a detailed report of all examples tried is printed to standard
output, along with assorted summaries at the end.
You can force verbose mode by passing ‘verbose=True’ to *note testmod():
1436, or prohibit it by passing ‘verbose=False’. In either of those
cases, *note sys.argv: 1258. is not examined by *note testmod(): 1436.
(so passing ‘-v’ or not has no effect).
There is also a command line shortcut for running *note testmod(): 1436,
see section *note Command-line Usage: 3ff1.
For more information on *note testmod(): 1436, see section *note Basic
API: 3ff2.
File: python.info, Node: Simple Usage Checking Examples in a Text File, Next: Command-line Usage, Prev: Simple Usage Checking Examples in Docstrings, Up: doctest — Test interactive Python examples
5.27.4.2 Simple Usage: Checking Examples in a Text File
.......................................................
Another simple application of doctest is testing interactive examples in
a text file. This can be done with the *note testfile(): 3ff5.
function:
import doctest
doctest.testfile("example.txt")
That short script executes and verifies any interactive Python examples
contained in the file ‘example.txt’. The file content is treated as if
it were a single giant docstring; the file doesn’t need to contain a
Python program! For example, perhaps ‘example.txt’ contains this:
The ``example`` module
======================
Using ``factorial``
-------------------
This is an example text file in reStructuredText format. First import
``factorial`` from the ``example`` module:
>>> from example import factorial
Now use it:
>>> factorial(6)
120
Running ‘doctest.testfile("example.txt")’ then finds the error in this
documentation:
File "./example.txt", line 14, in example.txt
Failed example:
factorial(6)
Expected:
120
Got:
720
As with *note testmod(): 1436, *note testfile(): 3ff5. won’t display
anything unless an example fails. If an example does fail, then the
failing example(s) and the cause(s) of the failure(s) are printed to
stdout, using the same format as ‘testmod()’.
By default, *note testfile(): 3ff5. looks for files in the calling
module’s directory. See section *note Basic API: 3ff2. for a
description of the optional arguments that can be used to tell it to
look for files in other locations.
Like *note testmod(): 1436, *note testfile(): 3ff5.’s verbosity can be
set with the ‘-v’ command-line switch or with the optional keyword
argument 'verbose'.
There is also a command line shortcut for running *note testfile():
3ff5, see section *note Command-line Usage: 3ff1.
For more information on *note testfile(): 3ff5, see section *note Basic
API: 3ff2.
File: python.info, Node: Command-line Usage, Next: How It Works, Prev: Simple Usage Checking Examples in a Text File, Up: doctest — Test interactive Python examples
5.27.4.3 Command-line Usage
...........................
The *note doctest: 3a. module can be invoked as a script from the
command line:
python -m doctest [-v] [-o OPTION] [-f] file [file ...]
-- Option: -v, --verbose
Detailed report of all examples tried is printed to standard
output, along with assorted summaries at the end:
python -m doctest -v example.py
This will import ‘example.py’ as a standalone module and run *note
testmod(): 1436. on it. Note that this may not work correctly if
the file is part of a package and imports other submodules from
that package.
If the file name does not end with ‘.py’, ‘doctest’ infers that it
must be run with *note testfile(): 3ff5. instead:
python -m doctest -v example.txt
-- Option: -o, --option
Option flags control various aspects of doctest’s behavior, see
section *note Option Flags: f2e.
Added in version 3.4.
-- Option: -f, --fail-fast
This is shorthand for ‘-o FAIL_FAST’.
Added in version 3.4.
File: python.info, Node: How It Works, Next: Basic API, Prev: Command-line Usage, Up: doctest — Test interactive Python examples
5.27.4.4 How It Works
.....................
This section examines in detail how doctest works: which docstrings it
looks at, how it finds interactive examples, what execution context it
uses, how it handles exceptions, and how option flags can be used to
control its behavior. This is the information that you need to know to
write doctest examples; for information about actually running doctest
on these examples, see the following sections.
* Menu:
* Which Docstrings Are Examined?::
* How are Docstring Examples Recognized?::
* What’s the Execution Context?::
* What About Exceptions?::
* Option Flags::
* Directives::
* Warnings: Warnings<2>.
File: python.info, Node: Which Docstrings Are Examined?, Next: How are Docstring Examples Recognized?, Up: How It Works
5.27.4.5 Which Docstrings Are Examined?
.......................................
The module docstring, and all function, class and method docstrings are
searched. Objects imported into the module are not searched.
In addition, there are cases when you want tests to be part of a module
but not part of the help text, which requires that the tests not be
included in the docstring. Doctest looks for a module-level variable
called ‘__test__’ and uses it to locate other tests. If ‘M.__test__’
exists, it must be a dict, and each entry maps a (string) name to a
function object, class object, or string. Function and class object
docstrings found from ‘M.__test__’ are searched, and strings are treated
as if they were docstrings. In output, a key ‘K’ in ‘M.__test__’
appears with name ‘M.__test__.K’.
For example, place this block of code at the top of ‘example.py’:
__test__ = {
'numbers': """
>>> factorial(6)
720
>>> [factorial(n) for n in range(6)]
[1, 1, 2, 6, 24, 120]
"""
}
The value of ‘example.__test__["numbers"]’ will be treated as a
docstring and all the tests inside it will be run. It is important to
note that the value can be mapped to a function, class object, or
module; if so, ‘doctest’ searches them recursively for docstrings, which
are then scanned for tests.
Any classes found are recursively searched similarly, to test docstrings
in their contained methods and nested classes.
File: python.info, Node: How are Docstring Examples Recognized?, Next: What’s the Execution Context?, Prev: Which Docstrings Are Examined?, Up: How It Works
5.27.4.6 How are Docstring Examples Recognized?
...............................................
In most cases a copy-and-paste of an interactive console session works
fine, but doctest isn’t trying to do an exact emulation of any specific
Python shell.
>>> # comments are ignored
>>> x = 12
>>> x
12
>>> if x == 13:
... print("yes")
... else:
... print("no")
... print("NO")
... print("NO!!!")
...
no
NO
NO!!!
>>>
Any expected output must immediately follow the final ‘'>>> '’ or ‘'...
'’ line containing the code, and the expected output (if any) extends to
the next ‘'>>> '’ or all-whitespace line.
The fine print:
* Expected output cannot contain an all-whitespace line, since such a
line is taken to signal the end of expected output. If expected
output does contain a blank line, put ‘’ in your doctest
example each place a blank line is expected.
* All hard tab characters are expanded to spaces, using 8-column tab
stops. Tabs in output generated by the tested code are not
modified. Because any hard tabs in the sample output 'are'
expanded, this means that if the code output includes hard tabs,
the only way the doctest can pass is if the *note
NORMALIZE_WHITESPACE: 4004. option or *note directive: 4005. is in
effect. Alternatively, the test can be rewritten to capture the
output and compare it to an expected value as part of the test.
This handling of tabs in the source was arrived at through trial
and error, and has proven to be the least error prone way of
handling them. It is possible to use a different algorithm for
handling tabs by writing a custom *note DocTestParser: 4006. class.
* Output to stdout is captured, but not output to stderr (exception
tracebacks are captured via a different means).
* If you continue a line via backslashing in an interactive session,
or for any other reason use a backslash, you should use a raw
docstring, which will preserve your backslashes exactly as you type
them:
>>> def f(x):
... r'''Backslashes in a raw docstring: m\n'''
...
>>> print(f.__doc__)
Backslashes in a raw docstring: m\n
Otherwise, the backslash will be interpreted as part of the string.
For example, the ‘\n’ above would be interpreted as a newline
character. Alternatively, you can double each backslash in the
doctest version (and not use a raw string):
>>> def f(x):
... '''Backslashes in a raw docstring: m\\n'''
...
>>> print(f.__doc__)
Backslashes in a raw docstring: m\n
* The starting column doesn’t matter:
>>> assert "Easy!"
>>> import math
>>> math.floor(1.9)
1
and as many leading whitespace characters are stripped from the
expected output as appeared in the initial ‘'>>> '’ line that
started the example.
File: python.info, Node: What’s the Execution Context?, Next: What About Exceptions?, Prev: How are Docstring Examples Recognized?, Up: How It Works
5.27.4.7 What’s the Execution Context?
......................................
By default, each time *note doctest: 3a. finds a docstring to test, it
uses a 'shallow copy' of ‘M’’s globals, so that running tests doesn’t
change the module’s real globals, and so that one test in ‘M’ can’t
leave behind crumbs that accidentally allow another test to work. This
means examples can freely use any names defined at top-level in ‘M’, and
names defined earlier in the docstring being run. Examples cannot see
names defined in other docstrings.
You can force use of your own dict as the execution context by passing
‘globs=your_dict’ to *note testmod(): 1436. or *note testfile(): 3ff5.
instead.
File: python.info, Node: What About Exceptions?, Next: Option Flags, Prev: What’s the Execution Context?, Up: How It Works
5.27.4.8 What About Exceptions?
...............................
No problem, provided that the traceback is the only output produced by
the example: just paste in the traceback. (1) Since tracebacks contain
details that are likely to change rapidly (for example, exact file paths
and line numbers), this is one case where doctest works hard to be
flexible in what it accepts.
Simple example:
>>> [1, 2, 3].remove(42)
Traceback (most recent call last):
File "", line 1, in
ValueError: list.remove(x): x not in list
That doctest succeeds if *note ValueError: 204. is raised, with the
‘list.remove(x): x not in list’ detail as shown.
The expected output for an exception must start with a traceback header,
which may be either of the following two lines, indented the same as the
first line of the example:
Traceback (most recent call last):
Traceback (innermost last):
The traceback header is followed by an optional traceback stack, whose
contents are ignored by doctest. The traceback stack is typically
omitted, or copied verbatim from an interactive session.
The traceback stack is followed by the most interesting part: the
line(s) containing the exception type and detail. This is usually the
last line of a traceback, but can extend across multiple lines if the
exception has a multi-line detail:
>>> raise ValueError('multi\n line\ndetail')
Traceback (most recent call last):
File "", line 1, in
ValueError: multi
line
detail
The last three lines (starting with *note ValueError: 204.) are compared
against the exception’s type and detail, and the rest are ignored.
Best practice is to omit the traceback stack, unless it adds significant
documentation value to the example. So the last example is probably
better as:
>>> raise ValueError('multi\n line\ndetail')
Traceback (most recent call last):
...
ValueError: multi
line
detail
Note that tracebacks are treated very specially. In particular, in the
rewritten example, the use of ‘...’ is independent of doctest’s *note
ELLIPSIS: 1437. option. The ellipsis in that example could be left out,
or could just as well be three (or three hundred) commas or digits, or
an indented transcript of a Monty Python skit.
Some details you should read once, but won’t need to remember:
* Doctest can’t guess whether your expected output came from an
exception traceback or from ordinary printing. So, e.g., an
example that expects ‘ValueError: 42 is prime’ will pass whether
*note ValueError: 204. is actually raised or if the example merely
prints that traceback text. In practice, ordinary output rarely
begins with a traceback header line, so this doesn’t create real
problems.
* Each line of the traceback stack (if present) must be indented
further than the first line of the example, 'or' start with a
non-alphanumeric character. The first line following the traceback
header indented the same and starting with an alphanumeric is taken
to be the start of the exception detail. Of course this does the
right thing for genuine tracebacks.
* When the *note IGNORE_EXCEPTION_DETAIL: 12fc. doctest option is
specified, everything following the leftmost colon and any module
information in the exception name is ignored.
* The interactive shell omits the traceback header line for some
*note SyntaxError: 18d.s. But doctest uses the traceback header
line to distinguish exceptions from non-exceptions. So in the rare
case where you need to test a ‘SyntaxError’ that omits the
traceback header, you will need to manually add the traceback
header line to your test example.
* For some exceptions, Python displays the position of the error
using ‘^’ markers and tildes:
>>> 1 + None
File "", line 1
1 + None
~~^~~~~~
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
Since the lines showing the position of the error come before the
exception type and detail, they are not checked by doctest. For
example, the following test would pass, even though it puts the ‘^’
marker in the wrong location:
>>> 1 + None
File "", line 1
1 + None
^~~~~~~~
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
---------- Footnotes ----------
(1) Examples containing both expected output and an exception are not
supported. Trying to guess where one ends and the other begins is too
error-prone, and that also makes for a confusing test.
File: python.info, Node: Option Flags, Next: Directives, Prev: What About Exceptions?, Up: How It Works
5.27.4.9 Option Flags
.....................
A number of option flags control various aspects of doctest’s behavior.
Symbolic names for the flags are supplied as module constants, which can
be *note bitwise ORed: 2071. together and passed to various functions.
The names can also be used in *note doctest directives: 4005, and may be
passed to the doctest command line interface via the ‘-o’ option.
The first group of options define test semantics, controlling aspects of
how doctest decides whether actual output matches an example’s expected
output:
-- Data: doctest.DONT_ACCEPT_TRUE_FOR_1
By default, if an expected output block contains just ‘1’, an
actual output block containing just ‘1’ or just ‘True’ is
considered to be a match, and similarly for ‘0’ versus ‘False’.
When *note DONT_ACCEPT_TRUE_FOR_1: 400d. is specified, neither
substitution is allowed. The default behavior caters to that
Python changed the return type of many functions from integer to
boolean; doctests expecting “little integer” output still work in
these cases. This option will probably go away, but not for
several years.
-- Data: doctest.DONT_ACCEPT_BLANKLINE
By default, if an expected output block contains a line containing
only the string ‘’, then that line will match a blank
line in the actual output. Because a genuinely blank line delimits
the expected output, this is the only way to communicate that a
blank line is expected. When *note DONT_ACCEPT_BLANKLINE: 400e. is
specified, this substitution is not allowed.
-- Data: doctest.NORMALIZE_WHITESPACE
When specified, all sequences of whitespace (blanks and newlines)
are treated as equal. Any sequence of whitespace within the
expected output will match any sequence of whitespace within the
actual output. By default, whitespace must match exactly. *note
NORMALIZE_WHITESPACE: 4004. is especially useful when a line of
expected output is very long, and you want to wrap it across
multiple lines in your source.
-- Data: doctest.ELLIPSIS
When specified, an ellipsis marker (‘...’) in the expected output
can match any substring in the actual output. This includes
substrings that span line boundaries, and empty substrings, so it’s
best to keep usage of this simple. Complicated uses can lead to
the same kinds of “oops, it matched too much!” surprises that ‘.*’
is prone to in regular expressions.
-- Data: doctest.IGNORE_EXCEPTION_DETAIL
When specified, doctests expecting exceptions pass so long as an
exception of the expected type is raised, even if the details
(message and fully qualified exception name) don’t match.
For example, an example expecting ‘ValueError: 42’ will pass if the
actual exception raised is ‘ValueError: 3*14’, but will fail if,
say, a *note TypeError: 534. is raised instead. It will also
ignore any fully qualified name included before the exception
class, which can vary between implementations and versions of
Python and the code/libraries in use. Hence, all three of these
variations will work with the flag specified:
>>> raise Exception('message')
Traceback (most recent call last):
Exception: message
>>> raise Exception('message')
Traceback (most recent call last):
builtins.Exception: message
>>> raise Exception('message')
Traceback (most recent call last):
__main__.Exception: message
Note that *note ELLIPSIS: 1437. can also be used to ignore the
details of the exception message, but such a test may still fail
based on whether the module name is present or matches exactly.
Changed in version 3.2: *note IGNORE_EXCEPTION_DETAIL: 12fc. now
also ignores any information relating to the module containing the
exception under test.
-- Data: doctest.SKIP
When specified, do not run the example at all. This can be useful
in contexts where doctest examples serve as both documentation and
test cases, and an example should be included for documentation
purposes, but should not be checked. E.g., the example’s output
might be random; or the example might depend on resources which
would be unavailable to the test driver.
The SKIP flag can also be used for temporarily “commenting out”
examples.
-- Data: doctest.COMPARISON_FLAGS
A bitmask or’ing together all the comparison flags above.
The second group of options controls how test failures are reported:
-- Data: doctest.REPORT_UDIFF
When specified, failures that involve multi-line expected and
actual outputs are displayed using a unified diff.
-- Data: doctest.REPORT_CDIFF
When specified, failures that involve multi-line expected and
actual outputs will be displayed using a context diff.
-- Data: doctest.REPORT_NDIFF
When specified, differences are computed by ‘difflib.Differ’, using
the same algorithm as the popular ‘ndiff.py’ utility. This is the
only method that marks differences within lines as well as across
lines. For example, if a line of expected output contains digit
‘1’ where actual output contains letter ‘l’, a line is inserted
with a caret marking the mismatching column positions.
-- Data: doctest.REPORT_ONLY_FIRST_FAILURE
When specified, display the first failing example in each doctest,
but suppress output for all remaining examples. This will prevent
doctest from reporting correct examples that break because of
earlier failures; but it might also hide incorrect examples that
fail independently of the first failure. When *note
REPORT_ONLY_FIRST_FAILURE: 4011. is specified, the remaining
examples are still run, and still count towards the total number of
failures reported; only the output is suppressed.
-- Data: doctest.FAIL_FAST
When specified, exit after the first failing example and don’t
attempt to run the remaining examples. Thus, the number of
failures reported will be at most 1. This flag may be useful
during debugging, since examples after the first failure won’t even
produce debugging output.
-- Data: doctest.REPORTING_FLAGS
A bitmask or’ing together all the reporting flags above.
There is also a way to register new option flag names, though this isn’t
useful unless you intend to extend *note doctest: 3a. internals via
subclassing:
-- Function: doctest.register_optionflag (name)
Create a new option flag with a given name, and return the new
flag’s integer value. *note register_optionflag(): 4013. can be
used when subclassing *note OutputChecker: 4014. or *note
DocTestRunner: 4015. to create new options that are supported by
your subclasses. *note register_optionflag(): 4013. should always
be called using the following idiom:
MY_FLAG = register_optionflag('MY_FLAG')
File: python.info, Node: Directives, Next: Warnings<2>, Prev: Option Flags, Up: How It Works
5.27.4.10 Directives
....................
Doctest directives may be used to modify the *note option flags: f2e.
for an individual example. Doctest directives are special Python
comments following an example’s source code:
directive ::= "#" "doctest:" *note directive_options: 4018.
directive_options ::= *note directive_option: 4019. ("," *note directive_option: 4019.)*
directive_option ::= *note on_or_off: 401a. *note directive_option_name: 401b.
on_or_off ::= "+" | "-"
directive_option_name ::= "DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...
Whitespace is not allowed between the ‘+’ or ‘-’ and the directive
option name. The directive option name can be any of the option flag
names explained above.
An example’s doctest directives modify doctest’s behavior for that
single example. Use ‘+’ to enable the named behavior, or ‘-’ to disable
it.
For example, this test passes:
>>> print(list(range(20))) # doctest: +NORMALIZE_WHITESPACE
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Without the directive it would fail, both because the actual output
doesn’t have two blanks before the single-digit list elements, and
because the actual output is on a single line. This test also passes,
and also requires a directive to do so:
>>> print(list(range(20))) # doctest: +ELLIPSIS
[0, 1, ..., 18, 19]
Multiple directives can be used on a single physical line, separated by
commas:
>>> print(list(range(20))) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
[0, 1, ..., 18, 19]
If multiple directive comments are used for a single example, then they
are combined:
>>> print(list(range(20))) # doctest: +ELLIPSIS
... # doctest: +NORMALIZE_WHITESPACE
[0, 1, ..., 18, 19]
As the previous example shows, you can add ‘...’ lines to your example
containing only directives. This can be useful when an example is too
long for a directive to comfortably fit on the same line:
>>> print(list(range(5)) + list(range(10, 20)) + list(range(30, 40)))
... # doctest: +ELLIPSIS
[0, ..., 4, 10, ..., 19, 30, ..., 39]
Note that since all options are disabled by default, and directives
apply only to the example they appear in, enabling options (via ‘+’ in a
directive) is usually the only meaningful choice. However, option flags
can also be passed to functions that run doctests, establishing
different defaults. In such cases, disabling an option via ‘-’ in a
directive can be useful.
File: python.info, Node: Warnings<2>, Prev: Directives, Up: How It Works
5.27.4.11 Warnings
..................
*note doctest: 3a. is serious about requiring exact matches in expected
output. If even a single character doesn’t match, the test fails. This
will probably surprise you a few times, as you learn exactly what Python
does and doesn’t guarantee about output. For example, when printing a
set, Python doesn’t guarantee that the element is printed in any
particular order, so a test like
>>> foo()
{"spam", "eggs"}
is vulnerable! One workaround is to do
>>> foo() == {"spam", "eggs"}
True
instead. Another is to do
>>> d = sorted(foo())
>>> d
['eggs', 'spam']
There are others, but you get the idea.
Another bad idea is to print things that embed an object address, like
>>> id(1.0) # certain to fail some of the time
7948648
>>> class C: pass
>>> C() # the default repr() for instances embeds an address
The *note ELLIPSIS: 1437. directive gives a nice approach for the last
example:
>>> C() # doctest: +ELLIPSIS
Floating-point numbers are also subject to small output variations
across platforms, because Python defers to the platform C library for
some floating-point calculations, and C libraries vary widely in quality
here.
>>> 1000**0.1 # risky
1.9952623149688797
>>> round(1000**0.1, 9) # safer
1.995262315
>>> print(f'{1000**0.1:.4f}') # much safer
1.9953
Numbers of the form ‘I/2.**J’ are safe across all platforms, and I often
contrive doctest examples to produce numbers of that form:
>>> 3./4 # utterly safe
0.75
Simple fractions are also easier for people to understand, and that
makes for better documentation.
File: python.info, Node: Basic API, Next: Unittest API, Prev: How It Works, Up: doctest — Test interactive Python examples
5.27.4.12 Basic API
...................
The functions *note testmod(): 1436. and *note testfile(): 3ff5. provide
a simple interface to doctest that should be sufficient for most basic
uses. For a less formal introduction to these two functions, see
sections *note Simple Usage; Checking Examples in Docstrings: 3fef. and
*note Simple Usage; Checking Examples in a Text File: 3ff3.
-- Function: doctest.testfile (filename, module_relative=True,
name=None, package=None, globs=None, verbose=None,
report=True, optionflags=0, extraglobs=None,
raise_on_error=False, parser=DocTestParser(), encoding=None)
All arguments except 'filename' are optional, and should be
specified in keyword form.
Test examples in the file named 'filename'. Return
‘(failure_count, test_count)’.
Optional argument 'module_relative' specifies how the filename
should be interpreted:
* If 'module_relative' is ‘True’ (the default), then 'filename'
specifies an OS-independent module-relative path. By default,
this path is relative to the calling module’s directory; but
if the 'package' argument is specified, then it is relative to
that package. To ensure OS-independence, 'filename' should
use ‘/’ characters to separate path segments, and may not be
an absolute path (i.e., it may not begin with ‘/’).
* If 'module_relative' is ‘False’, then 'filename' specifies an
OS-specific path. The path may be absolute or relative;
relative paths are resolved with respect to the current
working directory.
Optional argument 'name' gives the name of the test; by default, or
if ‘None’, ‘os.path.basename(filename)’ is used.
Optional argument 'package' is a Python package or the name of a
Python package whose directory should be used as the base directory
for a module-relative filename. If no package is specified, then
the calling module’s directory is used as the base directory for
module-relative filenames. It is an error to specify 'package' if
'module_relative' is ‘False’.
Optional argument 'globs' gives a dict to be used as the globals
when executing examples. A new shallow copy of this dict is
created for the doctest, so its examples start with a clean slate.
By default, or if ‘None’, a new empty dict is used.
Optional argument 'extraglobs' gives a dict merged into the globals
used to execute examples. This works like *note dict.update():
142e.: if 'globs' and 'extraglobs' have a common key, the
associated value in 'extraglobs' appears in the combined dict. By
default, or if ‘None’, no extra globals are used. This is an
advanced feature that allows parameterization of doctests. For
example, a doctest can be written for a base class, using a generic
name for the class, then reused to test any number of subclasses by
passing an 'extraglobs' dict mapping the generic name to the
subclass to be tested.
Optional argument 'verbose' prints lots of stuff if true, and
prints only failures if false; by default, or if ‘None’, it’s true
if and only if ‘'-v'’ is in *note sys.argv: 1258.
Optional argument 'report' prints a summary at the end when true,
else prints nothing at the end. In verbose mode, the summary is
detailed, else the summary is very brief (in fact, empty if all
tests passed).
Optional argument 'optionflags' (default value ‘0’) takes the *note
bitwise OR: 2071. of option flags. See section *note Option Flags:
f2e.
Optional argument 'raise_on_error' defaults to false. If true, an
exception is raised upon the first failure or unexpected exception
in an example. This allows failures to be post-mortem debugged.
Default behavior is to continue running examples.
Optional argument 'parser' specifies a *note DocTestParser: 4006.
(or subclass) that should be used to extract tests from the files.
It defaults to a normal parser (i.e., ‘DocTestParser()’).
Optional argument 'encoding' specifies an encoding that should be
used to convert the file to unicode.
-- Function: doctest.testmod (m=None, name=None, globs=None,
verbose=None, report=True, optionflags=0, extraglobs=None,
raise_on_error=False, exclude_empty=False)
All arguments are optional, and all except for 'm' should be
specified in keyword form.
Test examples in docstrings in functions and classes reachable from
module 'm' (or module *note __main__: 1. if 'm' is not supplied or
is ‘None’), starting with ‘m.__doc__’.
Also test examples reachable from dict ‘m.__test__’, if it exists.
‘m.__test__’ maps names (strings) to functions, classes and
strings; function and class docstrings are searched for examples;
strings are searched directly, as if they were docstrings.
Only docstrings attached to objects belonging to module 'm' are
searched.
Return ‘(failure_count, test_count)’.
Optional argument 'name' gives the name of the module; by default,
or if ‘None’, ‘m.__name__’ is used.
Optional argument 'exclude_empty' defaults to false. If true,
objects for which no doctests are found are excluded from
consideration. The default is a backward compatibility hack, so
that code still using *note doctest.master.summarize: 401f. in
conjunction with *note testmod(): 1436. continues to get output for
objects with no tests. The 'exclude_empty' argument to the newer
*note DocTestFinder: 1632. constructor defaults to true.
Optional arguments 'extraglobs', 'verbose', 'report',
'optionflags', 'raise_on_error', and 'globs' are the same as for
function *note testfile(): 3ff5. above, except that 'globs'
defaults to ‘m.__dict__’.
-- Function: doctest.run_docstring_examples (f, globs, verbose=False,
name='NoName', compileflags=None, optionflags=0)
Test examples associated with object 'f'; for example, 'f' may be a
string, a module, a function, or a class object.
A shallow copy of dictionary argument 'globs' is used for the
execution context.
Optional argument 'name' is used in failure messages, and defaults
to ‘"NoName"’.
If optional argument 'verbose' is true, output is generated even if
there are no failures. By default, output is generated only in
case of an example failure.
Optional argument 'compileflags' gives the set of flags that should
be used by the Python compiler when running the examples. By
default, or if ‘None’, flags are deduced corresponding to the set
of future features found in 'globs'.
Optional argument 'optionflags' works as for function *note
testfile(): 3ff5. above.
File: python.info, Node: Unittest API, Next: Advanced API, Prev: Basic API, Up: doctest — Test interactive Python examples
5.27.4.13 Unittest API
......................
As your collection of doctest’ed modules grows, you’ll want a way to run
all their doctests systematically. *note doctest: 3a. provides two
functions that can be used to create *note unittest: 106. test suites
from modules and text files containing doctests. To integrate with
*note unittest: 106. test discovery, include a *note load_tests: 4023.
function in your test module:
import unittest
import doctest
import my_module_with_doctests
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
return tests
There are two main functions for creating *note unittest.TestSuite: df3.
instances from text files and modules with doctests:
-- Function: doctest.DocFileSuite (*paths, module_relative=True,
package=None, setUp=None, tearDown=None, globs=None,
optionflags=0, parser=DocTestParser(), encoding=None)
Convert doctest tests from one or more text files to a *note
unittest.TestSuite: df3.
The returned *note unittest.TestSuite: df3. is to be run by the
unittest framework and runs the interactive examples in each file.
If an example in any file fails, then the synthesized unit test
fails, and a *note failureException: 4025. exception is raised
showing the name of the file containing the test and a (sometimes
approximate) line number. If all the examples in a file are
skipped, then the synthesized unit test is also marked as skipped.
Pass one or more paths (as strings) to text files to be examined.
Options may be provided as keyword arguments:
Optional argument 'module_relative' specifies how the filenames in
'paths' should be interpreted:
* If 'module_relative' is ‘True’ (the default), then each
filename in 'paths' specifies an OS-independent
module-relative path. By default, this path is relative to
the calling module’s directory; but if the 'package' argument
is specified, then it is relative to that package. To ensure
OS-independence, each filename should use ‘/’ characters to
separate path segments, and may not be an absolute path (i.e.,
it may not begin with ‘/’).
* If 'module_relative' is ‘False’, then each filename in 'paths'
specifies an OS-specific path. The path may be absolute or
relative; relative paths are resolved with respect to the
current working directory.
Optional argument 'package' is a Python package or the name of a
Python package whose directory should be used as the base directory
for module-relative filenames in 'paths'. If no package is
specified, then the calling module’s directory is used as the base
directory for module-relative filenames. It is an error to specify
'package' if 'module_relative' is ‘False’.
Optional argument 'setUp' specifies a set-up function for the test
suite. This is called before running the tests in each file. The
'setUp' function will be passed a *note DocTest: 16b2. object. The
'setUp' function can access the test globals as the *note globs:
4026. attribute of the test passed.
Optional argument 'tearDown' specifies a tear-down function for the
test suite. This is called after running the tests in each file.
The 'tearDown' function will be passed a *note DocTest: 16b2.
object. The 'tearDown' function can access the test globals as the
*note globs: 4026. attribute of the test passed.
Optional argument 'globs' is a dictionary containing the initial
global variables for the tests. A new copy of this dictionary is
created for each test. By default, 'globs' is a new empty
dictionary.
Optional argument 'optionflags' specifies the default doctest
options for the tests, created by or-ing together individual option
flags. See section *note Option Flags: f2e. See function *note
set_unittest_reportflags(): 4027. below for a better way to set
reporting options.
Optional argument 'parser' specifies a *note DocTestParser: 4006.
(or subclass) that should be used to extract tests from the files.
It defaults to a normal parser (i.e., ‘DocTestParser()’).
Optional argument 'encoding' specifies an encoding that should be
used to convert the file to unicode.
The global ‘__file__’ is added to the globals provided to doctests
loaded from a text file using *note DocFileSuite(): 4024.
-- Function: doctest.DocTestSuite (module=None, globs=None,
extraglobs=None, test_finder=None, setUp=None, tearDown=None,
optionflags=0, checker=None)
Convert doctest tests for a module to a *note unittest.TestSuite:
df3.
The returned *note unittest.TestSuite: df3. is to be run by the
unittest framework and runs each doctest in the module. Each
docstring is run as a separate unit test. If any of the doctests
fail, then the synthesized unit test fails, and a *note
unittest.TestCase.failureException: 4025. exception is raised
showing the name of the file containing the test and a (sometimes
approximate) line number. If all the examples in a docstring are
skipped, then the
Optional argument 'module' provides the module to be tested. It
can be a module object or a (possibly dotted) module name. If not
specified, the module calling this function is used.
Optional argument 'globs' is a dictionary containing the initial
global variables for the tests. A new copy of this dictionary is
created for each test. By default, 'globs' is the module’s *note
__dict__: 1f22.
Optional argument 'extraglobs' specifies an extra set of global
variables, which is merged into 'globs'. By default, no extra
globals are used.
Optional argument 'test_finder' is the *note DocTestFinder: 1632.
object (or a drop-in replacement) that is used to extract doctests
from the module.
Optional arguments 'setUp', 'tearDown', and 'optionflags' are the
same as for function *note DocFileSuite(): 4024. above, but they
are called for each docstring.
This function uses the same search technique as *note testmod():
1436.
Changed in version 3.5: *note DocTestSuite(): df2. returns an empty
*note unittest.TestSuite: df3. if 'module' contains no docstrings
instead of raising *note ValueError: 204.
Under the covers, *note DocTestSuite(): df2. creates a *note
unittest.TestSuite: df3. out of ‘doctest.DocTestCase’ instances, and
‘DocTestCase’ is a subclass of *note unittest.TestCase: 449.
‘DocTestCase’ isn’t documented here (it’s an internal detail), but
studying its code can answer questions about the exact details of *note
unittest: 106. integration.
Similarly, *note DocFileSuite(): 4024. creates a *note
unittest.TestSuite: df3. out of ‘doctest.DocFileCase’ instances, and
‘DocFileCase’ is a subclass of ‘DocTestCase’.
So both ways of creating a *note unittest.TestSuite: df3. run instances
of ‘DocTestCase’. This is important for a subtle reason: when you run
*note doctest: 3a. functions yourself, you can control the ‘doctest’
options in use directly, by passing option flags to ‘doctest’ functions.
However, if you’re writing a *note unittest: 106. framework, ‘unittest’
ultimately controls when and how tests get run. The framework author
typically wants to control ‘doctest’ reporting options (perhaps, e.g.,
specified by command line options), but there’s no way to pass options
through ‘unittest’ to ‘doctest’ test runners.
For this reason, *note doctest: 3a. also supports a notion of ‘doctest’
reporting flags specific to *note unittest: 106. support, via this
function:
-- Function: doctest.set_unittest_reportflags (flags)
Set the *note doctest: 3a. reporting flags to use.
Argument 'flags' takes the *note bitwise OR: 2071. of option flags.
See section *note Option Flags: f2e. Only “reporting flags” can be
used.
This is a module-global setting, and affects all future doctests
run by module *note unittest: 106.: the ‘runTest()’ method of
‘DocTestCase’ looks at the option flags specified for the test case
when the ‘DocTestCase’ instance was constructed. If no reporting
flags were specified (which is the typical and expected case),
‘doctest’’s ‘unittest’ reporting flags are *note bitwise ORed:
2071. into the option flags, and the option flags so augmented are
passed to the *note DocTestRunner: 4015. instance created to run
the doctest. If any reporting flags were specified when the
‘DocTestCase’ instance was constructed, ‘doctest’’s ‘unittest’
reporting flags are ignored.
The value of the *note unittest: 106. reporting flags in effect
before the function was called is returned by the function.
File: python.info, Node: Advanced API, Next: Debugging, Prev: Unittest API, Up: doctest — Test interactive Python examples
5.27.4.14 Advanced API
......................
The basic API is a simple wrapper that’s intended to make doctest easy
to use. It is fairly flexible, and should meet most users’ needs;
however, if you require more fine-grained control over testing, or wish
to extend doctest’s capabilities, then you should use the advanced API.
The advanced API revolves around two container classes, which are used
to store the interactive examples extracted from doctest cases:
* *note Example: 402a.: A single Python *note statement: 278b, paired
with its expected output.
* *note DocTest: 16b2.: A collection of *note Example: 402a.s,
typically extracted from a single docstring or text file.
Additional processing classes are defined to find, parse, and run, and
check doctest examples:
* *note DocTestFinder: 1632.: Finds all docstrings in a given module,
and uses a *note DocTestParser: 4006. to create a *note DocTest:
16b2. from every docstring that contains interactive examples.
* *note DocTestParser: 4006.: Creates a *note DocTest: 16b2. object
from a string (such as an object’s docstring).
* *note DocTestRunner: 4015.: Executes the examples in a *note
DocTest: 16b2, and uses an *note OutputChecker: 4014. to verify
their output.
* *note OutputChecker: 4014.: Compares the actual output from a
doctest example with the expected output, and decides whether they
match.
The relationships among these processing classes are summarized in the
following diagram:
list of:
+------+ +---------+
|module| --DocTestFinder-> | DocTest | --DocTestRunner-> results
+------+ | ^ +---------+ | ^ (printed)
| | | Example | | |
v | | ... | v |
DocTestParser | Example | OutputChecker
+---------+
* Menu:
* DocTest Objects::
* Example Objects::
* DocTestFinder objects::
* DocTestParser objects::
* TestResults objects::
* DocTestRunner objects::
* OutputChecker objects::
File: python.info, Node: DocTest Objects, Next: Example Objects, Up: Advanced API
5.27.4.15 DocTest Objects
.........................
-- Class: doctest.DocTest (examples, globs, name, filename, lineno,
docstring)
A collection of doctest examples that should be run in a single
namespace. The constructor arguments are used to initialize the
attributes of the same names.
*note DocTest: 16b2. defines the following attributes. They are
initialized by the constructor, and should not be modified
directly.
-- Attribute: examples
A list of *note Example: 402a. objects encoding the individual
interactive Python examples that should be run by this test.
-- Attribute: globs
The namespace (aka globals) that the examples should be run
in. This is a dictionary mapping names to values. Any
changes to the namespace made by the examples (such as binding
new variables) will be reflected in *note globs: 4026. after
the test is run.
-- Attribute: name
A string name identifying the *note DocTest: 16b2. Typically,
this is the name of the object or file that the test was
extracted from.
-- Attribute: filename
The name of the file that this *note DocTest: 16b2. was
extracted from; or ‘None’ if the filename is unknown, or if
the ‘DocTest’ was not extracted from a file.
-- Attribute: lineno
The line number within *note filename: 402f. where this *note
DocTest: 16b2. begins, or ‘None’ if the line number is
unavailable. This line number is zero-based with respect to
the beginning of the file.
-- Attribute: docstring
The string that the test was extracted from, or ‘None’ if the
string is unavailable, or if the test was not extracted from a
string.
File: python.info, Node: Example Objects, Next: DocTestFinder objects, Prev: DocTest Objects, Up: Advanced API
5.27.4.16 Example Objects
.........................
-- Class: doctest.Example (source, want, exc_msg=None, lineno=0,
indent=0, options=None)
A single interactive example, consisting of a Python statement and
its expected output. The constructor arguments are used to
initialize the attributes of the same names.
*note Example: 402a. defines the following attributes. They are
initialized by the constructor, and should not be modified
directly.
-- Attribute: source
A string containing the example’s source code. This source
code consists of a single Python statement, and always ends
with a newline; the constructor adds a newline when necessary.
-- Attribute: want
The expected output from running the example’s source code
(either from stdout, or a traceback in case of exception).
*note want: 4034. ends with a newline unless no output is
expected, in which case it’s an empty string. The constructor
adds a newline when necessary.
-- Attribute: exc_msg
The exception message generated by the example, if the example
is expected to generate an exception; or ‘None’ if it is not
expected to generate an exception. This exception message is
compared against the return value of *note
traceback.format_exception_only(): 84a. *note exc_msg: 4035.
ends with a newline unless it’s ‘None’. The constructor adds
a newline if needed.
-- Attribute: lineno
The line number within the string containing this example
where the example begins. This line number is zero-based with
respect to the beginning of the containing string.
-- Attribute: indent
The example’s indentation in the containing string, i.e., the
number of space characters that precede the example’s first
prompt.
-- Attribute: options
A dictionary mapping from option flags to ‘True’ or ‘False’,
which is used to override default options for this example.
Any option flags not contained in this dictionary are left at
their default value (as specified by the *note DocTestRunner:
4015.’s *note optionflags: f2e.). By default, no options are
set.
File: python.info, Node: DocTestFinder objects, Next: DocTestParser objects, Prev: Example Objects, Up: Advanced API
5.27.4.17 DocTestFinder objects
...............................
-- Class: doctest.DocTestFinder (verbose=False, parser=DocTestParser(),
recurse=True, exclude_empty=True)
A processing class used to extract the *note DocTest: 16b2.s that
are relevant to a given object, from its docstring and the
docstrings of its contained objects. *note DocTest: 16b2.s can be
extracted from modules, classes, functions, methods, staticmethods,
classmethods, and properties.
The optional argument 'verbose' can be used to display the objects
searched by the finder. It defaults to ‘False’ (no output).
The optional argument 'parser' specifies the *note DocTestParser:
4006. object (or a drop-in replacement) that is used to extract
doctests from docstrings.
If the optional argument 'recurse' is false, then *note
DocTestFinder.find(): 16fe. will only examine the given object, and
not any contained objects.
If the optional argument 'exclude_empty' is false, then *note
DocTestFinder.find(): 16fe. will include tests for objects with
empty docstrings.
*note DocTestFinder: 1632. defines the following method:
-- Method: find (obj[, name][, module][, globs][, extraglobs])
Return a list of the *note DocTest: 16b2.s that are defined by
'obj'’s docstring, or by any of its contained objects’
docstrings.
The optional argument 'name' specifies the object’s name; this
name will be used to construct names for the returned *note
DocTest: 16b2.s. If 'name' is not specified, then
‘obj.__name__’ is used.
The optional parameter 'module' is the module that contains
the given object. If the module is not specified or is
‘None’, then the test finder will attempt to automatically
determine the correct module. The object’s module is used:
* As a default namespace, if 'globs' is not specified.
* To prevent the DocTestFinder from extracting DocTests
from objects that are imported from other modules.
(Contained objects with modules other than 'module' are
ignored.)
* To find the name of the file containing the object.
* To help find the line number of the object within its
file.
If 'module' is ‘False’, no attempt to find the module will be
made. This is obscure, of use mostly in testing doctest
itself: if 'module' is ‘False’, or is ‘None’ but cannot be
found automatically, then all objects are considered to belong
to the (non-existent) module, so all contained objects will
(recursively) be searched for doctests.
The globals for each *note DocTest: 16b2. is formed by
combining 'globs' and 'extraglobs' (bindings in 'extraglobs'
override bindings in 'globs'). A new shallow copy of the
globals dictionary is created for each ‘DocTest’. If 'globs'
is not specified, then it defaults to the module’s *note
__dict__: 1f22, if specified, or ‘{}’ otherwise. If
'extraglobs' is not specified, then it defaults to ‘{}’.
File: python.info, Node: DocTestParser objects, Next: TestResults objects, Prev: DocTestFinder objects, Up: Advanced API
5.27.4.18 DocTestParser objects
...............................
-- Class: doctest.DocTestParser
A processing class used to extract interactive examples from a
string, and use them to create a *note DocTest: 16b2. object.
*note DocTestParser: 4006. defines the following methods:
-- Method: get_doctest (string, globs, name, filename, lineno)
Extract all doctest examples from the given string, and
collect them into a *note DocTest: 16b2. object.
'globs', 'name', 'filename', and 'lineno' are attributes for
the new ‘DocTest’ object. See the documentation for *note
DocTest: 16b2. for more information.
-- Method: get_examples (string, name='')
Extract all doctest examples from the given string, and return
them as a list of *note Example: 402a. objects. Line numbers
are 0-based. The optional argument 'name' is a name
identifying this string, and is only used for error messages.
-- Method: parse (string, name='')
Divide the given string into examples and intervening text,
and return them as a list of alternating *note Example: 402a.s
and strings. Line numbers for the ‘Example’s are 0-based.
The optional argument 'name' is a name identifying this
string, and is only used for error messages.
File: python.info, Node: TestResults objects, Next: DocTestRunner objects, Prev: DocTestParser objects, Up: Advanced API
5.27.4.19 TestResults objects
.............................
-- Class: doctest.TestResults (failed, attempted)
-- Attribute: failed
Number of failed tests.
-- Attribute: attempted
Number of attempted tests.
-- Attribute: skipped
Number of skipped tests.
Added in version 3.13.
File: python.info, Node: DocTestRunner objects, Next: OutputChecker objects, Prev: TestResults objects, Up: Advanced API
5.27.4.20 DocTestRunner objects
...............................
-- Class: doctest.DocTestRunner (checker=None, verbose=None,
optionflags=0)
A processing class used to execute and verify the interactive
examples in a *note DocTest: 16b2.
The comparison between expected outputs and actual outputs is done
by an *note OutputChecker: 4014. This comparison may be customized
with a number of option flags; see section *note Option Flags: f2e.
for more information. If the option flags are insufficient, then
the comparison may also be customized by passing a subclass of
‘OutputChecker’ to the constructor.
The test runner’s display output can be controlled in two ways.
First, an output function can be passed to *note run(): 1de.; this
function will be called with strings that should be displayed. It
defaults to ‘sys.stdout.write’. If capturing the output is not
sufficient, then the display output can be also customized by
subclassing DocTestRunner, and overriding the methods *note
report_start(): 4046, *note report_success(): 4047, *note
report_unexpected_exception(): 4048, and *note report_failure():
4049.
The optional keyword argument 'checker' specifies the *note
OutputChecker: 4014. object (or drop-in replacement) that should be
used to compare the expected outputs to the actual outputs of
doctest examples.
The optional keyword argument 'verbose' controls the *note
DocTestRunner: 4015.’s verbosity. If 'verbose' is ‘True’, then
information is printed about each example, as it is run. If
'verbose' is ‘False’, then only failures are printed. If 'verbose'
is unspecified, or ‘None’, then verbose output is used iff the
command-line switch ‘-v’ is used.
The optional keyword argument 'optionflags' can be used to control
how the test runner compares expected output to actual output, and
how it displays failures. For more information, see section *note
Option Flags: f2e.
The test runner accumulates statistics. The aggregated number of
attempted, failed and skipped examples is also available via the
*note tries: 404a, *note failures: 404b. and *note skips: 1df.
attributes. The *note run(): 1de. and *note summarize(): 401f.
methods return a *note TestResults: 4041. instance.
*note DocTestRunner: 4015. defines the following methods:
-- Method: report_start (out, test, example)
Report that the test runner is about to process the given
example. This method is provided to allow subclasses of *note
DocTestRunner: 4015. to customize their output; it should not
be called directly.
'example' is the example about to be processed. 'test' is the
test containing 'example'. 'out' is the output function that
was passed to *note DocTestRunner.run(): 1de.
-- Method: report_success (out, test, example, got)
Report that the given example ran successfully. This method
is provided to allow subclasses of *note DocTestRunner: 4015.
to customize their output; it should not be called directly.
'example' is the example about to be processed. 'got' is the
actual output from the example. 'test' is the test containing
'example'. 'out' is the output function that was passed to
*note DocTestRunner.run(): 1de.
-- Method: report_failure (out, test, example, got)
Report that the given example failed. This method is provided
to allow subclasses of *note DocTestRunner: 4015. to customize
their output; it should not be called directly.
'example' is the example about to be processed. 'got' is the
actual output from the example. 'test' is the test containing
'example'. 'out' is the output function that was passed to
*note DocTestRunner.run(): 1de.
-- Method: report_unexpected_exception (out, test, example,
exc_info)
Report that the given example raised an unexpected exception.
This method is provided to allow subclasses of *note
DocTestRunner: 4015. to customize their output; it should not
be called directly.
'example' is the example about to be processed. 'exc_info' is
a tuple containing information about the unexpected exception
(as returned by *note sys.exc_info(): 686.). 'test' is the
test containing 'example'. 'out' is the output function that
was passed to *note DocTestRunner.run(): 1de.
-- Method: run (test, compileflags=None, out=None,
clear_globs=True)
Run the examples in 'test' (a *note DocTest: 16b2. object),
and display the results using the writer function 'out'.
Return a *note TestResults: 4041. instance.
The examples are run in the namespace ‘test.globs’. If
'clear_globs' is true (the default), then this namespace will
be cleared after the test runs, to help with garbage
collection. If you would like to examine the namespace after
the test completes, then use 'clear_globs=False'.
'compileflags' gives the set of flags that should be used by
the Python compiler when running the examples. If not
specified, then it will default to the set of future-import
flags that apply to 'globs'.
The output of each example is checked using the *note
DocTestRunner: 4015.’s output checker, and the results are
formatted by the ‘DocTestRunner.report_*()’ methods.
-- Method: summarize (verbose=None)
Print a summary of all the test cases that have been run by
this DocTestRunner, and return a *note TestResults: 4041.
instance.
The optional 'verbose' argument controls how detailed the
summary is. If the verbosity is not specified, then the *note
DocTestRunner: 4015.’s verbosity is used.
*note DocTestParser: 4006. has the following attributes:
-- Attribute: tries
Number of attempted examples.
-- Attribute: failures
Number of failed examples.
-- Attribute: skips
Number of skipped examples.
Added in version 3.13.
File: python.info, Node: OutputChecker objects, Prev: DocTestRunner objects, Up: Advanced API
5.27.4.21 OutputChecker objects
...............................
-- Class: doctest.OutputChecker
A class used to check the whether the actual output from a doctest
example matches the expected output. *note OutputChecker: 4014.
defines two methods: *note check_output(): 404e, which compares a
given pair of outputs, and returns ‘True’ if they match; and *note
output_difference(): 404f, which returns a string describing the
differences between two outputs.
*note OutputChecker: 4014. defines the following methods:
-- Method: check_output (want, got, optionflags)
Return ‘True’ iff the actual output from an example ('got')
matches the expected output ('want'). These strings are
always considered to match if they are identical; but
depending on what option flags the test runner is using,
several non-exact match types are also possible. See section
*note Option Flags: f2e. for more information about option
flags.
-- Method: output_difference (example, got, optionflags)
Return a string describing the differences between the
expected output for a given example ('example') and the actual
output ('got'). 'optionflags' is the set of option flags used
to compare 'want' and 'got'.
File: python.info, Node: Debugging, Next: Soapbox, Prev: Advanced API, Up: doctest — Test interactive Python examples
5.27.4.22 Debugging
...................
Doctest provides several mechanisms for debugging doctest examples:
* Several functions convert doctests to executable Python programs,
which can be run under the Python debugger, *note pdb: a5.
* The *note DebugRunner: 4052. class is a subclass of *note
DocTestRunner: 4015. that raises an exception for the first failing
example, containing information about that example. This
information can be used to perform post-mortem debugging on the
example.
* The *note unittest: 106. cases generated by *note DocTestSuite():
df2. support the *note debug(): 4053. method defined by *note
unittest.TestCase: 449.
* You can add a call to *note pdb.set_trace(): 237. in a doctest
example, and you’ll drop into the Python debugger when that line is
executed. Then you can inspect current values of variables, and so
on. For example, suppose ‘a.py’ contains just this module
docstring:
"""
>>> def f(x):
... g(x*2)
>>> def g(x):
... print(x+3)
... import pdb; pdb.set_trace()
>>> f(3)
9
"""
Then an interactive Python session may look like this:
>>> import a, doctest
>>> doctest.testmod(a)
--Return--
> (3)g()->None
-> import pdb; pdb.set_trace()
(Pdb) list
1 def g(x):
2 print(x+3)
3 -> import pdb; pdb.set_trace()
[EOF]
(Pdb) p x
6
(Pdb) step
--Return--
> (2)f()->None
-> g(x*2)
(Pdb) list
1 def f(x):
2 -> g(x*2)
[EOF]
(Pdb) p x
3
(Pdb) step
--Return--
> (1)?()->None
-> f(3)
(Pdb) cont
(0, 3)
>>>
Functions that convert doctests to Python code, and possibly run the
synthesized code under the debugger:
-- Function: doctest.script_from_examples (s)
Convert text with examples to a script.
Argument 's' is a string containing doctest examples. The string
is converted to a Python script, where doctest examples in 's' are
converted to regular code, and everything else is converted to
Python comments. The generated script is returned as a string.
For example,
import doctest
print(doctest.script_from_examples(r"""
Set x and y to 1 and 2.
>>> x, y = 1, 2
Print their sum:
>>> print(x+y)
3
"""))
displays:
# Set x and y to 1 and 2.
x, y = 1, 2
#
# Print their sum:
print(x+y)
# Expected:
## 3
This function is used internally by other functions (see below),
but can also be useful when you want to transform an interactive
Python session into a Python script.
-- Function: doctest.testsource (module, name)
Convert the doctest for an object to a script.
Argument 'module' is a module object, or dotted name of a module,
containing the object whose doctests are of interest. Argument
'name' is the name (within the module) of the object with the
doctests of interest. The result is a string, containing the
object’s docstring converted to a Python script, as described for
*note script_from_examples(): 4054. above. For example, if module
‘a.py’ contains a top-level function ‘f()’, then
import a, doctest
print(doctest.testsource(a, "a.f"))
prints a script version of function ‘f()’’s docstring, with
doctests converted to code, and the rest placed in comments.
-- Function: doctest.debug (module, name, pm=False)
Debug the doctests for an object.
The 'module' and 'name' arguments are the same as for function
*note testsource(): 4055. above. The synthesized Python script for
the named object’s docstring is written to a temporary file, and
then that file is run under the control of the Python debugger,
*note pdb: a5.
A shallow copy of ‘module.__dict__’ is used for both local and
global execution context.
Optional argument 'pm' controls whether post-mortem debugging is
used. If 'pm' has a true value, the script file is run directly,
and the debugger gets involved only if the script terminates via
raising an unhandled exception. If it does, then post-mortem
debugging is invoked, via *note pdb.post_mortem(): 13a6, passing
the traceback object from the unhandled exception. If 'pm' is not
specified, or is false, the script is run under the debugger from
the start, via passing an appropriate *note exec(): 17f. call to
*note pdb.run(): 4056.
-- Function: doctest.debug_src (src, pm=False, globs=None)
Debug the doctests in a string.
This is like function *note debug(): 4053. above, except that a
string containing doctest examples is specified directly, via the
'src' argument.
Optional argument 'pm' has the same meaning as in function *note
debug(): 4053. above.
Optional argument 'globs' gives a dictionary to use as both local
and global execution context. If not specified, or ‘None’, an
empty dictionary is used. If specified, a shallow copy of the
dictionary is used.
The *note DebugRunner: 4052. class, and the special exceptions it may
raise, are of most interest to testing framework authors, and will only
be sketched here. See the source code, and especially *note
DebugRunner: 4052.’s docstring (which is a doctest!) for more details:
-- Class: doctest.DebugRunner (checker=None, verbose=None,
optionflags=0)
A subclass of *note DocTestRunner: 4015. that raises an exception
as soon as a failure is encountered. If an unexpected exception
occurs, an *note UnexpectedException: 4058. exception is raised,
containing the test, the example, and the original exception. If
the output doesn’t match, then a *note DocTestFailure: 4059.
exception is raised, containing the test, the example, and the
actual output.
For information about the constructor parameters and methods, see
the documentation for *note DocTestRunner: 4015. in section *note
Advanced API: 4029.
There are two exceptions that may be raised by *note DebugRunner: 4052.
instances:
-- Exception: doctest.DocTestFailure (test, example, got)
An exception raised by *note DocTestRunner: 4015. to signal that a
doctest example’s actual output did not match its expected output.
The constructor arguments are used to initialize the attributes of
the same names.
*note DocTestFailure: 4059. defines the following attributes:
-- Attribute: DocTestFailure.test
The *note DocTest: 16b2. object that was being run when the example
failed.
-- Attribute: DocTestFailure.example
The *note Example: 402a. that failed.
-- Attribute: DocTestFailure.got
The example’s actual output.
-- Exception: doctest.UnexpectedException (test, example, exc_info)
An exception raised by *note DocTestRunner: 4015. to signal that a
doctest example raised an unexpected exception. The constructor
arguments are used to initialize the attributes of the same names.
*note UnexpectedException: 4058. defines the following attributes:
-- Attribute: UnexpectedException.test
The *note DocTest: 16b2. object that was being run when the example
failed.
-- Attribute: UnexpectedException.example
The *note Example: 402a. that failed.
-- Attribute: UnexpectedException.exc_info
A tuple containing information about the unexpected exception, as
returned by *note sys.exc_info(): 686.
File: python.info, Node: Soapbox, Prev: Debugging, Up: doctest — Test interactive Python examples
5.27.4.23 Soapbox
.................
As mentioned in the introduction, *note doctest: 3a. has grown to have
three primary uses:
1. Checking examples in docstrings.
2. Regression testing.
3. Executable documentation / literate testing.
These uses have different requirements, and it is important to
distinguish them. In particular, filling your docstrings with obscure
test cases makes for bad documentation.
When writing a docstring, choose docstring examples with care. There’s
an art to this that needs to be learned—it may not be natural at first.
Examples should add genuine value to the documentation. A good example
can often be worth many words. If done with care, the examples will be
invaluable for your users, and will pay back the time it takes to
collect them many times over as the years go by and things change. I’m
still amazed at how often one of my *note doctest: 3a. examples stops
working after a “harmless” change.
Doctest also makes an excellent tool for regression testing, especially
if you don’t skimp on explanatory text. By interleaving prose and
examples, it becomes much easier to keep track of what’s actually being
tested, and why. When a test fails, good prose can make it much easier
to figure out what the problem is, and how it should be fixed. It’s
true that you could write extensive comments in code-based testing, but
few programmers do. Many have found that using doctest approaches
instead leads to much clearer tests. Perhaps this is simply because
doctest makes writing prose a little easier than writing code, while
writing comments in code is a little harder. I think it goes deeper
than just that: the natural attitude when writing a doctest-based test
is that you want to explain the fine points of your software, and
illustrate them with examples. This in turn naturally leads to test
files that start with the simplest features, and logically progress to
complications and edge cases. A coherent narrative is the result,
instead of a collection of isolated functions that test isolated bits of
functionality seemingly at random. It’s a different attitude, and
produces different results, blurring the distinction between testing and
explaining.
Regression testing is best confined to dedicated objects or files.
There are several options for organizing tests:
* Write text files containing test cases as interactive examples, and
test the files using *note testfile(): 3ff5. or *note
DocFileSuite(): 4024. This is recommended, although is easiest to
do for new projects, designed from the start to use doctest.
* Define functions named ‘_regrtest_topic’ that consist of single
docstrings, containing test cases for the named topics. These
functions can be included in the same file as the module, or
separated out into a separate test file.
* Define a *note __test__: 4001. dictionary mapping from regression
test topics to docstrings containing test cases.
When you have placed your tests in a module, the module can itself be
the test runner. When a test fails, you can arrange for your test
runner to re-run only the failing doctest while you debug the problem.
Here is a minimal example of such a test runner:
if __name__ == '__main__':
import doctest
flags = doctest.REPORT_NDIFF|doctest.FAIL_FAST
if len(sys.argv) > 1:
name = sys.argv[1]
if name in globals():
obj = globals()[name]
else:
obj = __test__[name]
doctest.run_docstring_examples(obj, globals(), name=name,
optionflags=flags)
else:
fail, total = doctest.testmod(optionflags=flags)
print(f"{fail} failures out of {total} tests")
File: python.info, Node: unittest — Unit testing framework, Next: unittest mock — mock object library, Prev: doctest — Test interactive Python examples, Up: Development Tools
5.27.5 ‘unittest’ — Unit testing framework
------------------------------------------
'Source code:' Lib/unittest/__init__.py(1)
__________________________________________________________________
(If you are already familiar with the basic concepts of testing, you
might want to skip to *note the list of assert methods: 4064.)
The *note unittest: 106. unit testing framework was originally inspired
by JUnit and has a similar flavor as major unit testing frameworks in
other languages. It supports test automation, sharing of setup and
shutdown code for tests, aggregation of tests into collections, and
independence of the tests from the reporting framework.
To achieve this, *note unittest: 106. supports some important concepts
in an object-oriented way:
test fixture
A 'test fixture' represents the preparation needed to perform one
or more tests, and any associated cleanup actions. This may
involve, for example, creating temporary or proxy databases,
directories, or starting a server process.
test case
A 'test case' is the individual unit of testing. It checks for a
specific response to a particular set of inputs. *note unittest:
106. provides a base class, *note TestCase: 449, which may be used
to create new test cases.
test suite
A 'test suite' is a collection of test cases, test suites, or both.
It is used to aggregate tests that should be executed together.
test runner
A 'test runner' is a component which orchestrates the execution of
tests and provides the outcome to the user. The runner may use a
graphical interface, a textual interface, or return a special value
to indicate the results of executing the tests.
See also
........
Module *note doctest: 3a.
Another test-support module with a very different flavor.
Simple Smalltalk Testing: With Patterns(2)
Kent Beck’s original paper on testing frameworks using the pattern
shared by *note unittest: 106.
pytest(3)
Third-party unittest framework with a lighter-weight syntax for
writing tests. For example, ‘assert func(10) == 42’.
The Python Testing Tools Taxonomy(4)
An extensive list of Python testing tools including functional
testing frameworks and mock object libraries.
Testing in Python Mailing List(5)
A special-interest-group for discussion of testing, and testing
tools, in Python.
The script ‘Tools/unittestgui/unittestgui.py’ in the Python source
distribution is a GUI tool for test discovery and execution. This is
intended largely for ease of use for those new to unit testing. For
production environments it is recommended that tests be driven by a
continuous integration system such as Buildbot(6), Jenkins(7), GitHub
Actions(8), or AppVeyor(9).
* Menu:
* Basic example::
* Command-Line Interface: Command-Line Interface<3>.
* Test Discovery::
* Organizing test code::
* Re-using old test code::
* Skipping tests and expected failures::
* Distinguishing test iterations using subtests::
* Classes and functions::
* Class and Module Fixtures::
* Signal Handling::
---------- Footnotes ----------
(1)
https://github.com/python/cpython/tree/3.13/Lib/unittest/__init__.py
(2)
https://web.archive.org/web/20150315073817/http://www.xprogramming.com/testfram.htm
(3) https://docs.pytest.org/
(4) https://wiki.python.org/moin/PythonTestingToolsTaxonomy
(5) http://lists.idyll.org/listinfo/testing-in-python
(6) https://buildbot.net/
(7) https://www.jenkins.io/
(8) https://github.com/features/actions
(9) https://www.appveyor.com/
File: python.info, Node: Basic example, Next: Command-Line Interface<3>, Up: unittest — Unit testing framework
5.27.5.1 Basic example
......................
The *note unittest: 106. module provides a rich set of tools for
constructing and running tests. This section demonstrates that a small
subset of the tools suffice to meet the needs of most users.
Here is a short script to test three string methods:
import unittest
class TestStringMethods(unittest.TestCase):
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
def test_isupper(self):
self.assertTrue('FOO'.isupper())
self.assertFalse('Foo'.isupper())
def test_split(self):
s = 'hello world'
self.assertEqual(s.split(), ['hello', 'world'])
# check that s.split fails when the separator is not a string
with self.assertRaises(TypeError):
s.split(2)
if __name__ == '__main__':
unittest.main()
A test case is created by subclassing *note unittest.TestCase: 449. The
three individual tests are defined with methods whose names start with
the letters ‘test’. This naming convention informs the test runner
about which methods represent tests.
The crux of each test is a call to *note assertEqual(): 524. to check
for an expected result; *note assertTrue(): 522. or *note assertFalse():
523. to verify a condition; or *note assertRaises(): 528. to verify that
a specific exception gets raised. These methods are used instead of the
*note assert: 968. statement so the test runner can accumulate all test
results and produce a report.
The *note setUp(): 132c. and *note tearDown(): 132d. methods allow you
to define instructions that will be executed before and after each test
method. They are covered in more detail in the section *note Organizing
test code: 4067.
The final block shows a simple way to run the tests. *note
unittest.main(): fde. provides a command-line interface to the test
script. When run from the command line, the above script produces an
output that looks like this:
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
Passing the ‘-v’ option to your test script will instruct *note
unittest.main(): fde. to enable a higher level of verbosity, and produce
the following output:
test_isupper (__main__.TestStringMethods.test_isupper) ... ok
test_split (__main__.TestStringMethods.test_split) ... ok
test_upper (__main__.TestStringMethods.test_upper) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.001s
OK
The above examples show the most commonly used *note unittest: 106.
features which are sufficient to meet many everyday testing needs. The
remainder of the documentation explores the full feature set from first
principles.
Changed in version 3.11: The behavior of returning a value from a test
method (other than the default ‘None’ value), is now deprecated.
File: python.info, Node: Command-Line Interface<3>, Next: Test Discovery, Prev: Basic example, Up: unittest — Unit testing framework
5.27.5.2 Command-Line Interface
...............................
The unittest module can be used from the command line to run tests from
modules, classes or even individual test methods:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
You can pass in a list with any combination of module names, and fully
qualified class or method names.
Test modules can be specified by file path as well:
python -m unittest tests/test_something.py
This allows you to use the shell filename completion to specify the test
module. The file specified must still be importable as a module. The
path is converted to a module name by removing the ‘.py’ and converting
path separators into ‘.’. If you want to execute a test file that isn’t
importable as a module you should execute the file directly instead.
You can run tests with more detail (higher verbosity) by passing in the
-v flag:
python -m unittest -v test_module
When executed without arguments *note Test Discovery: 406a. is started:
python -m unittest
For a list of all the command-line options:
python -m unittest -h
Changed in version 3.2: In earlier versions it was only possible to run
individual test methods and not modules or classes.
* Menu:
* Command-line options: Command-line options<3>.
File: python.info, Node: Command-line options<3>, Up: Command-Line Interface<3>
5.27.5.3 Command-line options
.............................
‘unittest’ supports these command-line options:
-- Option: -b, --buffer
The standard output and standard error streams are buffered during
the test run. Output during a passing test is discarded. Output
is echoed normally on test fail or error and is added to the
failure messages.
-- Option: -c, --catch
‘Control’-‘C’ during the test run waits for the current test to end
and then reports all the results so far. A second ‘Control’-‘C’
raises the normal *note KeyboardInterrupt: 9d1. exception.
See *note Signal Handling: 406e. for the functions that provide
this functionality.
-- Option: -f, --failfast
Stop the test run on the first error or failure.
-- Option: -k
Only run test methods and classes that match the pattern or
substring. This option may be used multiple times, in which case
all test cases that match any of the given patterns are included.
Patterns that contain a wildcard character (‘*’) are matched
against the test name using *note fnmatch.fnmatchcase(): 18a0.;
otherwise simple case-sensitive substring matching is used.
Patterns are matched against the fully qualified test method name
as imported by the test loader.
For example, ‘-k foo’ matches ‘foo_tests.SomeTest.test_something’,
‘bar_tests.SomeTest.test_foo’, but not
‘bar_tests.FooTest.test_something’.
-- Option: --locals
Show local variables in tracebacks.
-- Option: --durations N
Show the N slowest test cases (N=0 for all).
Added in version 3.2: The command-line options ‘-b’, ‘-c’ and ‘-f’ were
added.
Added in version 3.5: The command-line option ‘--locals’.
Added in version 3.7: The command-line option ‘-k’.
Added in version 3.12: The command-line option ‘--durations’.
The command line can also be used for test discovery, for running all of
the tests in a project or just a subset.
File: python.info, Node: Test Discovery, Next: Organizing test code, Prev: Command-Line Interface<3>, Up: unittest — Unit testing framework
5.27.5.4 Test Discovery
.......................
Added in version 3.2.
Unittest supports simple test discovery. In order to be compatible with
test discovery, all of the test files must be *note modules: 1c56. or
*note packages: 1c67. importable from the top-level directory of the
project (this means that their filenames must be valid *note
identifiers: 1e94.).
Test discovery is implemented in *note TestLoader.discover(): fe0, but
can also be used from the command line. The basic command-line usage
is:
cd project_directory
python -m unittest discover
Note: As a shortcut, ‘python -m unittest’ is the equivalent of
‘python -m unittest discover’. If you want to pass arguments to
test discovery the ‘discover’ sub-command must be used explicitly.
The ‘discover’ sub-command has the following options:
-- Option: -v, --verbose
Verbose output
-- Option: -s, --start-directory directory
Directory to start discovery (‘.’ default)
-- Option: -p, --pattern pattern
Pattern to match test files (‘test*.py’ default)
-- Option: -t, --top-level-directory directory
Top level directory of project (defaults to start directory)
The *note -s: 4076, *note -p: 4078, and *note -t: 407a. options can be
passed in as positional arguments in that order. The following two
command lines are equivalent:
python -m unittest discover -s project_directory -p "*_test.py"
python -m unittest discover project_directory "*_test.py"
As well as being a path it is possible to pass a package name, for
example ‘myproject.subpackage.test’, as the start directory. The
package name you supply will then be imported and its location on the
filesystem will be used as the start directory.
Caution: Test discovery loads tests by importing them. Once test
discovery has found all the test files from the start directory you
specify it turns the paths into package names to import. For
example ‘foo/bar/baz.py’ will be imported as ‘foo.bar.baz’.
If you have a package installed globally and attempt test discovery
on a different copy of the package then the import 'could' happen
from the wrong place. If this happens test discovery will warn you
and exit.
If you supply the start directory as a package name rather than a
path to a directory then discover assumes that whichever location
it imports from is the location you intended, so you will not get
the warning.
Test modules and packages can customize test loading and discovery by
through the *note load_tests protocol: 407c.
Changed in version 3.4: Test discovery supports *note namespace
packages: 1c68. for the start directory. Note that you need to specify
the top level directory too (e.g. ‘python -m unittest discover -s
root/namespace -t root’).
Changed in version 3.11: *note unittest: 106. dropped the *note
namespace packages: 1c68. support in Python 3.11. It has been broken
since Python 3.7. Start directory and subdirectories containing tests
must be regular package that have ‘__init__.py’ file.
Directories containing start directory still can be a namespace package.
In this case, you need to specify start directory as dotted package
name, and target directory explicitly. For example:
# proj/ <-- current directory
# namespace/
# mypkg/
# __init__.py
# test_mypkg.py
python -m unittest discover -s namespace.mypkg -t .
File: python.info, Node: Organizing test code, Next: Re-using old test code, Prev: Test Discovery, Up: unittest — Unit testing framework
5.27.5.5 Organizing test code
.............................
The basic building blocks of unit testing are 'test cases' — single
scenarios that must be set up and checked for correctness. In *note
unittest: 106, test cases are represented by *note unittest.TestCase:
449. instances. To make your own test cases you must write subclasses
of *note TestCase: 449. or use *note FunctionTestCase: 1708.
The testing code of a *note TestCase: 449. instance should be entirely
self contained, such that it can be run either in isolation or in
arbitrary combination with any number of other test cases.
The simplest *note TestCase: 449. subclass will simply implement a test
method (i.e. a method whose name starts with ‘test’) in order to
perform specific testing code:
import unittest
class DefaultWidgetSizeTestCase(unittest.TestCase):
def test_default_widget_size(self):
widget = Widget('The widget')
self.assertEqual(widget.size(), (50, 50))
Note that in order to test something, we use one of the *note assert*
methods: 4064. provided by the *note TestCase: 449. base class. If the
test fails, an exception will be raised with an explanatory message, and
*note unittest: 106. will identify the test case as a 'failure'. Any
other exceptions will be treated as 'errors'.
Tests can be numerous, and their set-up can be repetitive. Luckily, we
can factor out set-up code by implementing a method called *note
setUp(): 132c, which the testing framework will automatically call for
every single test we run:
import unittest
class WidgetTestCase(unittest.TestCase):
def setUp(self):
self.widget = Widget('The widget')
def test_default_widget_size(self):
self.assertEqual(self.widget.size(), (50,50),
'incorrect default size')
def test_widget_resize(self):
self.widget.resize(100,150)
self.assertEqual(self.widget.size(), (100,150),
'wrong size after resize')
Note: The order in which the various tests will be run is
determined by sorting the test method names with respect to the
built-in ordering for strings.
If the *note setUp(): 132c. method raises an exception while the test is
running, the framework will consider the test to have suffered an error,
and the test method will not be executed.
Similarly, we can provide a *note tearDown(): 132d. method that tidies
up after the test method has been run:
import unittest
class WidgetTestCase(unittest.TestCase):
def setUp(self):
self.widget = Widget('The widget')
def tearDown(self):
self.widget.dispose()
If *note setUp(): 132c. succeeded, *note tearDown(): 132d. will be run
whether the test method succeeded or not.
Such a working environment for the testing code is called a 'test
fixture'. A new TestCase instance is created as a unique test fixture
used to execute each individual test method. Thus *note setUp(): 132c,
*note tearDown(): 132d, and ‘__init__()’ will be called once per test.
It is recommended that you use TestCase implementations to group tests
together according to the features they test. *note unittest: 106.
provides a mechanism for this: the 'test suite', represented by *note
unittest: 106.’s *note TestSuite: df3. class. In most cases, calling
*note unittest.main(): fde. will do the right thing and collect all the
module’s test cases for you and execute them.
However, should you want to customize the building of your test suite,
you can do it yourself:
def suite():
suite = unittest.TestSuite()
suite.addTest(WidgetTestCase('test_default_widget_size'))
suite.addTest(WidgetTestCase('test_widget_resize'))
return suite
if __name__ == '__main__':
runner = unittest.TextTestRunner()
runner.run(suite())
You can place the definitions of test cases and test suites in the same
modules as the code they are to test (such as ‘widget.py’), but there
are several advantages to placing the test code in a separate module,
such as ‘test_widget.py’:
* The test module can be run standalone from the command line.
* The test code can more easily be separated from shipped code.
* There is less temptation to change test code to fit the code it
tests without a good reason.
* Test code should be modified much less frequently than the code it
tests.
* Tested code can be refactored more easily.
* Tests for modules written in C must be in separate modules anyway,
so why not be consistent?
* If the testing strategy changes, there is no need to change the
source code.
File: python.info, Node: Re-using old test code, Next: Skipping tests and expected failures, Prev: Organizing test code, Up: unittest — Unit testing framework
5.27.5.6 Re-using old test code
...............................
Some users will find that they have existing test code that they would
like to run from *note unittest: 106, without converting every old test
function to a *note TestCase: 449. subclass.
For this reason, *note unittest: 106. provides a *note FunctionTestCase:
1708. class. This subclass of *note TestCase: 449. can be used to wrap
an existing test function. Set-up and tear-down functions can also be
provided.
Given the following test function:
def testSomething():
something = makeSomething()
assert something.name is not None
# ...
one can create an equivalent test case instance as follows, with
optional set-up and tear-down methods:
testcase = unittest.FunctionTestCase(testSomething,
setUp=makeSomethingDB,
tearDown=deleteSomethingDB)
Note: Even though *note FunctionTestCase: 1708. can be used to
quickly convert an existing test base over to a *note unittest:
106.-based system, this approach is not recommended. Taking the
time to set up proper *note TestCase: 449. subclasses will make
future test refactorings infinitely easier.
In some cases, the existing tests may have been written using the *note
doctest: 3a. module. If so, *note doctest: 3a. provides a
‘DocTestSuite’ class that can automatically build *note
unittest.TestSuite: df3. instances from the existing *note doctest:
3a.-based tests.
File: python.info, Node: Skipping tests and expected failures, Next: Distinguishing test iterations using subtests, Prev: Re-using old test code, Up: unittest — Unit testing framework
5.27.5.7 Skipping tests and expected failures
.............................................
Added in version 3.1.
Unittest supports skipping individual test methods and even whole
classes of tests. In addition, it supports marking a test as an
“expected failure,” a test that is broken and will fail, but shouldn’t
be counted as a failure on a *note TestResult: 115b.
Skipping a test is simply a matter of using the *note skip(): 19d8.
*note decorator: 72c. or one of its conditional variants, calling *note
TestCase.skipTest(): 4082. within a *note setUp(): 132c. or test method,
or raising *note SkipTest: fdf. directly.
Basic skipping looks like this:
class MyTestCase(unittest.TestCase):
@unittest.skip("demonstrating skipping")
def test_nothing(self):
self.fail("shouldn't happen")
@unittest.skipIf(mylib.__version__ < (1, 3),
"not supported in this library version")
def test_format(self):
# Tests that work for only a certain version of the library.
pass
@unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")
def test_windows_support(self):
# windows specific testing code
pass
def test_maybe_skipped(self):
if not external_resource_available():
self.skipTest("external resource not available")
# test code that depends on the external resource
pass
This is the output of running the example above in verbose mode:
test_format (__main__.MyTestCase.test_format) ... skipped 'not supported in this library version'
test_nothing (__main__.MyTestCase.test_nothing) ... skipped 'demonstrating skipping'
test_maybe_skipped (__main__.MyTestCase.test_maybe_skipped) ... skipped 'external resource not available'
test_windows_support (__main__.MyTestCase.test_windows_support) ... skipped 'requires Windows'
----------------------------------------------------------------------
Ran 4 tests in 0.005s
OK (skipped=4)
Classes can be skipped just like methods:
@unittest.skip("showing class skipping")
class MySkippedTestCase(unittest.TestCase):
def test_not_run(self):
pass
*note TestCase.setUp(): 132c. can also skip the test. This is useful
when a resource that needs to be set up is not available.
Expected failures use the *note expectedFailure(): 4083. decorator.
class ExpectedFailureTestCase(unittest.TestCase):
@unittest.expectedFailure
def test_fail(self):
self.assertEqual(1, 0, "broken")
It’s easy to roll your own skipping decorators by making a decorator
that calls *note skip(): 19d8. on the test when it wants it to be
skipped. This decorator skips the test unless the passed object has a
certain attribute:
def skipUnlessHasattr(obj, attr):
if hasattr(obj, attr):
return lambda func: func
return unittest.skip("{!r} doesn't have {!r}".format(obj, attr))
The following decorators and exception implement test skipping and
expected failures:
-- Function: @unittest.skip (reason)
Unconditionally skip the decorated test. 'reason' should describe
why the test is being skipped.
-- Function: @unittest.skipIf (condition, reason)
Skip the decorated test if 'condition' is true.
-- Function: @unittest.skipUnless (condition, reason)
Skip the decorated test unless 'condition' is true.
-- Function: @unittest.expectedFailure
Mark the test as an expected failure or error. If the test fails
or errors in the test function itself (rather than in one of the
'test fixture' methods) then it will be considered a success. If
the test passes, it will be considered a failure.
-- Exception: unittest.SkipTest (reason)
This exception is raised to skip a test.
Usually you can use *note TestCase.skipTest(): 4082. or one of the
skipping decorators instead of raising this directly.
Skipped tests will not have *note setUp(): 132c. or *note tearDown():
132d. run around them. Skipped classes will not have *note
setUpClass(): a4e. or *note tearDownClass(): 132a. run. Skipped modules
will not have ‘setUpModule()’ or ‘tearDownModule()’ run.
File: python.info, Node: Distinguishing test iterations using subtests, Next: Classes and functions, Prev: Skipping tests and expected failures, Up: unittest — Unit testing framework
5.27.5.8 Distinguishing test iterations using subtests
......................................................
Added in version 3.4.
When there are very small differences among your tests, for instance
some parameters, unittest allows you to distinguish them inside the body
of a test method using the *note subTest(): fdc. context manager.
For example, the following test:
class NumbersTest(unittest.TestCase):
def test_even(self):
"""
Test that numbers between 0 and 5 are all even.
"""
for i in range(0, 6):
with self.subTest(i=i):
self.assertEqual(i % 2, 0)
will produce the following output:
======================================================================
FAIL: test_even (__main__.NumbersTest.test_even) (i=1)
Test that numbers between 0 and 5 are all even.
----------------------------------------------------------------------
Traceback (most recent call last):
File "subtests.py", line 11, in test_even
self.assertEqual(i % 2, 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 0
======================================================================
FAIL: test_even (__main__.NumbersTest.test_even) (i=3)
Test that numbers between 0 and 5 are all even.
----------------------------------------------------------------------
Traceback (most recent call last):
File "subtests.py", line 11, in test_even
self.assertEqual(i % 2, 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 0
======================================================================
FAIL: test_even (__main__.NumbersTest.test_even) (i=5)
Test that numbers between 0 and 5 are all even.
----------------------------------------------------------------------
Traceback (most recent call last):
File "subtests.py", line 11, in test_even
self.assertEqual(i % 2, 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 0
Without using a subtest, execution would stop after the first failure,
and the error would be less easy to diagnose because the value of ‘i’
wouldn’t be displayed:
======================================================================
FAIL: test_even (__main__.NumbersTest.test_even)
----------------------------------------------------------------------
Traceback (most recent call last):
File "subtests.py", line 32, in test_even
self.assertEqual(i % 2, 0)
AssertionError: 1 != 0
File: python.info, Node: Classes and functions, Next: Class and Module Fixtures, Prev: Distinguishing test iterations using subtests, Up: unittest — Unit testing framework
5.27.5.9 Classes and functions
..............................
This section describes in depth the API of *note unittest: 106.
* Menu:
* Test cases::
* Grouping tests::
* Loading and running tests::
File: python.info, Node: Test cases, Next: Grouping tests, Up: Classes and functions
5.27.5.10 Test cases
....................
-- Class: unittest.TestCase (methodName='runTest')
Instances of the *note TestCase: 449. class represent the logical
test units in the *note unittest: 106. universe. This class is
intended to be used as a base class, with specific tests being
implemented by concrete subclasses. This class implements the
interface needed by the test runner to allow it to drive the tests,
and methods that the test code can use to check for and report
various kinds of failure.
Each instance of *note TestCase: 449. will run a single base
method: the method named 'methodName'. In most uses of *note
TestCase: 449, you will neither change the 'methodName' nor
reimplement the default ‘runTest()’ method.
Changed in version 3.2: *note TestCase: 449. can be instantiated
successfully without providing a 'methodName'. This makes it
easier to experiment with *note TestCase: 449. from the interactive
interpreter.
*note TestCase: 449. instances provide three groups of methods: one
group used to run the test, another used by the test implementation
to check conditions and report failures, and some inquiry methods
allowing information about the test itself to be gathered.
Methods in the first group (running the test) are:
-- Method: setUp ()
Method called to prepare the test fixture. This is called
immediately before calling the test method; other than *note
AssertionError: 6a5. or *note SkipTest: fdf, any exception
raised by this method will be considered an error rather than
a test failure. The default implementation does nothing.
-- Method: tearDown ()
Method called immediately after the test method has been
called and the result recorded. This is called even if the
test method raised an exception, so the implementation in
subclasses may need to be particularly careful about checking
internal state. Any exception, other than *note
AssertionError: 6a5. or *note SkipTest: fdf, raised by this
method will be considered an additional error rather than a
test failure (thus increasing the total number of reported
errors). This method will only be called if the *note
setUp(): 132c. succeeds, regardless of the outcome of the test
method. The default implementation does nothing.
-- Method: setUpClass ()
A class method called before tests in an individual class are
run. ‘setUpClass’ is called with the class as the only
argument and must be decorated as a *note classmethod(): 166.:
@classmethod
def setUpClass(cls):
...
See *note Class and Module Fixtures: 408b. for more details.
Added in version 3.2.
-- Method: tearDownClass ()
A class method called after tests in an individual class have
run. ‘tearDownClass’ is called with the class as the only
argument and must be decorated as a *note classmethod(): 166.:
@classmethod
def tearDownClass(cls):
...
See *note Class and Module Fixtures: 408b. for more details.
Added in version 3.2.
-- Method: run (result=None)
Run the test, collecting the result into the *note TestResult:
115b. object passed as 'result'. If 'result' is omitted or
‘None’, a temporary result object is created (by calling the
*note defaultTestResult(): 408c. method) and used. The result
object is returned to *note run(): 115a.’s caller.
The same effect may be had by simply calling the *note
TestCase: 449. instance.
Changed in version 3.3: Previous versions of ‘run’ did not
return the result. Neither did calling an instance.
-- Method: skipTest (reason)
Calling this during a test method or *note setUp(): 132c.
skips the current test. See *note Skipping tests and expected
failures: 4081. for more information.
Added in version 3.1.
-- Method: subTest (msg=None, **params)
Return a context manager which executes the enclosed code
block as a subtest. 'msg' and 'params' are optional,
arbitrary values which are displayed whenever a subtest fails,
allowing you to identify them clearly.
A test case can contain any number of subtest declarations,
and they can be arbitrarily nested.
See *note Distinguishing test iterations using subtests: fdd.
for more information.
Added in version 3.4.
-- Method: debug ()
Run the test without collecting the result. This allows
exceptions raised by the test to be propagated to the caller,
and can be used to support running tests under a debugger.
The *note TestCase: 449. class provides several assert methods to
check for and report failures. The following table lists the most
commonly used methods (see the tables below for more assert
methods):
Method Checks that New in
----------------------------------------------------------------------------------------------------
*note assertEqual(a, b): 524. ‘a == b’
*note assertNotEqual(a, b): 525. ‘a != b’
*note assertTrue(x): 522. ‘bool(x) is True’
*note assertFalse(x): 523. ‘bool(x) is False’
*note assertIs(a, b): 132e. ‘a is b’ 3.1
*note assertIsNot(a, b): 132f. ‘a is not b’ 3.1
*note assertIsNone(x): 127d. ‘x is None’ 3.1
*note assertIsNotNone(x): 127e. ‘x is not None’ 3.1
*note assertIn(a, b): 1337. ‘a in b’ 3.1
*note assertNotIn(a, b): 1338. ‘a not in b’ 3.1
*note assertIsInstance(a, b): 1330. ‘isinstance(a, b)’ 3.2
*note assertNotIsInstance(a, b): 1331. ‘not isinstance(a, b)’ 3.2
All the assert methods accept a 'msg' argument that, if specified,
is used as the error message on failure (see also *note
longMessage: 1329.). Note that the 'msg' keyword argument can be
passed to *note assertRaises(): 528, *note assertRaisesRegex():
52a, *note assertWarns(): 1158, *note assertWarnsRegex(): 1159.
only when they are used as a context manager.
-- Method: assertEqual (first, second, msg=None)
Test that 'first' and 'second' are equal. If the values do
not compare equal, the test will fail.
In addition, if 'first' and 'second' are the exact same type
and one of list, tuple, dict, set, frozenset or str or any
type that a subclass registers with *note
addTypeEqualityFunc(): 133b. the type-specific equality
function will be called in order to generate a more useful
default error message (see also the *note list of
type-specific methods: 408d.).
Changed in version 3.1: Added the automatic calling of
type-specific equality function.
Changed in version 3.2: *note assertMultiLineEqual(): 1336.
added as the default type equality function for comparing
strings.
-- Method: assertNotEqual (first, second, msg=None)
Test that 'first' and 'second' are not equal. If the values
do compare equal, the test will fail.
-- Method: assertTrue (expr, msg=None)
-- Method: assertFalse (expr, msg=None)
Test that 'expr' is true (or false).
Note that this is equivalent to ‘bool(expr) is True’ and not
to ‘expr is True’ (use ‘assertIs(expr, True)’ for the latter).
This method should also be avoided when more specific methods
are available (e.g. ‘assertEqual(a, b)’ instead of
‘assertTrue(a == b)’), because they provide a better error
message in case of failure.
-- Method: assertIs (first, second, msg=None)
-- Method: assertIsNot (first, second, msg=None)
Test that 'first' and 'second' are (or are not) the same
object.
Added in version 3.1.
-- Method: assertIsNone (expr, msg=None)
-- Method: assertIsNotNone (expr, msg=None)
Test that 'expr' is (or is not) ‘None’.
Added in version 3.1.
-- Method: assertIn (member, container, msg=None)
-- Method: assertNotIn (member, container, msg=None)
Test that 'member' is (or is not) in 'container'.
Added in version 3.1.
-- Method: assertIsInstance (obj, cls, msg=None)
-- Method: assertNotIsInstance (obj, cls, msg=None)
Test that 'obj' is (or is not) an instance of 'cls' (which can
be a class or a tuple of classes, as supported by *note
isinstance(): 43d.). To check for the exact type, use *note
assertIs(type(obj), cls): 132e.
Added in version 3.2.
It is also possible to check the production of exceptions,
warnings, and log messages using the following methods:
Method Checks that New in
--------------------------------------------------------------------------------------------------------------------------
*note assertRaises(exc, fun, *args, **kwds): 528. ‘fun(*args, **kwds)’ raises 'exc'
*note assertRaisesRegex(exc, r, fun, *args, **kwds): 52a. ‘fun(*args, **kwds)’ raises 'exc' and 3.1
the message matches regex 'r'
*note assertWarns(warn, fun, *args, **kwds): 1158. ‘fun(*args, **kwds)’ raises 'warn' 3.2
*note assertWarnsRegex(warn, r, fun, *args, **kwds): 1159. ‘fun(*args, **kwds)’ raises 'warn' and 3.2
the message matches regex 'r'
*note assertLogs(logger, level): 855. The ‘with’ block logs on 'logger' with 3.4
minimum 'level'
*note assertNoLogs(logger, level): 854. The ‘with’ block does not log on 3.10
'logger' with minimum 'level'
-- Method: assertRaises (exception, callable, *args, **kwds)
-- Method: assertRaises (exception, *, msg=None)
Test that an exception is raised when 'callable' is called
with any positional or keyword arguments that are also passed
to *note assertRaises(): 528. The test passes if 'exception'
is raised, is an error if another exception is raised, or
fails if no exception is raised. To catch any of a group of
exceptions, a tuple containing the exception classes may be
passed as 'exception'.
If only the 'exception' and possibly the 'msg' arguments are
given, return a context manager so that the code under test
can be written inline rather than as a function:
with self.assertRaises(SomeException):
do_something()
When used as a context manager, *note assertRaises(): 528.
accepts the additional keyword argument 'msg'.
The context manager will store the caught exception object in
its ‘exception’ attribute. This can be useful if the
intention is to perform additional checks on the exception
raised:
with self.assertRaises(SomeException) as cm:
do_something()
the_exception = cm.exception
self.assertEqual(the_exception.error_code, 3)
Changed in version 3.1: Added the ability to use *note
assertRaises(): 528. as a context manager.
Changed in version 3.2: Added the ‘exception’ attribute.
Changed in version 3.3: Added the 'msg' keyword argument when
used as a context manager.
-- Method: assertRaisesRegex (exception, regex, callable, *args,
**kwds)
-- Method: assertRaisesRegex (exception, regex, *, msg=None)
Like *note assertRaises(): 528. but also tests that 'regex'
matches on the string representation of the raised exception.
'regex' may be a regular expression object or a string
containing a regular expression suitable for use by *note
re.search(): 121f. Examples:
self.assertRaisesRegex(ValueError, "invalid literal for.*XYZ'$",
int, 'XYZ')
or:
with self.assertRaisesRegex(ValueError, 'literal'):
int('XYZ')
Added in version 3.1: Added under the name
‘assertRaisesRegexp’.
Changed in version 3.2: Renamed to *note assertRaisesRegex():
52a.
Changed in version 3.3: Added the 'msg' keyword argument when
used as a context manager.
-- Method: assertWarns (warning, callable, *args, **kwds)
-- Method: assertWarns (warning, *, msg=None)
Test that a warning is triggered when 'callable' is called
with any positional or keyword arguments that are also passed
to *note assertWarns(): 1158. The test passes if 'warning' is
triggered and fails if it isn’t. Any exception is an error.
To catch any of a group of warnings, a tuple containing the
warning classes may be passed as 'warnings'.
If only the 'warning' and possibly the 'msg' arguments are
given, return a context manager so that the code under test
can be written inline rather than as a function:
with self.assertWarns(SomeWarning):
do_something()
When used as a context manager, *note assertWarns(): 1158.
accepts the additional keyword argument 'msg'.
The context manager will store the caught warning object in
its ‘warning’ attribute, and the source line which triggered
the warnings in the ‘filename’ and ‘lineno’ attributes. This
can be useful if the intention is to perform additional checks
on the warning caught:
with self.assertWarns(SomeWarning) as cm:
do_something()
self.assertIn('myfile.py', cm.filename)
self.assertEqual(320, cm.lineno)
This method works regardless of the warning filters in place
when it is called.
Added in version 3.2.
Changed in version 3.3: Added the 'msg' keyword argument when
used as a context manager.
-- Method: assertWarnsRegex (warning, regex, callable, *args,
**kwds)
-- Method: assertWarnsRegex (warning, regex, *, msg=None)
Like *note assertWarns(): 1158. but also tests that 'regex'
matches on the message of the triggered warning. 'regex' may
be a regular expression object or a string containing a
regular expression suitable for use by *note re.search():
121f. Example:
self.assertWarnsRegex(DeprecationWarning,
r'legacy_function\(\) is deprecated',
legacy_function, 'XYZ')
or:
with self.assertWarnsRegex(RuntimeWarning, 'unsafe frobnicating'):
frobnicate('/etc/passwd')
Added in version 3.2.
Changed in version 3.3: Added the 'msg' keyword argument when
used as a context manager.
-- Method: assertLogs (logger=None, level=None)
A context manager to test that at least one message is logged
on the 'logger' or one of its children, with at least the
given 'level'.
If given, 'logger' should be a *note logging.Logger: b4f.
object or a *note str: 447. giving the name of a logger. The
default is the root logger, which will catch all messages that
were not blocked by a non-propagating descendent logger.
If given, 'level' should be either a numeric logging level or
its string equivalent (for example either ‘"ERROR"’ or *note
logging.ERROR: 1cf5.). The default is *note logging.INFO:
1cf3.
The test passes if at least one message emitted inside the
‘with’ block matches the 'logger' and 'level' conditions,
otherwise it fails.
The object returned by the context manager is a recording
helper which keeps tracks of the matching log messages. It
has two attributes:
-- Attribute: records
A list of *note logging.LogRecord: fe1. objects of the
matching log messages.
-- Attribute: output
A list of *note str: 447. objects with the formatted
output of matching messages.
Example:
with self.assertLogs('foo', level='INFO') as cm:
logging.getLogger('foo').info('first message')
logging.getLogger('foo.bar').error('second message')
self.assertEqual(cm.output, ['INFO:foo:first message',
'ERROR:foo.bar:second message'])
Added in version 3.4.
-- Method: assertNoLogs (logger=None, level=None)
A context manager to test that no messages are logged on the
'logger' or one of its children, with at least the given
'level'.
If given, 'logger' should be a *note logging.Logger: b4f.
object or a *note str: 447. giving the name of a logger. The
default is the root logger, which will catch all messages.
If given, 'level' should be either a numeric logging level or
its string equivalent (for example either ‘"ERROR"’ or *note
logging.ERROR: 1cf5.). The default is *note logging.INFO:
1cf3.
Unlike *note assertLogs(): 855, nothing will be returned by
the context manager.
Added in version 3.10.
There are also other methods used to perform more specific checks,
such as:
Method Checks that New in
----------------------------------------------------------------------------------------------------
*note assertAlmostEqual(a, b): 526. ‘round(a-b, 7) == 0’
*note assertNotAlmostEqual(a, b): 527. ‘round(a-b, 7) != 0’
*note assertGreater(a, b): 1332. ‘a > b’ 3.1
*note assertGreaterEqual(a, b): 1333. ‘a >= b’ 3.1
*note assertLess(a, b): 1334. ‘a < b’ 3.1
*note assertLessEqual(a, b): 1335. ‘a <= b’ 3.1
*note assertRegex(s, r): 529. ‘r.search(s)’ 3.1
*note assertNotRegex(s, r): 52b. ‘not r.search(s)’ 3.2
*note assertCountEqual(a, b): 121d. 'a' and 'b' have the same elements 3.2
in the same number, regardless of
their order.
-- Method: assertAlmostEqual (first, second, places=7, msg=None,
delta=None)
-- Method: assertNotAlmostEqual (first, second, places=7,
msg=None, delta=None)
Test that 'first' and 'second' are approximately (or not
approximately) equal by computing the difference, rounding to
the given number of decimal 'places' (default 7), and
comparing to zero. Note that these methods round the values
to the given number of 'decimal places' (i.e. like the *note
round(): 12cd. function) and not 'significant digits'.
If 'delta' is supplied instead of 'places' then the difference
between 'first' and 'second' must be less or equal to (or
greater than) 'delta'.
Supplying both 'delta' and 'places' raises a *note TypeError:
534.
Changed in version 3.2: *note assertAlmostEqual(): 526.
automatically considers almost equal objects that compare
equal. *note assertNotAlmostEqual(): 527. automatically fails
if the objects compare equal. Added the 'delta' keyword
argument.
-- Method: assertGreater (first, second, msg=None)
-- Method: assertGreaterEqual (first, second, msg=None)
-- Method: assertLess (first, second, msg=None)
-- Method: assertLessEqual (first, second, msg=None)
Test that 'first' is respectively >, >=, < or <= than 'second'
depending on the method name. If not, the test will fail:
>>> self.assertGreaterEqual(3, 4)
AssertionError: "3" unexpectedly not greater than or equal to "4"
Added in version 3.1.
-- Method: assertRegex (text, regex, msg=None)
-- Method: assertNotRegex (text, regex, msg=None)
Test that a 'regex' search matches (or does not match) 'text'.
In case of failure, the error message will include the pattern
and the 'text' (or the pattern and the part of 'text' that
unexpectedly matched). 'regex' may be a regular expression
object or a string containing a regular expression suitable
for use by *note re.search(): 121f.
Added in version 3.1: Added under the name
‘assertRegexpMatches’.
Changed in version 3.2: The method ‘assertRegexpMatches()’ has
been renamed to *note assertRegex(): 529.
Added in version 3.2: *note assertNotRegex(): 52b.
-- Method: assertCountEqual (first, second, msg=None)
Test that sequence 'first' contains the same elements as
'second', regardless of their order. When they don’t, an
error message listing the differences between the sequences
will be generated.
Duplicate elements are 'not' ignored when comparing 'first'
and 'second'. It verifies whether each element has the same
count in both sequences. Equivalent to:
‘assertEqual(Counter(list(first)), Counter(list(second)))’ but
works with sequences of unhashable objects as well.
Added in version 3.2.
The *note assertEqual(): 524. method dispatches the equality check
for objects of the same type to different type-specific methods.
These methods are already implemented for most of the built-in
types, but it’s also possible to register new methods using *note
addTypeEqualityFunc(): 133b.:
-- Method: addTypeEqualityFunc (typeobj, function)
Registers a type-specific method called by *note
assertEqual(): 524. to check if two objects of exactly the
same 'typeobj' (not subclasses) compare equal. 'function'
must take two positional arguments and a third msg=None
keyword argument just as *note assertEqual(): 524. does. It
must raise *note self.failureException(msg): 4025. when
inequality between the first two parameters is detected –
possibly providing useful information and explaining the
inequalities in details in the error message.
Added in version 3.1.
The list of type-specific methods automatically used by *note
assertEqual(): 524. are summarized in the following table. Note
that it’s usually not necessary to invoke these methods directly.
Method Used to compare New in
---------------------------------------------------------------------------------------------------
*note assertMultiLineEqual(a, b): 1336. strings 3.1
*note assertSequenceEqual(a, b): 127c. sequences 3.1
*note assertListEqual(a, b): 127a. lists 3.1
*note assertTupleEqual(a, b): 127b. tuples 3.1
*note assertSetEqual(a, b): 1278. sets or frozensets 3.1
*note assertDictEqual(a, b): 1279. dicts 3.1
-- Method: assertMultiLineEqual (first, second, msg=None)
Test that the multiline string 'first' is equal to the string
'second'. When not equal a diff of the two strings
highlighting the differences will be included in the error
message. This method is used by default when comparing
strings with *note assertEqual(): 524.
Added in version 3.1.
-- Method: assertSequenceEqual (first, second, msg=None,
seq_type=None)
Tests that two sequences are equal. If a 'seq_type' is
supplied, both 'first' and 'second' must be instances of
'seq_type' or a failure will be raised. If the sequences are
different an error message is constructed that shows the
difference between the two.
This method is not called directly by *note assertEqual():
524, but it’s used to implement *note assertListEqual(): 127a.
and *note assertTupleEqual(): 127b.
Added in version 3.1.
-- Method: assertListEqual (first, second, msg=None)
-- Method: assertTupleEqual (first, second, msg=None)
Tests that two lists or tuples are equal. If not, an error
message is constructed that shows only the differences between
the two. An error is also raised if either of the parameters
are of the wrong type. These methods are used by default when
comparing lists or tuples with *note assertEqual(): 524.
Added in version 3.1.
-- Method: assertSetEqual (first, second, msg=None)
Tests that two sets are equal. If not, an error message is
constructed that lists the differences between the sets. This
method is used by default when comparing sets or frozensets
with *note assertEqual(): 524.
Fails if either of 'first' or 'second' does not have a
‘set.difference()’ method.
Added in version 3.1.
-- Method: assertDictEqual (first, second, msg=None)
Test that two dictionaries are equal. If not, an error
message is constructed that shows the differences in the
dictionaries. This method will be used by default to compare
dictionaries in calls to *note assertEqual(): 524.
Added in version 3.1.
Finally the *note TestCase: 449. provides the following methods and
attributes:
-- Method: fail (msg=None)
Signals a test failure unconditionally, with 'msg' or ‘None’
for the error message.
-- Attribute: failureException
This class attribute gives the exception raised by the test
method. If a test framework needs to use a specialized
exception, possibly to carry additional information, it must
subclass this exception in order to “play fair” with the
framework. The initial value of this attribute is *note
AssertionError: 6a5.
-- Attribute: longMessage
This class attribute determines what happens when a custom
failure message is passed as the msg argument to an assertXYY
call that fails. ‘True’ is the default value. In this case,
the custom message is appended to the end of the standard
failure message. When set to ‘False’, the custom message
replaces the standard message.
The class setting can be overridden in individual test methods
by assigning an instance attribute, self.longMessage, to
‘True’ or ‘False’ before calling the assert methods.
The class setting gets reset before each test call.
Added in version 3.1.
-- Attribute: maxDiff
This attribute controls the maximum length of diffs output by
assert methods that report diffs on failure. It defaults to
80*8 characters. Assert methods affected by this attribute
are *note assertSequenceEqual(): 127c. (including all the
sequence comparison methods that delegate to it), *note
assertDictEqual(): 1279. and *note assertMultiLineEqual():
1336.
Setting ‘maxDiff’ to ‘None’ means that there is no maximum
length of diffs.
Added in version 3.2.
Testing frameworks can use the following methods to collect
information on the test:
-- Method: countTestCases ()
Return the number of tests represented by this test object.
For *note TestCase: 449. instances, this will always be ‘1’.
-- Method: defaultTestResult ()
Return an instance of the test result class that should be
used for this test case class (if no other result instance is
provided to the *note run(): 115a. method).
For *note TestCase: 449. instances, this will always be an
instance of *note TestResult: 115b.; subclasses of *note
TestCase: 449. should override this as necessary.
-- Method: id ()
Return a string identifying the specific test case. This is
usually the full name of the test method, including the module
and class name.
-- Method: shortDescription ()
Returns a description of the test, or ‘None’ if no description
has been provided. The default implementation of this method
returns the first line of the test method’s docstring, if
available, or ‘None’.
Changed in version 3.1: In 3.1 this was changed to add the
test name to the short description even in the presence of a
docstring. This caused compatibility issues with unittest
extensions and adding the test name was moved to the *note
TextTestResult: 52c. in Python 3.2.
-- Method: addCleanup (function, /, *args, **kwargs)
Add a function to be called after *note tearDown(): 132d. to
cleanup resources used during the test. Functions will be
called in reverse order to the order they are added (LIFO).
They are called with any arguments and keyword arguments
passed into *note addCleanup(): a81. when they are added.
If *note setUp(): 132c. fails, meaning that *note tearDown():
132d. is not called, then any cleanup functions added will
still be called.
Added in version 3.1.
-- Method: enterContext (cm)
Enter the supplied *note context manager: 5d0. If successful,
also add its *note __exit__(): 12f3. method as a cleanup
function by *note addCleanup(): a81. and return the result of
the *note __enter__(): 5c4. method.
Added in version 3.11.
-- Method: doCleanups ()
This method is called unconditionally after *note tearDown():
132d, or after *note setUp(): 132c. if *note setUp(): 132c.
raises an exception.
It is responsible for calling all the cleanup functions added
by *note addCleanup(): a81. If you need cleanup functions to
be called 'prior' to *note tearDown(): 132d. then you can call
*note doCleanups(): 132b. yourself.
*note doCleanups(): 132b. pops methods off the stack of
cleanup functions one at a time, so it can be called at any
time.
Added in version 3.1.
-- Method: classmethod addClassCleanup (function, /, *args,
**kwargs)
Add a function to be called after *note tearDownClass(): 132a.
to cleanup resources used during the test class. Functions
will be called in reverse order to the order they are added
(LIFO). They are called with any arguments and keyword
arguments passed into *note addClassCleanup(): a4d. when they
are added.
If *note setUpClass(): a4e. fails, meaning that *note
tearDownClass(): 132a. is not called, then any cleanup
functions added will still be called.
Added in version 3.8.
-- Method: classmethod enterClassContext (cm)
Enter the supplied *note context manager: 5d0. If successful,
also add its *note __exit__(): 12f3. method as a cleanup
function by *note addClassCleanup(): a4d. and return the
result of the *note __enter__(): 5c4. method.
Added in version 3.11.
-- Method: classmethod doClassCleanups ()
This method is called unconditionally after *note
tearDownClass(): 132a, or after *note setUpClass(): a4e. if
*note setUpClass(): a4e. raises an exception.
It is responsible for calling all the cleanup functions added
by *note addClassCleanup(): a4d. If you need cleanup
functions to be called 'prior' to *note tearDownClass(): 132a.
then you can call *note doClassCleanups(): 4095. yourself.
*note doClassCleanups(): 4095. pops methods off the stack of
cleanup functions one at a time, so it can be called at any
time.
Added in version 3.8.
-- Class: unittest.IsolatedAsyncioTestCase (methodName='runTest')
This class provides an API similar to *note TestCase: 449. and also
accepts coroutines as test functions.
Added in version 3.8.
-- Attribute: loop_factory
The 'loop_factory' passed to *note asyncio.Runner: 5fc.
Override in subclasses with *note asyncio.EventLoop: 16f6. to
avoid using the asyncio policy system.
Added in version 3.13.
-- Method: async asyncSetUp ()
Method called to prepare the test fixture. This is called
after ‘setUp()’. This is called immediately before calling
the test method; other than *note AssertionError: 6a5. or
*note SkipTest: fdf, any exception raised by this method will
be considered an error rather than a test failure. The
default implementation does nothing.
-- Method: async asyncTearDown ()
Method called immediately after the test method has been
called and the result recorded. This is called before
‘tearDown()’. This is called even if the test method raised
an exception, so the implementation in subclasses may need to
be particularly careful about checking internal state. Any
exception, other than *note AssertionError: 6a5. or *note
SkipTest: fdf, raised by this method will be considered an
additional error rather than a test failure (thus increasing
the total number of reported errors). This method will only
be called if the *note asyncSetUp(): 4097. succeeds,
regardless of the outcome of the test method. The default
implementation does nothing.
-- Method: addAsyncCleanup (function, /, *args, **kwargs)
This method accepts a coroutine that can be used as a cleanup
function.
-- Method: async enterAsyncContext (cm)
Enter the supplied *note asynchronous context manager: 5d3.
If successful, also add its *note __aexit__(): 162a. method as
a cleanup function by *note addAsyncCleanup(): 4099. and
return the result of the *note __aenter__(): 1fd0. method.
Added in version 3.11.
-- Method: run (result=None)
Sets up a new event loop to run the test, collecting the
result into the *note TestResult: 115b. object passed as
'result'. If 'result' is omitted or ‘None’, a temporary
result object is created (by calling the ‘defaultTestResult()’
method) and used. The result object is returned to *note
run(): 409a.’s caller. At the end of the test all the tasks
in the event loop are cancelled.
An example illustrating the order:
from unittest import IsolatedAsyncioTestCase
events = []
class Test(IsolatedAsyncioTestCase):
def setUp(self):
events.append("setUp")
async def asyncSetUp(self):
self._async_connection = await AsyncConnection()
events.append("asyncSetUp")
async def test_response(self):
events.append("test_response")
response = await self._async_connection.get("https://example.com")
self.assertEqual(response.status_code, 200)
self.addAsyncCleanup(self.on_cleanup)
def tearDown(self):
events.append("tearDown")
async def asyncTearDown(self):
await self._async_connection.close()
events.append("asyncTearDown")
async def on_cleanup(self):
events.append("cleanup")
if __name__ == "__main__":
unittest.main()
After running the test, ‘events’ would contain ‘["setUp",
"asyncSetUp", "test_response", "asyncTearDown", "tearDown",
"cleanup"]’.
-- Class: unittest.FunctionTestCase (testFunc, setUp=None,
tearDown=None, description=None)
This class implements the portion of the *note TestCase: 449.
interface which allows the test runner to drive the test, but does
not provide the methods which test code can use to check and report
errors. This is used to create test cases using legacy test code,
allowing it to be integrated into a *note unittest: 106.-based test
framework.
File: python.info, Node: Grouping tests, Next: Loading and running tests, Prev: Test cases, Up: Classes and functions
5.27.5.11 Grouping tests
........................
-- Class: unittest.TestSuite (tests=())
This class represents an aggregation of individual test cases and
test suites. The class presents the interface needed by the test
runner to allow it to be run as any other test case. Running a
*note TestSuite: df3. instance is the same as iterating over the
suite, running each test individually.
If 'tests' is given, it must be an iterable of individual test
cases or other test suites that will be used to build the suite
initially. Additional methods are provided to add test cases and
suites to the collection later on.
*note TestSuite: df3. objects behave much like *note TestCase: 449.
objects, except they do not actually implement a test. Instead,
they are used to aggregate tests into groups of tests that should
be run together. Some additional methods are available to add
tests to *note TestSuite: df3. instances:
-- Method: addTest (test)
Add a *note TestCase: 449. or *note TestSuite: df3. to the
suite.
-- Method: addTests (tests)
Add all the tests from an iterable of *note TestCase: 449. and
*note TestSuite: df3. instances to this test suite.
This is equivalent to iterating over 'tests', calling *note
addTest(): 409d. for each element.
*note TestSuite: df3. shares the following methods with *note
TestCase: 449.:
-- Method: run (result)
Run the tests associated with this suite, collecting the
result into the test result object passed as 'result'. Note
that unlike *note TestCase.run(): 115a, *note TestSuite.run():
409f. requires the result object to be passed in.
-- Method: debug ()
Run the tests associated with this suite without collecting
the result. This allows exceptions raised by the test to be
propagated to the caller and can be used to support running
tests under a debugger.
-- Method: countTestCases ()
Return the number of tests represented by this test object,
including all individual tests and sub-suites.
-- Method: __iter__ ()
Tests grouped by a *note TestSuite: df3. are always accessed
by iteration. Subclasses can lazily provide tests by
overriding ‘__iter__()’. Note that this method may be called
several times on a single suite (for example when counting
tests or comparing for equality) so the tests returned by
repeated iterations before *note TestSuite.run(): 409f. must
be the same for each call iteration. After *note
TestSuite.run(): 409f, callers should not rely on the tests
returned by this method unless the caller uses a subclass that
overrides ‘TestSuite._removeTestAtIndex()’ to preserve test
references.
Changed in version 3.2: In earlier versions the *note
TestSuite: df3. accessed tests directly rather than through
iteration, so overriding ‘__iter__()’ wasn’t sufficient for
providing tests.
Changed in version 3.4: In earlier versions the *note
TestSuite: df3. held references to each *note TestCase: 449.
after *note TestSuite.run(): 409f. Subclasses can restore
that behavior by overriding ‘TestSuite._removeTestAtIndex()’.
In the typical usage of a *note TestSuite: df3. object, the *note
run(): 409f. method is invoked by a ‘TestRunner’ rather than by the
end-user test harness.
File: python.info, Node: Loading and running tests, Prev: Grouping tests, Up: Classes and functions
5.27.5.12 Loading and running tests
...................................
-- Class: unittest.TestLoader
The *note TestLoader: 290. class is used to create test suites from
classes and modules. Normally, there is no need to create an
instance of this class; the *note unittest: 106. module provides an
instance that can be shared as *note unittest.defaultTestLoader:
40a3. Using a subclass or instance, however, allows customization
of some configurable properties.
*note TestLoader: 290. objects have the following attributes:
-- Attribute: errors
A list of the non-fatal errors encountered while loading
tests. Not reset by the loader at any point. Fatal errors
are signalled by the relevant method raising an exception to
the caller. Non-fatal errors are also indicated by a
synthetic test that will raise the original error when run.
Added in version 3.5.
*note TestLoader: 290. objects have the following methods:
-- Method: loadTestsFromTestCase (testCaseClass)
Return a suite of all test cases contained in the *note
TestCase: 449.-derived ‘testCaseClass’.
A test case instance is created for each method named by *note
getTestCaseNames(): 293. By default these are the method
names beginning with ‘test’. If *note getTestCaseNames():
293. returns no methods, but the ‘runTest()’ method is
implemented, a single test case is created for that method
instead.
-- Method: loadTestsFromModule (module, *, pattern=None)
Return a suite of all test cases contained in the given
module. This method searches 'module' for classes derived
from *note TestCase: 449. and creates an instance of the class
for each test method defined for the class.
Note: While using a hierarchy of *note TestCase:
449.-derived classes can be convenient in sharing
fixtures and helper functions, defining test methods on
base classes that are not intended to be instantiated
directly does not play well with this method. Doing so,
however, can be useful when the fixtures are different
and defined in subclasses.
If a module provides a ‘load_tests’ function it will be called
to load the tests. This allows modules to customize test
loading. This is the *note load_tests protocol: 407c. The
'pattern' argument is passed as the third argument to
‘load_tests’.
Changed in version 3.2: Support for ‘load_tests’ added.
Changed in version 3.5: Support for a keyword-only argument
'pattern' has been added.
Changed in version 3.12: The undocumented and unofficial
'use_load_tests' parameter has been removed.
-- Method: loadTestsFromName (name, module=None)
Return a suite of all test cases given a string specifier.
The specifier 'name' is a “dotted name” that may resolve
either to a module, a test case class, a test method within a
test case class, a *note TestSuite: df3. instance, or a
callable object which returns a *note TestCase: 449. or *note
TestSuite: df3. instance. These checks are applied in the
order listed here; that is, a method on a possible test case
class will be picked up as “a test method within a test case
class”, rather than “a callable object”.
For example, if you have a module ‘SampleTests’ containing a
*note TestCase: 449.-derived class ‘SampleTestCase’ with three
test methods (‘test_one()’, ‘test_two()’, and ‘test_three()’),
the specifier ‘'SampleTests.SampleTestCase'’ would cause this
method to return a suite which will run all three test
methods. Using the specifier
‘'SampleTests.SampleTestCase.test_two'’ would cause it to
return a test suite which will run only the ‘test_two()’ test
method. The specifier can refer to modules and packages which
have not been imported; they will be imported as a
side-effect.
The method optionally resolves 'name' relative to the given
'module'.
Changed in version 3.5: If an *note ImportError: 415. or *note
AttributeError: 348. occurs while traversing 'name' then a
synthetic test that raises that error when run will be
returned. These errors are included in the errors accumulated
by self.errors.
-- Method: loadTestsFromNames (names, module=None)
Similar to *note loadTestsFromName(): 1339, but takes a
sequence of names rather than a single name. The return value
is a test suite which supports all the tests defined for each
name.
-- Method: getTestCaseNames (testCaseClass)
Return a sorted sequence of method names found within
'testCaseClass'; this should be a subclass of *note TestCase:
449.
-- Method: discover (start_dir, pattern='test*.py',
top_level_dir=None)
Find all the test modules by recursing into subdirectories
from the specified start directory, and return a TestSuite
object containing them. Only test files that match 'pattern'
will be loaded. (Using shell style pattern matching.) Only
module names that are importable (i.e. are valid Python
identifiers) will be loaded.
All test modules must be importable from the top level of the
project. If the start directory is not the top level
directory then 'top_level_dir' must be specified separately.
If importing a module fails, for example due to a syntax
error, then this will be recorded as a single error and
discovery will continue. If the import failure is due to
*note SkipTest: fdf. being raised, it will be recorded as a
skip instead of an error.
If a package (a directory containing a file named
‘__init__.py’) is found, the package will be checked for a
‘load_tests’ function. If this exists then it will be called
‘package.load_tests(loader, tests, pattern)’. Test discovery
takes care to ensure that a package is only checked for tests
once during an invocation, even if the load_tests function
itself calls ‘loader.discover’.
If ‘load_tests’ exists then discovery does 'not' recurse into
the package, ‘load_tests’ is responsible for loading all tests
in the package.
The pattern is deliberately not stored as a loader attribute
so that packages can continue discovery themselves.
'top_level_dir' is stored internally, and used as a default to
any nested calls to ‘discover()’. That is, if a package’s
‘load_tests’ calls ‘loader.discover()’, it does not need to
pass this argument.
'start_dir' can be a dotted module name as well as a
directory.
Added in version 3.2.
Changed in version 3.4: Modules that raise *note SkipTest:
fdf. on import are recorded as skips, not errors.
Changed in version 3.4: 'start_dir' can be a *note namespace
packages: 1c68.
Changed in version 3.4: Paths are sorted before being imported
so that execution order is the same even if the underlying
file system’s ordering is not dependent on file name.
Changed in version 3.5: Found packages are now checked for
‘load_tests’ regardless of whether their path matches
'pattern', because it is impossible for a package name to
match the default pattern.
Changed in version 3.11: 'start_dir' can not be a *note
namespace packages: 1c68. It has been broken since Python 3.7
and Python 3.11 officially remove it.
Changed in version 3.13: 'top_level_dir' is only stored for
the duration of 'discover' call.
The following attributes of a *note TestLoader: 290. can be
configured either by subclassing or assignment on an instance:
-- Attribute: testMethodPrefix
String giving the prefix of method names which will be
interpreted as test methods. The default value is ‘'test'’.
This affects *note getTestCaseNames(): 293. and all the
‘loadTestsFrom*’ methods.
-- Attribute: sortTestMethodsUsing
Function to be used to compare method names when sorting them
in *note getTestCaseNames(): 293. and all the ‘loadTestsFrom*’
methods.
-- Attribute: suiteClass
Callable object that constructs a test suite from a list of
tests. No methods on the resulting object are needed. The
default value is the *note TestSuite: df3. class.
This affects all the ‘loadTestsFrom*’ methods.
-- Attribute: testNamePatterns
List of Unix shell-style wildcard test name patterns that test
methods have to match to be included in test suites (see ‘-k’
option).
If this attribute is not ‘None’ (the default), all test
methods to be included in test suites must match one of the
patterns in this list. Note that matches are always performed
using *note fnmatch.fnmatchcase(): 18a0, so unlike patterns
passed to the ‘-k’ option, simple substring patterns will have
to be converted using ‘*’ wildcards.
This affects all the ‘loadTestsFrom*’ methods.
Added in version 3.7.
-- Class: unittest.TestResult
This class is used to compile information about which tests have
succeeded and which have failed.
A *note TestResult: 115b. object stores the results of a set of
tests. The *note TestCase: 449. and *note TestSuite: df3. classes
ensure that results are properly recorded; test authors do not need
to worry about recording the outcome of tests.
Testing frameworks built on top of *note unittest: 106. may want
access to the *note TestResult: 115b. object generated by running a
set of tests for reporting purposes; a *note TestResult: 115b.
instance is returned by the ‘TestRunner.run()’ method for this
purpose.
*note TestResult: 115b. instances have the following attributes
that will be of interest when inspecting the results of running a
set of tests:
-- Attribute: errors
A list containing 2-tuples of *note TestCase: 449. instances
and strings holding formatted tracebacks. Each tuple
represents a test which raised an unexpected exception.
-- Attribute: failures
A list containing 2-tuples of *note TestCase: 449. instances
and strings holding formatted tracebacks. Each tuple
represents a test where a failure was explicitly signalled
using the *note assert* methods: 4064.
-- Attribute: skipped
A list containing 2-tuples of *note TestCase: 449. instances
and strings holding the reason for skipping the test.
Added in version 3.1.
-- Attribute: expectedFailures
A list containing 2-tuples of *note TestCase: 449. instances
and strings holding formatted tracebacks. Each tuple
represents an expected failure or error of the test case.
-- Attribute: unexpectedSuccesses
A list containing *note TestCase: 449. instances that were
marked as expected failures, but succeeded.
-- Attribute: collectedDurations
A list containing 2-tuples of test case names and floats
representing the elapsed time of each test which was run.
Added in version 3.12.
-- Attribute: shouldStop
Set to ‘True’ when the execution of tests should stop by *note
stop(): 40ae.
-- Attribute: testsRun
The total number of tests run so far.
-- Attribute: buffer
If set to true, ‘sys.stdout’ and ‘sys.stderr’ will be buffered
in between *note startTest(): 40b1. and *note stopTest():
40b2. being called. Collected output will only be echoed onto
the real ‘sys.stdout’ and ‘sys.stderr’ if the test fails or
errors. Any output is also attached to the failure / error
message.
Added in version 3.2.
-- Attribute: failfast
If set to true *note stop(): 40ae. will be called on the first
failure or error, halting the test run.
Added in version 3.2.
-- Attribute: tb_locals
If set to true then local variables will be shown in
tracebacks.
Added in version 3.5.
-- Method: wasSuccessful ()
Return ‘True’ if all tests run so far have passed, otherwise
returns ‘False’.
Changed in version 3.4: Returns ‘False’ if there were any
*note unexpectedSuccesses: 40ac. from tests marked with the
*note expectedFailure(): 4083. decorator.
-- Method: stop ()
This method can be called to signal that the set of tests
being run should be aborted by setting the *note shouldStop:
40ad. attribute to ‘True’. ‘TestRunner’ objects should
respect this flag and return without running any additional
tests.
For example, this feature is used by the *note TextTestRunner:
1745. class to stop the test framework when the user signals
an interrupt from the keyboard. Interactive tools which
provide ‘TestRunner’ implementations can use this in a similar
manner.
The following methods of the *note TestResult: 115b. class are used
to maintain the internal data structures, and may be extended in
subclasses to support additional reporting requirements. This is
particularly useful in building tools which support interactive
reporting while tests are being run.
-- Method: startTest (test)
Called when the test case 'test' is about to be run.
-- Method: stopTest (test)
Called after the test case 'test' has been executed,
regardless of the outcome.
-- Method: startTestRun ()
Called once before any tests are executed.
Added in version 3.1.
-- Method: stopTestRun ()
Called once after all tests are executed.
Added in version 3.1.
-- Method: addError (test, err)
Called when the test case 'test' raises an unexpected
exception. 'err' is a tuple of the form returned by *note
sys.exc_info(): 686.: ‘(type, value, traceback)’.
The default implementation appends a tuple ‘(test,
formatted_err)’ to the instance’s *note errors: 40a8.
attribute, where 'formatted_err' is a formatted traceback
derived from 'err'.
-- Method: addFailure (test, err)
Called when the test case 'test' signals a failure. 'err' is
a tuple of the form returned by *note sys.exc_info(): 686.:
‘(type, value, traceback)’.
The default implementation appends a tuple ‘(test,
formatted_err)’ to the instance’s *note failures: 40a9.
attribute, where 'formatted_err' is a formatted traceback
derived from 'err'.
-- Method: addSuccess (test)
Called when the test case 'test' succeeds.
The default implementation does nothing.
-- Method: addSkip (test, reason)
Called when the test case 'test' is skipped. 'reason' is the
reason the test gave for skipping.
The default implementation appends a tuple ‘(test, reason)’ to
the instance’s *note skipped: 40aa. attribute.
-- Method: addExpectedFailure (test, err)
Called when the test case 'test' fails or errors, but was
marked with the *note expectedFailure(): 4083. decorator.
The default implementation appends a tuple ‘(test,
formatted_err)’ to the instance’s *note expectedFailures:
40ab. attribute, where 'formatted_err' is a formatted
traceback derived from 'err'.
-- Method: addUnexpectedSuccess (test)
Called when the test case 'test' was marked with the *note
expectedFailure(): 4083. decorator, but succeeded.
The default implementation appends the test to the instance’s
*note unexpectedSuccesses: 40ac. attribute.
-- Method: addSubTest (test, subtest, outcome)
Called when a subtest finishes. 'test' is the test case
corresponding to the test method. 'subtest' is a custom *note
TestCase: 449. instance describing the subtest.
If 'outcome' is *note None: 671, the subtest succeeded.
Otherwise, it failed with an exception where 'outcome' is a
tuple of the form returned by *note sys.exc_info(): 686.:
‘(type, value, traceback)’.
The default implementation does nothing when the outcome is a
success, and records subtest failures as normal failures.
Added in version 3.4.
-- Method: addDuration (test, elapsed)
Called when the test case finishes. 'elapsed' is the time
represented in seconds, and it includes the execution of
cleanup functions.
Added in version 3.12.
-- Class: unittest.TextTestResult (stream, descriptions, verbosity, *,
durations=None)
A concrete implementation of *note TestResult: 115b. used by the
*note TextTestRunner: 1745. Subclasses should accept ‘**kwargs’ to
ensure compatibility as the interface changes.
Added in version 3.2.
Changed in version 3.12: Added the 'durations' keyword parameter.
-- Data: unittest.defaultTestLoader
Instance of the *note TestLoader: 290. class intended to be shared.
If no customization of the *note TestLoader: 290. is needed, this
instance can be used instead of repeatedly creating new instances.
-- Class: unittest.TextTestRunner (stream=None, descriptions=True,
verbosity=1, failfast=False, buffer=False, resultclass=None,
warnings=None, *, tb_locals=False, durations=None)
A basic test runner implementation that outputs results to a
stream. If 'stream' is ‘None’, the default, *note sys.stderr: 939.
is used as the output stream. This class has a few configurable
parameters, but is essentially very simple. Graphical applications
which run test suites should provide alternate implementations.
Such implementations should accept ‘**kwargs’ as the interface to
construct runners changes when features are added to unittest.
By default this runner shows *note DeprecationWarning: 1a5, *note
PendingDeprecationWarning: 8c7, *note ResourceWarning: 246. and
*note ImportWarning: 4f6. even if they are *note ignored by
default: 40b9. This behavior can be overridden using Python’s
‘-Wd’ or ‘-Wa’ options (see *note Warning control: 1d38.) and
leaving 'warnings' to ‘None’.
Changed in version 3.2: Added the 'warnings' parameter.
Changed in version 3.2: The default stream is set to *note
sys.stderr: 939. at instantiation time rather than import time.
Changed in version 3.5: Added the 'tb_locals' parameter.
Changed in version 3.12: Added the 'durations' parameter.
-- Method: _makeResult ()
This method returns the instance of ‘TestResult’ used by *note
run(): 40bb. It is not intended to be called directly, but
can be overridden in subclasses to provide a custom
‘TestResult’.
‘_makeResult()’ instantiates the class or callable passed in
the ‘TextTestRunner’ constructor as the ‘resultclass’
argument. It defaults to *note TextTestResult: 52c. if no
‘resultclass’ is provided. The result class is instantiated
with the following arguments:
stream, descriptions, verbosity
-- Method: run (test)
This method is the main public interface to the
‘TextTestRunner’. This method takes a *note TestSuite: df3.
or *note TestCase: 449. instance. A *note TestResult: 115b.
is created by calling *note _makeResult(): 40ba. and the
test(s) are run and the results printed to stdout.
-- Function: unittest.main (module='__main__', defaultTest=None,
argv=None, testRunner=None,
testLoader=unittest.defaultTestLoader, exit=True, verbosity=1,
failfast=None, catchbreak=None, buffer=None, warnings=None)
A command-line program that loads a set of tests from 'module' and
runs them; this is primarily for making test modules conveniently
executable. The simplest use for this function is to include the
following line at the end of a test script:
if __name__ == '__main__':
unittest.main()
You can run tests with more detailed information by passing in the
verbosity argument:
if __name__ == '__main__':
unittest.main(verbosity=2)
The 'defaultTest' argument is either the name of a single test or
an iterable of test names to run if no test names are specified via
'argv'. If not specified or ‘None’ and no test names are provided
via 'argv', all tests found in 'module' are run.
The 'argv' argument can be a list of options passed to the program,
with the first element being the program name. If not specified or
‘None’, the values of *note sys.argv: 1258. are used.
The 'testRunner' argument can either be a test runner class or an
already created instance of it. By default ‘main’ calls *note
sys.exit(): 133c. with an exit code indicating success (0) or
failure (1) of the tests run. An exit code of 5 indicates that no
tests were run or skipped.
The 'testLoader' argument has to be a *note TestLoader: 290.
instance, and defaults to *note defaultTestLoader: 40a3.
‘main’ supports being used from the interactive interpreter by
passing in the argument ‘exit=False’. This displays the result on
standard output without calling *note sys.exit(): 133c.:
>>> from unittest import main
>>> main(module='test_module', exit=False)
The 'failfast', 'catchbreak' and 'buffer' parameters have the same
effect as the same-name *note command-line options: 406b.
The 'warnings' argument specifies the *note warning filter: 8c8.
that should be used while running the tests. If it’s not
specified, it will remain ‘None’ if a ‘-W’ option is passed to
‘python’ (see *note Warning control: 1d38.), otherwise it will be
set to ‘'default'’.
Calling ‘main’ returns an object with the ‘result’ attribute that
contains the result of the tests run as a *note
unittest.TestResult: 115b.
Changed in version 3.1: The 'exit' parameter was added.
Changed in version 3.2: The 'verbosity', 'failfast', 'catchbreak',
'buffer' and 'warnings' parameters were added.
Changed in version 3.4: The 'defaultTest' parameter was changed to
also accept an iterable of test names.
* Menu:
* load_tests Protocol::
File: python.info, Node: load_tests Protocol, Up: Loading and running tests
5.27.5.13 load_tests Protocol
.............................
Added in version 3.2.
Modules or packages can customize how tests are loaded from them during
normal test runs or test discovery by implementing a function called
‘load_tests’.
If a test module defines ‘load_tests’ it will be called by *note
TestLoader.loadTestsFromModule(): 291. with the following arguments:
load_tests(loader, standard_tests, pattern)
where 'pattern' is passed straight through from ‘loadTestsFromModule’.
It defaults to ‘None’.
It should return a *note TestSuite: df3.
'loader' is the instance of *note TestLoader: 290. doing the loading.
'standard_tests' are the tests that would be loaded by default from the
module. It is common for test modules to only want to add or remove
tests from the standard set of tests. The third argument is used when
loading packages as part of test discovery.
A typical ‘load_tests’ function that loads tests from a specific set of
*note TestCase: 449. classes may look like:
test_cases = (TestCase1, TestCase2, TestCase3)
def load_tests(loader, tests, pattern):
suite = TestSuite()
for test_class in test_cases:
tests = loader.loadTestsFromTestCase(test_class)
suite.addTests(tests)
return suite
If discovery is started in a directory containing a package, either from
the command line or by calling *note TestLoader.discover(): fe0, then
the package ‘__init__.py’ will be checked for ‘load_tests’. If that
function does not exist, discovery will recurse into the package as
though it were just another directory. Otherwise, discovery of the
package’s tests will be left up to ‘load_tests’ which is called with the
following arguments:
load_tests(loader, standard_tests, pattern)
This should return a *note TestSuite: df3. representing all the tests
from the package. (‘standard_tests’ will only contain tests collected
from ‘__init__.py’.)
Because the pattern is passed into ‘load_tests’ the package is free to
continue (and potentially modify) test discovery. A ‘do nothing’
‘load_tests’ function for a test package would look like:
def load_tests(loader, standard_tests, pattern):
# top level directory cached on loader instance
this_dir = os.path.dirname(__file__)
package_tests = loader.discover(start_dir=this_dir, pattern=pattern)
standard_tests.addTests(package_tests)
return standard_tests
Changed in version 3.5: Discovery no longer checks package names for
matching 'pattern' due to the impossibility of package names matching
the default pattern.
File: python.info, Node: Class and Module Fixtures, Next: Signal Handling, Prev: Classes and functions, Up: unittest — Unit testing framework
5.27.5.14 Class and Module Fixtures
...................................
Class and module level fixtures are implemented in *note TestSuite: df3.
When the test suite encounters a test from a new class then
‘tearDownClass()’ from the previous class (if there is one) is called,
followed by ‘setUpClass()’ from the new class.
Similarly if a test is from a different module from the previous test
then ‘tearDownModule’ from the previous module is run, followed by
‘setUpModule’ from the new module.
After all the tests have run the final ‘tearDownClass’ and
‘tearDownModule’ are run.
Note that shared fixtures do not play well with [potential] features
like test parallelization and they break test isolation. They should be
used with care.
The default ordering of tests created by the unittest test loaders is to
group all tests from the same modules and classes together. This will
lead to ‘setUpClass’ / ‘setUpModule’ (etc) being called exactly once per
class and module. If you randomize the order, so that tests from
different modules and classes are adjacent to each other, then these
shared fixture functions may be called multiple times in a single test
run.
Shared fixtures are not intended to work with suites with non-standard
ordering. A ‘BaseTestSuite’ still exists for frameworks that don’t want
to support shared fixtures.
If there are any exceptions raised during one of the shared fixture
functions the test is reported as an error. Because there is no
corresponding test instance an ‘_ErrorHolder’ object (that has the same
interface as a *note TestCase: 449.) is created to represent the error.
If you are just using the standard unittest test runner then this detail
doesn’t matter, but if you are a framework author it may be relevant.
* Menu:
* setUpClass and tearDownClass::
* setUpModule and tearDownModule::
File: python.info, Node: setUpClass and tearDownClass, Next: setUpModule and tearDownModule, Up: Class and Module Fixtures
5.27.5.15 setUpClass and tearDownClass
......................................
These must be implemented as class methods:
import unittest
class Test(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls._connection = createExpensiveConnectionObject()
@classmethod
def tearDownClass(cls):
cls._connection.destroy()
If you want the ‘setUpClass’ and ‘tearDownClass’ on base classes called
then you must call up to them yourself. The implementations in *note
TestCase: 449. are empty.
If an exception is raised during a ‘setUpClass’ then the tests in the
class are not run and the ‘tearDownClass’ is not run. Skipped classes
will not have ‘setUpClass’ or ‘tearDownClass’ run. If the exception is
a *note SkipTest: fdf. exception then the class will be reported as
having been skipped instead of as an error.
File: python.info, Node: setUpModule and tearDownModule, Prev: setUpClass and tearDownClass, Up: Class and Module Fixtures
5.27.5.16 setUpModule and tearDownModule
........................................
These should be implemented as functions:
def setUpModule():
createConnection()
def tearDownModule():
closeConnection()
If an exception is raised in a ‘setUpModule’ then none of the tests in
the module will be run and the ‘tearDownModule’ will not be run. If the
exception is a *note SkipTest: fdf. exception then the module will be
reported as having been skipped instead of as an error.
To add cleanup code that must be run even in the case of an exception,
use ‘addModuleCleanup’:
-- Function: unittest.addModuleCleanup (function, /, *args, **kwargs)
Add a function to be called after ‘tearDownModule()’ to cleanup
resources used during the test class. Functions will be called in
reverse order to the order they are added (LIFO). They are called
with any arguments and keyword arguments passed into *note
addModuleCleanup(): a4c. when they are added.
If ‘setUpModule()’ fails, meaning that ‘tearDownModule()’ is not
called, then any cleanup functions added will still be called.
Added in version 3.8.
-- Function: unittest.enterModuleContext (cm)
Enter the supplied *note context manager: 5d0. If successful, also
add its *note __exit__(): 12f3. method as a cleanup function by
*note addModuleCleanup(): a4c. and return the result of the *note
__enter__(): 5c4. method.
Added in version 3.11.
-- Function: unittest.doModuleCleanups ()
This function is called unconditionally after ‘tearDownModule()’,
or after ‘setUpModule()’ if ‘setUpModule()’ raises an exception.
It is responsible for calling all the cleanup functions added by
*note addModuleCleanup(): a4c. If you need cleanup functions to be
called 'prior' to ‘tearDownModule()’ then you can call *note
doModuleCleanups(): 183c. yourself.
*note doModuleCleanups(): 183c. pops methods off the stack of
cleanup functions one at a time, so it can be called at any time.
Added in version 3.8.
File: python.info, Node: Signal Handling, Prev: Class and Module Fixtures, Up: unittest — Unit testing framework
5.27.5.17 Signal Handling
.........................
Added in version 3.2.
The *note -c/-catch: 1326. command-line option to unittest, along with
the ‘catchbreak’ parameter to *note unittest.main(): fde, provide more
friendly handling of control-C during a test run. With catch break
behavior enabled control-C will allow the currently running test to
complete, and the test run will then end and report all the results so
far. A second control-c will raise a *note KeyboardInterrupt: 9d1. in
the usual way.
The control-c handling signal handler attempts to remain compatible with
code or tests that install their own *note signal.SIGINT: 840. handler.
If the ‘unittest’ handler is called but 'isn’t' the installed *note
signal.SIGINT: 840. handler, i.e. it has been replaced by the system
under test and delegated to, then it calls the default handler. This
will normally be the expected behavior by code that replaces an
installed handler and delegates to it. For individual tests that need
‘unittest’ control-c handling disabled the *note removeHandler(): 1327.
decorator can be used.
There are a few utility functions for framework authors to enable
control-c handling functionality within test frameworks.
-- Function: unittest.installHandler ()
Install the control-c handler. When a *note signal.SIGINT: 840. is
received (usually in response to the user pressing control-c) all
registered results have *note stop(): 40ae. called.
-- Function: unittest.registerResult (result)
Register a *note TestResult: 115b. object for control-c handling.
Registering a result stores a weak reference to it, so it doesn’t
prevent the result from being garbage collected.
Registering a *note TestResult: 115b. object has no side-effects if
control-c handling is not enabled, so test frameworks can
unconditionally register all results they create independently of
whether or not handling is enabled.
-- Function: unittest.removeResult (result)
Remove a registered result. Once a result has been removed then
*note stop(): 40ae. will no longer be called on that result object
in response to a control-c.
-- Function: unittest.removeHandler (function=None)
When called without arguments this function removes the control-c
handler if it has been installed. This function can also be used
as a test decorator to temporarily remove the handler while the
test is being executed:
@unittest.removeHandler
def test_signal_handling(self):
...
File: python.info, Node: unittest mock — mock object library, Next: unittest mock — getting started, Prev: unittest — Unit testing framework, Up: Development Tools
5.27.6 ‘unittest.mock’ — mock object library
--------------------------------------------
Added in version 3.3.
'Source code:' Lib/unittest/mock.py(1)
__________________________________________________________________
*note unittest.mock: 107. is a library for testing in Python. It allows
you to replace parts of your system under test with mock objects and
make assertions about how they have been used.
*note unittest.mock: 107. provides a core *note Mock: a4b. class
removing the need to create a host of stubs throughout your test suite.
After performing an action, you can make assertions about which methods
/ attributes were used and arguments they were called with. You can
also specify return values and set needed attributes in the normal way.
Additionally, mock provides a *note patch(): e8b. decorator that handles
patching module and class level attributes within the scope of a test,
along with *note sentinel: ba0. for creating unique objects. See the
*note quick guide: 40c3. for some examples of how to use *note Mock:
a4b, *note MagicMock: e8a. and *note patch(): e8b.
Mock is designed for use with *note unittest: 106. and is based on the
‘action -> assertion’ pattern instead of ‘record -> replay’ used by many
mocking frameworks.
There is a backport of *note unittest.mock: 107. for earlier versions of
Python, available as mock(2) on PyPI.
* Menu:
* Quick Guide::
* The Mock Class::
* The patchers::
* MagicMock and magic method support::
* Helpers::
* Order of precedence of side_effect, return_value and wraps: Order of precedence of side_effect return_value and wraps.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/unittest/mock.py
(2) https://pypi.org/project/mock/
File: python.info, Node: Quick Guide, Next: The Mock Class, Up: unittest mock — mock object library
5.27.6.1 Quick Guide
....................
*note Mock: a4b. and *note MagicMock: e8a. objects create all attributes
and methods as you access them and store details of how they have been
used. You can configure them, to specify return values or limit what
attributes are available, and then make assertions about how they have
been used:
>>> from unittest.mock import MagicMock
>>> thing = ProductionClass()
>>> thing.method = MagicMock(return_value=3)
>>> thing.method(3, 4, 5, key='value')
3
>>> thing.method.assert_called_with(3, 4, 5, key='value')
*note side_effect: 40c4. allows you to perform side effects, including
raising an exception when a mock is called:
>>> from unittest.mock import Mock
>>> mock = Mock(side_effect=KeyError('foo'))
>>> mock()
Traceback (most recent call last):
...
KeyError: 'foo'
>>> values = {'a': 1, 'b': 2, 'c': 3}
>>> def side_effect(arg):
... return values[arg]
...
>>> mock.side_effect = side_effect
>>> mock('a'), mock('b'), mock('c')
(1, 2, 3)
>>> mock.side_effect = [5, 4, 3, 2, 1]
>>> mock(), mock(), mock()
(5, 4, 3)
Mock has many other ways you can configure it and control its behaviour.
For example the 'spec' argument configures the mock to take its
specification from another object. Attempting to access attributes or
methods on the mock that don’t exist on the spec will fail with an *note
AttributeError: 348.
The *note patch(): e8b. decorator / context manager makes it easy to
mock classes or objects in a module under test. The object you specify
will be replaced with a mock (or other object) during the test and
restored when the test ends:
>>> from unittest.mock import patch
>>> @patch('module.ClassName2')
... @patch('module.ClassName1')
... def test(MockClass1, MockClass2):
... module.ClassName1()
... module.ClassName2()
... assert MockClass1 is module.ClassName1
... assert MockClass2 is module.ClassName2
... assert MockClass1.called
... assert MockClass2.called
...
>>> test()
Note: When you nest patch decorators the mocks are passed in to the
decorated function in the same order they applied (the normal
'Python' order that decorators are applied). This means from the
bottom up, so in the example above the mock for ‘module.ClassName1’
is passed in first.
With *note patch(): e8b. it matters that you patch objects in the
namespace where they are looked up. This is normally
straightforward, but for a quick guide read *note where to patch:
40c5.
As well as a decorator *note patch(): e8b. can be used as a context
manager in a with statement:
>>> with patch.object(ProductionClass, 'method', return_value=None) as mock_method:
... thing = ProductionClass()
... thing.method(1, 2, 3)
...
>>> mock_method.assert_called_once_with(1, 2, 3)
There is also *note patch.dict(): 19f2. for setting values in a
dictionary just during a scope and restoring the dictionary to its
original state when the test ends:
>>> foo = {'key': 'value'}
>>> original = foo.copy()
>>> with patch.dict(foo, {'newkey': 'newvalue'}, clear=True):
... assert foo == {'newkey': 'newvalue'}
...
>>> assert foo == original
Mock supports the mocking of Python *note magic methods: 40c6. The
easiest way of using magic methods is with the *note MagicMock: e8a.
class. It allows you to do things like:
>>> mock = MagicMock()
>>> mock.__str__.return_value = 'foobarbaz'
>>> str(mock)
'foobarbaz'
>>> mock.__str__.assert_called_with()
Mock allows you to assign functions (or other Mock instances) to magic
methods and they will be called appropriately. The *note MagicMock:
e8a. class is just a Mock variant that has all of the magic methods
pre-created for you (well, all the useful ones anyway).
The following is an example of using magic methods with the ordinary
Mock class:
>>> mock = Mock()
>>> mock.__str__ = Mock(return_value='wheeeeee')
>>> str(mock)
'wheeeeee'
For ensuring that the mock objects in your tests have the same api as
the objects they are replacing, you can use *note auto-speccing: 40c7.
Auto-speccing can be done through the 'autospec' argument to patch, or
the *note create_autospec(): 1608. function. Auto-speccing creates mock
objects that have the same attributes and methods as the objects they
are replacing, and any functions and methods (including constructors)
have the same call signature as the real object.
This ensures that your mocks will fail in the same way as your
production code if they are used incorrectly:
>>> from unittest.mock import create_autospec
>>> def function(a, b, c):
... pass
...
>>> mock_function = create_autospec(function, return_value='fishy')
>>> mock_function(1, 2, 3)
'fishy'
>>> mock_function.assert_called_once_with(1, 2, 3)
>>> mock_function('wrong arguments')
Traceback (most recent call last):
...
TypeError: missing a required argument: 'b'
*note create_autospec(): 1608. can also be used on classes, where it
copies the signature of the ‘__init__’ method, and on callable objects
where it copies the signature of the ‘__call__’ method.
File: python.info, Node: The Mock Class, Next: The patchers, Prev: Quick Guide, Up: unittest mock — mock object library
5.27.6.2 The Mock Class
.......................
*note Mock: a4b. is a flexible mock object intended to replace the use
of stubs and test doubles throughout your code. Mocks are callable and
create attributes as new mocks when you access them (1). Accessing the
same attribute will always return the same mock. Mocks record how you
use them, allowing you to make assertions about what your code has done
to them.
*note MagicMock: e8a. is a subclass of *note Mock: a4b. with all the
magic methods pre-created and ready to use. There are also non-callable
variants, useful when you are mocking out objects that aren’t callable:
*note NonCallableMock: 40c9. and *note NonCallableMagicMock: 40ca.
The *note patch(): e8b. decorators makes it easy to temporarily replace
classes in a particular module with a *note Mock: a4b. object. By
default *note patch(): e8b. will create a *note MagicMock: e8a. for you.
You can specify an alternative class of *note Mock: a4b. using the
'new_callable' argument to *note patch(): e8b.
-- Class: unittest.mock.Mock (spec=None, side_effect=None,
return_value=DEFAULT, wraps=None, name=None, spec_set=None,
unsafe=False, **kwargs)
Create a new *note Mock: a4b. object. *note Mock: a4b. takes
several optional arguments that specify the behaviour of the Mock
object:
* 'spec': This can be either a list of strings or an existing
object (a class or instance) that acts as the specification
for the mock object. If you pass in an object then a list of
strings is formed by calling dir on the object (excluding
unsupported magic attributes and methods). Accessing any
attribute not in this list will raise an *note AttributeError:
348.
If 'spec' is an object (rather than a list of strings) then
*note __class__: 1476. returns the class of the spec object.
This allows mocks to pass *note isinstance(): 43d. tests.
* 'spec_set': A stricter variant of 'spec'. If used, attempting
to 'set' or get an attribute on the mock that isn’t on the
object passed as 'spec_set' will raise an *note
AttributeError: 348.
* 'side_effect': A function to be called whenever the Mock is
called. See the *note side_effect: 40c4. attribute. Useful
for raising exceptions or dynamically changing return values.
The function is called with the same arguments as the mock,
and unless it returns *note DEFAULT: 40cb, the return value of
this function is used as the return value.
Alternatively 'side_effect' can be an exception class or
instance. In this case the exception will be raised when the
mock is called.
If 'side_effect' is an iterable then each call to the mock
will return the next value from the iterable.
A 'side_effect' can be cleared by setting it to ‘None’.
* 'return_value': The value returned when the mock is called.
By default this is a new Mock (created on first access). See
the *note return_value: 40cc. attribute.
* 'unsafe': By default, accessing any attribute whose name
starts with 'assert', 'assret', 'asert', 'aseert' or 'assrt'
will raise an *note AttributeError: 348. Passing
‘unsafe=True’ will allow access to these attributes.
Added in version 3.5.
* 'wraps': Item for the mock object to wrap. If 'wraps' is not
‘None’ then calling the Mock will pass the call through to the
wrapped object (returning the real result). Attribute access
on the mock will return a Mock object that wraps the
corresponding attribute of the wrapped object (so attempting
to access an attribute that doesn’t exist will raise an *note
AttributeError: 348.).
If the mock has an explicit 'return_value' set then calls are
not passed to the wrapped object and the 'return_value' is
returned instead.
* 'name': If the mock has a name then it will be used in the
repr of the mock. This can be useful for debugging. The name
is propagated to child mocks.
Mocks can also be called with arbitrary keyword arguments. These
will be used to set attributes on the mock after it is created.
See the *note configure_mock(): 40cd. method for details.
-- Method: assert_called ()
Assert that the mock was called at least once.
>>> mock = Mock()
>>> mock.method()
>>> mock.method.assert_called()
Added in version 3.6.
-- Method: assert_called_once ()
Assert that the mock was called exactly once.
>>> mock = Mock()
>>> mock.method()
>>> mock.method.assert_called_once()
>>> mock.method()
>>> mock.method.assert_called_once()
Traceback (most recent call last):
...
AssertionError: Expected 'method' to have been called once. Called 2 times.
Calls: [call(), call()].
Added in version 3.6.
-- Method: assert_called_with (*args, **kwargs)
This method is a convenient way of asserting that the last
call has been made in a particular way:
>>> mock = Mock()
>>> mock.method(1, 2, 3, test='wow')
>>> mock.method.assert_called_with(1, 2, 3, test='wow')
-- Method: assert_called_once_with (*args, **kwargs)
Assert that the mock was called exactly once and that call was
with the specified arguments.
>>> mock = Mock(return_value=None)
>>> mock('foo', bar='baz')
>>> mock.assert_called_once_with('foo', bar='baz')
>>> mock('other', bar='values')
>>> mock.assert_called_once_with('other', bar='values')
Traceback (most recent call last):
...
AssertionError: Expected 'mock' to be called once. Called 2 times.
Calls: [call('foo', bar='baz'), call('other', bar='values')].
-- Method: assert_any_call (*args, **kwargs)
assert the mock has been called with the specified arguments.
The assert passes if the mock has 'ever' been called, unlike
*note assert_called_with(): 1a20. and *note
assert_called_once_with(): 40ce. that only pass if the call is
the most recent one, and in the case of *note
assert_called_once_with(): 40ce. it must also be the only
call.
>>> mock = Mock(return_value=None)
>>> mock(1, 2, arg='thing')
>>> mock('some', 'thing', 'else')
>>> mock.assert_any_call(1, 2, arg='thing')
-- Method: assert_has_calls (calls, any_order=False)
assert the mock has been called with the specified calls. The
*note mock_calls: 40d0. list is checked for the calls.
If 'any_order' is false then the calls must be sequential.
There can be extra calls before or after the specified calls.
If 'any_order' is true then the calls can be in any order, but
they must all appear in *note mock_calls: 40d0.
>>> mock = Mock(return_value=None)
>>> mock(1)
>>> mock(2)
>>> mock(3)
>>> mock(4)
>>> calls = [call(2), call(3)]
>>> mock.assert_has_calls(calls)
>>> calls = [call(4), call(2), call(3)]
>>> mock.assert_has_calls(calls, any_order=True)
-- Method: assert_not_called ()
Assert the mock was never called.
>>> m = Mock()
>>> m.hello.assert_not_called()
>>> obj = m.hello()
>>> m.hello.assert_not_called()
Traceback (most recent call last):
...
AssertionError: Expected 'hello' to not have been called. Called 1 times.
Calls: [call()].
Added in version 3.5.
-- Method: reset_mock (*, return_value=False, side_effect=False)
The reset_mock method resets all the call attributes on a mock
object:
>>> mock = Mock(return_value=None)
>>> mock('hello')
>>> mock.called
True
>>> mock.reset_mock()
>>> mock.called
False
This can be useful where you want to make a series of
assertions that reuse the same object.
'return_value' parameter when set to ‘True’ resets *note
return_value: 40cc.:
>>> mock = Mock(return_value=5)
>>> mock('hello')
5
>>> mock.reset_mock(return_value=True)
>>> mock('hello')
'side_effect' parameter when set to ‘True’ resets *note
side_effect: 40c4.:
>>> mock = Mock(side_effect=ValueError)
>>> mock('hello')
Traceback (most recent call last):
...
ValueError
>>> mock.reset_mock(side_effect=True)
>>> mock('hello')
Note that *note reset_mock(): cfa. 'doesn’t' clear the *note
return_value: 40cc, *note side_effect: 40c4. or any child
attributes you have set using normal assignment by default.
Child mocks are reset as well.
Changed in version 3.6: Added two keyword-only arguments to
the reset_mock function.
-- Method: mock_add_spec (spec, spec_set=False)
Add a spec to a mock. 'spec' can either be an object or a
list of strings. Only attributes on the 'spec' can be fetched
as attributes from the mock.
If 'spec_set' is true then only attributes on the spec can be
set.
-- Method: attach_mock (mock, attribute)
Attach a mock as an attribute of this one, replacing its name
and parent. Calls to the attached mock will be recorded in
the *note method_calls: 40d2. and *note mock_calls: 40d0.
attributes of this one.
-- Method: configure_mock (**kwargs)
Set attributes on the mock through keyword arguments.
Attributes plus return values and side effects can be set on
child mocks using standard dot notation and unpacking a
dictionary in the method call:
>>> mock = Mock()
>>> attrs = {'method.return_value': 3, 'other.side_effect': KeyError}
>>> mock.configure_mock(**attrs)
>>> mock.method()
3
>>> mock.other()
Traceback (most recent call last):
...
KeyError
The same thing can be achieved in the constructor call to
mocks:
>>> attrs = {'method.return_value': 3, 'other.side_effect': KeyError}
>>> mock = Mock(some_attribute='eggs', **attrs)
>>> mock.some_attribute
'eggs'
>>> mock.method()
3
>>> mock.other()
Traceback (most recent call last):
...
KeyError
*note configure_mock(): 40cd. exists to make it easier to do
configuration after the mock has been created.
-- Method: __dir__ ()
*note Mock: a4b. objects limit the results of ‘dir(some_mock)’
to useful results. For mocks with a 'spec' this includes all
the permitted attributes for the mock.
See *note FILTER_DIR: 40d4. for what this filtering does, and
how to switch it off.
-- Method: _get_child_mock (**kw)
Create the child mocks for attributes and return value. By
default child mocks will be the same type as the parent.
Subclasses of Mock may want to override this to customize the
way child mocks are made.
For non-callable mocks the callable variant will be used
(rather than any custom subclass).
-- Attribute: called
A boolean representing whether or not the mock object has been
called:
>>> mock = Mock(return_value=None)
>>> mock.called
False
>>> mock()
>>> mock.called
True
-- Attribute: call_count
An integer telling you how many times the mock object has been
called:
>>> mock = Mock(return_value=None)
>>> mock.call_count
0
>>> mock()
>>> mock()
>>> mock.call_count
2
-- Attribute: return_value
Set this to configure the value returned by calling the mock:
>>> mock = Mock()
>>> mock.return_value = 'fish'
>>> mock()
'fish'
The default return value is a mock object and you can
configure it in the normal way:
>>> mock = Mock()
>>> mock.return_value.attribute = sentinel.Attribute
>>> mock.return_value()
>>> mock.return_value.assert_called_with()
*note return_value: 40cc. can also be set in the constructor:
>>> mock = Mock(return_value=3)
>>> mock.return_value
3
>>> mock()
3
-- Attribute: side_effect
This can either be a function to be called when the mock is
called, an iterable or an exception (class or instance) to be
raised.
If you pass in a function it will be called with same
arguments as the mock and unless the function returns the
*note DEFAULT: 40cb. singleton the call to the mock will then
return whatever the function returns. If the function returns
*note DEFAULT: 40cb. then the mock will return its normal
value (from the *note return_value: 40cc.).
If you pass in an iterable, it is used to retrieve an iterator
which must yield a value on every call. This value can either
be an exception instance to be raised, or a value to be
returned from the call to the mock (*note DEFAULT: 40cb.
handling is identical to the function case).
An example of a mock that raises an exception (to test
exception handling of an API):
>>> mock = Mock()
>>> mock.side_effect = Exception('Boom!')
>>> mock()
Traceback (most recent call last):
...
Exception: Boom!
Using *note side_effect: 40c4. to return a sequence of values:
>>> mock = Mock()
>>> mock.side_effect = [3, 2, 1]
>>> mock(), mock(), mock()
(3, 2, 1)
Using a callable:
>>> mock = Mock(return_value=3)
>>> def side_effect(*args, **kwargs):
... return DEFAULT
...
>>> mock.side_effect = side_effect
>>> mock()
3
*note side_effect: 40c4. can be set in the constructor.
Here’s an example that adds one to the value the mock is
called with and returns it:
>>> side_effect = lambda value: value + 1
>>> mock = Mock(side_effect=side_effect)
>>> mock(3)
4
>>> mock(-8)
-7
Setting *note side_effect: 40c4. to ‘None’ clears it:
>>> m = Mock(side_effect=KeyError, return_value=3)
>>> m()
Traceback (most recent call last):
...
KeyError
>>> m.side_effect = None
>>> m()
3
-- Attribute: call_args
This is either ‘None’ (if the mock hasn’t been called), or the
arguments that the mock was last called with. This will be in
the form of a tuple: the first member, which can also be
accessed through the ‘args’ property, is any positional
arguments the mock was called with (or an empty tuple) and the
second member, which can also be accessed through the ‘kwargs’
property, is any keyword arguments (or an empty dictionary).
>>> mock = Mock(return_value=None)
>>> print(mock.call_args)
None
>>> mock()
>>> mock.call_args
call()
>>> mock.call_args == ()
True
>>> mock(3, 4)
>>> mock.call_args
call(3, 4)
>>> mock.call_args == ((3, 4),)
True
>>> mock.call_args.args
(3, 4)
>>> mock.call_args.kwargs
{}
>>> mock(3, 4, 5, key='fish', next='w00t!')
>>> mock.call_args
call(3, 4, 5, key='fish', next='w00t!')
>>> mock.call_args.args
(3, 4, 5)
>>> mock.call_args.kwargs
{'key': 'fish', 'next': 'w00t!'}
*note call_args: 197f, along with members of the lists *note
call_args_list: 40d8, *note method_calls: 40d2. and *note
mock_calls: 40d0. are *note call: 19cd. objects. These are
tuples, so they can be unpacked to get at the individual
arguments and make more complex assertions. See *note calls
as tuples: 40d9.
Changed in version 3.8: Added ‘args’ and ‘kwargs’ properties.
-- Attribute: call_args_list
This is a list of all the calls made to the mock object in
sequence (so the length of the list is the number of times it
has been called). Before any calls have been made it is an
empty list. The *note call: 19cd. object can be used for
conveniently constructing lists of calls to compare with *note
call_args_list: 40d8.
>>> mock = Mock(return_value=None)
>>> mock()
>>> mock(3, 4)
>>> mock(key='fish', next='w00t!')
>>> mock.call_args_list
[call(), call(3, 4), call(key='fish', next='w00t!')]
>>> expected = [(), ((3, 4),), ({'key': 'fish', 'next': 'w00t!'},)]
>>> mock.call_args_list == expected
True
Members of *note call_args_list: 40d8. are *note call: 19cd.
objects. These can be unpacked as tuples to get at the
individual arguments. See *note calls as tuples: 40d9.
-- Attribute: method_calls
As well as tracking calls to themselves, mocks also track
calls to methods and attributes, and 'their' methods and
attributes:
>>> mock = Mock()
>>> mock.method()
>>> mock.property.method.attribute()
>>> mock.method_calls
[call.method(), call.property.method.attribute()]
Members of *note method_calls: 40d2. are *note call: 19cd.
objects. These can be unpacked as tuples to get at the
individual arguments. See *note calls as tuples: 40d9.
-- Attribute: mock_calls
*note mock_calls: 40d0. records 'all' calls to the mock
object, its methods, magic methods 'and' return value mocks.
>>> mock = MagicMock()
>>> result = mock(1, 2, 3)
>>> mock.first(a=3)
>>> mock.second()
>>> int(mock)
1
>>> result(1)
>>> expected = [call(1, 2, 3), call.first(a=3), call.second(),
... call.__int__(), call()(1)]
>>> mock.mock_calls == expected
True
Members of *note mock_calls: 40d0. are *note call: 19cd.
objects. These can be unpacked as tuples to get at the
individual arguments. See *note calls as tuples: 40d9.
Note: The way *note mock_calls: 40d0. are recorded means
that where nested calls are made, the parameters of
ancestor calls are not recorded and so will always
compare equal:
>>> mock = MagicMock()
>>> mock.top(a=3).bottom()
>>> mock.mock_calls
[call.top(a=3), call.top().bottom()]
>>> mock.mock_calls[-1] == call.top(a=-1).bottom()
True
-- Attribute: __class__
Normally the ‘__class__’ attribute of an object will return
its type. For a mock object with a ‘spec’, ‘__class__’
returns the spec class instead. This allows mock objects to
pass *note isinstance(): 43d. tests for the object they are
replacing / masquerading as:
>>> mock = Mock(spec=3)
>>> isinstance(mock, int)
True
‘__class__’ is assignable to, this allows a mock to pass an
*note isinstance(): 43d. check without forcing you to use a
spec:
>>> mock = Mock()
>>> mock.__class__ = dict
>>> isinstance(mock, dict)
True
-- Class: unittest.mock.NonCallableMock (spec=None, wraps=None,
name=None, spec_set=None, **kwargs)
A non-callable version of *note Mock: a4b. The constructor
parameters have the same meaning of *note Mock: a4b, with the
exception of 'return_value' and 'side_effect' which have no meaning
on a non-callable mock.
Mock objects that use a class or an instance as a ‘spec’ or ‘spec_set’
are able to pass *note isinstance(): 43d. tests:
>>> mock = Mock(spec=SomeClass)
>>> isinstance(mock, SomeClass)
True
>>> mock = Mock(spec_set=SomeClass())
>>> isinstance(mock, SomeClass)
True
The *note Mock: a4b. classes have support for mocking magic methods.
See *note magic methods: 40c6. for the full details.
The mock classes and the *note patch(): e8b. decorators all take
arbitrary keyword arguments for configuration. For the *note patch():
e8b. decorators the keywords are passed to the constructor of the mock
being created. The keyword arguments are for configuring attributes of
the mock:
>>> m = MagicMock(attribute=3, other='fish')
>>> m.attribute
3
>>> m.other
'fish'
The return value and side effect of child mocks can be set in the same
way, using dotted notation. As you can’t use dotted names directly in a
call you have to create a dictionary and unpack it using ‘**’:
>>> attrs = {'method.return_value': 3, 'other.side_effect': KeyError}
>>> mock = Mock(some_attribute='eggs', **attrs)
>>> mock.some_attribute
'eggs'
>>> mock.method()
3
>>> mock.other()
Traceback (most recent call last):
...
KeyError
A callable mock which was created with a 'spec' (or a 'spec_set') will
introspect the specification object’s signature when matching calls to
the mock. Therefore, it can match the actual call’s arguments
regardless of whether they were passed positionally or by name:
>>> def f(a, b, c): pass
...
>>> mock = Mock(spec=f)
>>> mock(1, 2, c=3)
>>> mock.assert_called_with(1, 2, 3)
>>> mock.assert_called_with(a=1, b=2, c=3)
This applies to *note assert_called_with(): 1a20, *note
assert_called_once_with(): 40ce, *note assert_has_calls(): 16a8. and
*note assert_any_call(): 40cf. When *note Autospeccing: 40c7, it will
also apply to method calls on the mock object.
Changed in version 3.4: Added signature introspection on specced and
autospecced mock objects.
-- Class: unittest.mock.PropertyMock (*args, **kwargs)
A mock intended to be used as a *note property: 194, or other *note
descriptor: 1572, on a class. *note PropertyMock: 40db. provides
*note __get__(): 14b2. and *note __set__(): 1f6a. methods so you
can specify a return value when it is fetched.
Fetching a *note PropertyMock: 40db. instance from an object calls
the mock, with no args. Setting it calls the mock with the value
being set.
>>> class Foo:
... @property
... def foo(self):
... return 'something'
... @foo.setter
... def foo(self, value):
... pass
...
>>> with patch('__main__.Foo.foo', new_callable=PropertyMock) as mock_foo:
... mock_foo.return_value = 'mockity-mock'
... this_foo = Foo()
... print(this_foo.foo)
... this_foo.foo = 6
...
mockity-mock
>>> mock_foo.mock_calls
[call(), call(6)]
Because of the way mock attributes are stored you can’t directly attach
a *note PropertyMock: 40db. to a mock object. Instead you can attach it
to the mock type object:
>>> m = MagicMock()
>>> p = PropertyMock(return_value=3)
>>> type(m).foo = p
>>> m.foo
3
>>> p.assert_called_once_with()
Caution: If an *note AttributeError: 348. is raised by *note
PropertyMock: 40db, it will be interpreted as a missing descriptor
and *note __getattr__(): 4cf. will be called on the parent mock:
>>> m = MagicMock()
>>> no_attribute = PropertyMock(side_effect=AttributeError)
>>> type(m).my_property = no_attribute
>>> m.my_property
See *note __getattr__(): 4cf. for details.
-- Class: unittest.mock.AsyncMock (spec=None, side_effect=None,
return_value=DEFAULT, wraps=None, name=None, spec_set=None,
unsafe=False, **kwargs)
An asynchronous version of *note MagicMock: e8a. The *note
AsyncMock: a4a. object will behave so the object is recognized as
an async function, and the result of a call is an awaitable.
>>> mock = AsyncMock()
>>> asyncio.iscoroutinefunction(mock)
True
>>> inspect.isawaitable(mock())
True
The result of ‘mock()’ is an async function which will have the
outcome of ‘side_effect’ or ‘return_value’ after it has been
awaited:
- if ‘side_effect’ is a function, the async function will return
the result of that function,
- if ‘side_effect’ is an exception, the async function will
raise the exception,
- if ‘side_effect’ is an iterable, the async function will
return the next value of the iterable, however, if the
sequence of result is exhausted, ‘StopAsyncIteration’ is
raised immediately,
- if ‘side_effect’ is not defined, the async function will
return the value defined by ‘return_value’, hence, by default,
the async function returns a new *note AsyncMock: a4a. object.
Setting the 'spec' of a *note Mock: a4b. or *note MagicMock: e8a.
to an async function will result in a coroutine object being
returned after calling.
>>> async def async_func(): pass
...
>>> mock = MagicMock(async_func)
>>> mock
>>> mock()
Setting the 'spec' of a *note Mock: a4b, *note MagicMock: e8a, or
*note AsyncMock: a4a. to a class with asynchronous and synchronous
functions will automatically detect the synchronous functions and
set them as *note MagicMock: e8a. (if the parent mock is *note
AsyncMock: a4a. or *note MagicMock: e8a.) or *note Mock: a4b. (if
the parent mock is *note Mock: a4b.). All asynchronous functions
will be *note AsyncMock: a4a.
>>> class ExampleClass:
... def sync_foo():
... pass
... async def async_foo():
... pass
...
>>> a_mock = AsyncMock(ExampleClass)
>>> a_mock.sync_foo
>>> a_mock.async_foo
>>> mock = Mock(ExampleClass)
>>> mock.sync_foo
>>> mock.async_foo
Added in version 3.8.
-- Method: assert_awaited ()
Assert that the mock was awaited at least once. Note that
this is separate from the object having been called, the
‘await’ keyword must be used:
>>> mock = AsyncMock()
>>> async def main(coroutine_mock):
... await coroutine_mock
...
>>> coroutine_mock = mock()
>>> mock.called
True
>>> mock.assert_awaited()
Traceback (most recent call last):
...
AssertionError: Expected mock to have been awaited.
>>> asyncio.run(main(coroutine_mock))
>>> mock.assert_awaited()
-- Method: assert_awaited_once ()
Assert that the mock was awaited exactly once.
>>> mock = AsyncMock()
>>> async def main():
... await mock()
...
>>> asyncio.run(main())
>>> mock.assert_awaited_once()
>>> asyncio.run(main())
>>> mock.assert_awaited_once()
Traceback (most recent call last):
...
AssertionError: Expected mock to have been awaited once. Awaited 2 times.
-- Method: assert_awaited_with (*args, **kwargs)
Assert that the last await was with the specified arguments.
>>> mock = AsyncMock()
>>> async def main(*args, **kwargs):
... await mock(*args, **kwargs)
...
>>> asyncio.run(main('foo', bar='bar'))
>>> mock.assert_awaited_with('foo', bar='bar')
>>> mock.assert_awaited_with('other')
Traceback (most recent call last):
...
AssertionError: expected await not found.
Expected: mock('other')
Actual: mock('foo', bar='bar')
-- Method: assert_awaited_once_with (*args, **kwargs)
Assert that the mock was awaited exactly once and with the
specified arguments.
>>> mock = AsyncMock()
>>> async def main(*args, **kwargs):
... await mock(*args, **kwargs)
...
>>> asyncio.run(main('foo', bar='bar'))
>>> mock.assert_awaited_once_with('foo', bar='bar')
>>> asyncio.run(main('foo', bar='bar'))
>>> mock.assert_awaited_once_with('foo', bar='bar')
Traceback (most recent call last):
...
AssertionError: Expected mock to have been awaited once. Awaited 2 times.
-- Method: assert_any_await (*args, **kwargs)
Assert the mock has ever been awaited with the specified
arguments.
>>> mock = AsyncMock()
>>> async def main(*args, **kwargs):
... await mock(*args, **kwargs)
...
>>> asyncio.run(main('foo', bar='bar'))
>>> asyncio.run(main('hello'))
>>> mock.assert_any_await('foo', bar='bar')
>>> mock.assert_any_await('other')
Traceback (most recent call last):
...
AssertionError: mock('other') await not found
-- Method: assert_has_awaits (calls, any_order=False)
Assert the mock has been awaited with the specified calls.
The *note await_args_list: 197e. list is checked for the
awaits.
If 'any_order' is false then the awaits must be sequential.
There can be extra calls before or after the specified awaits.
If 'any_order' is true then the awaits can be in any order,
but they must all appear in *note await_args_list: 197e.
>>> mock = AsyncMock()
>>> async def main(*args, **kwargs):
... await mock(*args, **kwargs)
...
>>> calls = [call("foo"), call("bar")]
>>> mock.assert_has_awaits(calls)
Traceback (most recent call last):
...
AssertionError: Awaits not found.
Expected: [call('foo'), call('bar')]
Actual: []
>>> asyncio.run(main('foo'))
>>> asyncio.run(main('bar'))
>>> mock.assert_has_awaits(calls)
-- Method: assert_not_awaited ()
Assert that the mock was never awaited.
>>> mock = AsyncMock()
>>> mock.assert_not_awaited()
-- Method: reset_mock (*args, **kwargs)
See *note Mock.reset_mock(): cfa. Also sets *note
await_count: 40e4. to 0, *note await_args: 40e5. to None, and
clears the *note await_args_list: 197e.
-- Attribute: await_count
An integer keeping track of how many times the mock object has
been awaited.
>>> mock = AsyncMock()
>>> async def main():
... await mock()
...
>>> asyncio.run(main())
>>> mock.await_count
1
>>> asyncio.run(main())
>>> mock.await_count
2
-- Attribute: await_args
This is either ‘None’ (if the mock hasn’t been awaited), or
the arguments that the mock was last awaited with. Functions
the same as *note Mock.call_args: 197f.
>>> mock = AsyncMock()
>>> async def main(*args):
... await mock(*args)
...
>>> mock.await_args
>>> asyncio.run(main('foo'))
>>> mock.await_args
call('foo')
>>> asyncio.run(main('bar'))
>>> mock.await_args
call('bar')
-- Attribute: await_args_list
This is a list of all the awaits made to the mock object in
sequence (so the length of the list is the number of times it
has been awaited). Before any awaits have been made it is an
empty list.
>>> mock = AsyncMock()
>>> async def main(*args):
... await mock(*args)
...
>>> mock.await_args_list
[]
>>> asyncio.run(main('foo'))
>>> mock.await_args_list
[call('foo')]
>>> asyncio.run(main('bar'))
>>> mock.await_args_list
[call('foo'), call('bar')]
-- Class: unittest.mock.ThreadingMock (spec=None, side_effect=None,
return_value=DEFAULT, wraps=None, name=None, spec_set=None,
unsafe=False, *, timeout=UNSET, **kwargs)
A version of *note MagicMock: e8a. for multithreading tests. The
*note ThreadingMock: 40e6. object provides extra methods to wait
for a call to be invoked, rather than assert on it immediately.
The default timeout is specified by the ‘timeout’ argument, or if
unset by the *note ThreadingMock.DEFAULT_TIMEOUT: 40e7. attribute,
which defaults to blocking (‘None’).
You can configure the global default timeout by setting *note
ThreadingMock.DEFAULT_TIMEOUT: 40e7.
-- Method: wait_until_called (*, timeout=UNSET)
Waits until the mock is called.
If a timeout was passed at the creation of the mock or if a
timeout argument is passed to this function, the function
raises an *note AssertionError: 6a5. if the call is not
performed in time.
>>> mock = ThreadingMock()
>>> thread = threading.Thread(target=mock)
>>> thread.start()
>>> mock.wait_until_called(timeout=1)
>>> thread.join()
-- Method: wait_until_any_call_with (*args, **kwargs)
Waits until the mock is called with the specified arguments.
If a timeout was passed at the creation of the mock the
function raises an *note AssertionError: 6a5. if the call is
not performed in time.
>>> mock = ThreadingMock()
>>> thread = threading.Thread(target=mock, args=("arg1", "arg2",), kwargs={"arg": "thing"})
>>> thread.start()
>>> mock.wait_until_any_call_with("arg1", "arg2", arg="thing")
>>> thread.join()
-- Attribute: DEFAULT_TIMEOUT
Global default timeout in seconds to create instances of *note
ThreadingMock: 40e6.
Added in version 3.13.
* Menu:
* Calling::
* Deleting Attributes::
* Mock names and the name attribute::
* Attaching Mocks as Attributes::
---------- Footnotes ----------
(1) The only exceptions are magic methods and attributes (those that
have leading and trailing double underscores). Mock doesn’t create
these but instead raises an *note AttributeError: 348. This is because
the interpreter will often implicitly request these methods, and gets
'very' confused to get a new Mock object when it expects a magic method.
If you need magic method support see *note magic methods: 40c6.
File: python.info, Node: Calling, Next: Deleting Attributes, Up: The Mock Class
5.27.6.3 Calling
................
Mock objects are callable. The call will return the value set as the
*note return_value: 40cc. attribute. The default return value is a new
Mock object; it is created the first time the return value is accessed
(either explicitly or by calling the Mock) - but it is stored and the
same one returned each time.
Calls made to the object will be recorded in the attributes like *note
call_args: 197f. and *note call_args_list: 40d8.
If *note side_effect: 40c4. is set then it will be called after the call
has been recorded, so if ‘side_effect’ raises an exception the call is
still recorded.
The simplest way to make a mock raise an exception when called is to
make *note side_effect: 40c4. an exception class or instance:
>>> m = MagicMock(side_effect=IndexError)
>>> m(1, 2, 3)
Traceback (most recent call last):
...
IndexError
>>> m.mock_calls
[call(1, 2, 3)]
>>> m.side_effect = KeyError('Bang!')
>>> m('two', 'three', 'four')
Traceback (most recent call last):
...
KeyError: 'Bang!'
>>> m.mock_calls
[call(1, 2, 3), call('two', 'three', 'four')]
If *note side_effect: 40c4. is a function then whatever that function
returns is what calls to the mock return. The ‘side_effect’ function is
called with the same arguments as the mock. This allows you to vary the
return value of the call dynamically, based on the input:
>>> def side_effect(value):
... return value + 1
...
>>> m = MagicMock(side_effect=side_effect)
>>> m(1)
2
>>> m(2)
3
>>> m.mock_calls
[call(1), call(2)]
If you want the mock to still return the default return value (a new
mock), or any set return value, then there are two ways of doing this.
Either return *note return_value: 40cc. from inside *note side_effect:
40c4, or return *note DEFAULT: 40cb.:
>>> m = MagicMock()
>>> def side_effect(*args, **kwargs):
... return m.return_value
...
>>> m.side_effect = side_effect
>>> m.return_value = 3
>>> m()
3
>>> def side_effect(*args, **kwargs):
... return DEFAULT
...
>>> m.side_effect = side_effect
>>> m()
3
To remove a *note side_effect: 40c4, and return to the default
behaviour, set the ‘side_effect’ to ‘None’:
>>> m = MagicMock(return_value=6)
>>> def side_effect(*args, **kwargs):
... return 3
...
>>> m.side_effect = side_effect
>>> m()
3
>>> m.side_effect = None
>>> m()
6
The *note side_effect: 40c4. can also be any iterable object. Repeated
calls to the mock will return values from the iterable (until the
iterable is exhausted and a *note StopIteration: bfa. is raised):
>>> m = MagicMock(side_effect=[1, 2, 3])
>>> m()
1
>>> m()
2
>>> m()
3
>>> m()
Traceback (most recent call last):
...
StopIteration
If any members of the iterable are exceptions they will be raised
instead of returned:
>>> iterable = (33, ValueError, 66)
>>> m = MagicMock(side_effect=iterable)
>>> m()
33
>>> m()
Traceback (most recent call last):
...
ValueError
>>> m()
66
File: python.info, Node: Deleting Attributes, Next: Mock names and the name attribute, Prev: Calling, Up: The Mock Class
5.27.6.4 Deleting Attributes
............................
Mock objects create attributes on demand. This allows them to pretend
to be objects of any type.
You may want a mock object to return ‘False’ to a *note hasattr(): 4ce.
call, or raise an *note AttributeError: 348. when an attribute is
fetched. You can do this by providing an object as a ‘spec’ for a mock,
but that isn’t always convenient.
You “block” attributes by deleting them. Once deleted, accessing an
attribute will raise an *note AttributeError: 348.
>>> mock = MagicMock()
>>> hasattr(mock, 'm')
True
>>> del mock.m
>>> hasattr(mock, 'm')
False
>>> del mock.f
>>> mock.f
Traceback (most recent call last):
...
AttributeError: f
File: python.info, Node: Mock names and the name attribute, Next: Attaching Mocks as Attributes, Prev: Deleting Attributes, Up: The Mock Class
5.27.6.5 Mock names and the name attribute
..........................................
Since “name” is an argument to the *note Mock: a4b. constructor, if you
want your mock object to have a “name” attribute you can’t just pass it
in at creation time. There are two alternatives. One option is to use
*note configure_mock(): 40cd.:
>>> mock = MagicMock()
>>> mock.configure_mock(name='my_name')
>>> mock.name
'my_name'
A simpler option is to simply set the “name” attribute after mock
creation:
>>> mock = MagicMock()
>>> mock.name = "foo"
File: python.info, Node: Attaching Mocks as Attributes, Prev: Mock names and the name attribute, Up: The Mock Class
5.27.6.6 Attaching Mocks as Attributes
......................................
When you attach a mock as an attribute of another mock (or as the return
value) it becomes a “child” of that mock. Calls to the child are
recorded in the *note method_calls: 40d2. and *note mock_calls: 40d0.
attributes of the parent. This is useful for configuring child mocks
and then attaching them to the parent, or for attaching mocks to a
parent that records all calls to the children and allows you to make
assertions about the order of calls between mocks:
>>> parent = MagicMock()
>>> child1 = MagicMock(return_value=None)
>>> child2 = MagicMock(return_value=None)
>>> parent.child1 = child1
>>> parent.child2 = child2
>>> child1(1)
>>> child2(2)
>>> parent.mock_calls
[call.child1(1), call.child2(2)]
The exception to this is if the mock has a name. This allows you to
prevent the “parenting” if for some reason you don’t want it to happen.
>>> mock = MagicMock()
>>> not_a_child = MagicMock(name='not-a-child')
>>> mock.attribute = not_a_child
>>> mock.attribute()
>>> mock.mock_calls
[]
Mocks created for you by *note patch(): e8b. are automatically given
names. To attach mocks that have names to a parent you use the *note
attach_mock(): 160f. method:
>>> thing1 = object()
>>> thing2 = object()
>>> parent = MagicMock()
>>> with patch('__main__.thing1', return_value=None) as child1:
... with patch('__main__.thing2', return_value=None) as child2:
... parent.attach_mock(child1, 'child1')
... parent.attach_mock(child2, 'child2')
... child1('one')
... child2('two')
...
>>> parent.mock_calls
[call.child1('one'), call.child2('two')]
File: python.info, Node: The patchers, Next: MagicMock and magic method support, Prev: The Mock Class, Up: unittest mock — mock object library
5.27.6.7 The patchers
.....................
The patch decorators are used for patching objects only within the scope
of the function they decorate. They automatically handle the unpatching
for you, even if exceptions are raised. All of these functions can also
be used in with statements or as class decorators.
* Menu:
* patch::
* patch.object: patch object.
* patch.dict: patch dict.
* patch.multiple: patch multiple.
* patch methods; start and stop: patch methods start and stop.
* patch builtins::
* TEST_PREFIX::
* Nesting Patch Decorators::
* Where to patch::
* Patching Descriptors and Proxy Objects::
File: python.info, Node: patch, Next: patch object, Up: The patchers
5.27.6.8 patch
..............
Note: The key is to do the patching in the right namespace. See
the section *note where to patch: 40f1.
-- Function: unittest.mock.patch (target, new=DEFAULT, spec=None,
create=False, spec_set=None, autospec=None, new_callable=None,
**kwargs)
*note patch(): e8b. acts as a function decorator, class decorator
or a context manager. Inside the body of the function or with
statement, the 'target' is patched with a 'new' object. When the
function/with statement exits the patch is undone.
If 'new' is omitted, then the target is replaced with an *note
AsyncMock: a4a. if the patched object is an async function or a
*note MagicMock: e8a. otherwise. If *note patch(): e8b. is used as
a decorator and 'new' is omitted, the created mock is passed in as
an extra argument to the decorated function. If *note patch():
e8b. is used as a context manager the created mock is returned by
the context manager.
'target' should be a string in the form
‘'package.module.ClassName'’. The 'target' is imported and the
specified object replaced with the 'new' object, so the 'target'
must be importable from the environment you are calling *note
patch(): e8b. from. The target is imported when the decorated
function is executed, not at decoration time.
The 'spec' and 'spec_set' keyword arguments are passed to the *note
MagicMock: e8a. if patch is creating one for you.
In addition you can pass ‘spec=True’ or ‘spec_set=True’, which
causes patch to pass in the object being mocked as the
spec/spec_set object.
'new_callable' allows you to specify a different class, or callable
object, that will be called to create the 'new' object. By default
*note AsyncMock: a4a. is used for async functions and *note
MagicMock: e8a. for the rest.
A more powerful form of 'spec' is 'autospec'. If you set
‘autospec=True’ then the mock will be created with a spec from the
object being replaced. All attributes of the mock will also have
the spec of the corresponding attribute of the object being
replaced. Methods and functions being mocked will have their
arguments checked and will raise a *note TypeError: 534. if they
are called with the wrong signature. For mocks replacing a class,
their return value (the ‘instance’) will have the same spec as the
class. See the *note create_autospec(): 1608. function and *note
Autospeccing: 40c7.
Instead of ‘autospec=True’ you can pass ‘autospec=some_object’ to
use an arbitrary object as the spec instead of the one being
replaced.
By default *note patch(): e8b. will fail to replace attributes that
don’t exist. If you pass in ‘create=True’, and the attribute
doesn’t exist, patch will create the attribute for you when the
patched function is called, and delete it again after the patched
function has exited. This is useful for writing tests against
attributes that your production code creates at runtime. It is off
by default because it can be dangerous. With it switched on you
can write passing tests against APIs that don’t actually exist!
Note:
Changed in version 3.5: If you are patching builtins in a
module then you don’t need to pass ‘create=True’, it will be
added by default.
Patch can be used as a *note TestCase: 449. class decorator. It
works by decorating each test method in the class. This reduces
the boilerplate code when your test methods share a common
patchings set. *note patch(): e8b. finds tests by looking for
method names that start with ‘patch.TEST_PREFIX’. By default this
is ‘'test'’, which matches the way *note unittest: 106. finds
tests. You can specify an alternative prefix by setting
‘patch.TEST_PREFIX’.
Patch can be used as a context manager, with the with statement.
Here the patching applies to the indented block after the with
statement. If you use “as” then the patched object will be bound
to the name after the “as”; very useful if *note patch(): e8b. is
creating a mock object for you.
*note patch(): e8b. takes arbitrary keyword arguments. These will
be passed to *note AsyncMock: a4a. if the patched object is
asynchronous, to *note MagicMock: e8a. otherwise or to
'new_callable' if specified.
‘patch.dict(...)’, ‘patch.multiple(...)’ and ‘patch.object(...)’
are available for alternate use-cases.
*note patch(): e8b. as function decorator, creating the mock for you and
passing it into the decorated function:
>>> @patch('__main__.SomeClass')
... def function(normal_argument, mock_class):
... print(mock_class is SomeClass)
...
>>> function(None)
True
Patching a class replaces the class with a *note MagicMock: e8a.
'instance'. If the class is instantiated in the code under test then it
will be the *note return_value: 40cc. of the mock that will be used.
If the class is instantiated multiple times you could use *note
side_effect: 40c4. to return a new mock each time. Alternatively you
can set the 'return_value' to be anything you want.
To configure return values on methods of 'instances' on the patched
class you must do this on the *note return_value: 40cc. For example:
>>> class Class:
... def method(self):
... pass
...
>>> with patch('__main__.Class') as MockClass:
... instance = MockClass.return_value
... instance.method.return_value = 'foo'
... assert Class() is instance
... assert Class().method() == 'foo'
...
If you use 'spec' or 'spec_set' and *note patch(): e8b. is replacing a
'class', then the return value of the created mock will have the same
spec.
>>> Original = Class
>>> patcher = patch('__main__.Class', spec=True)
>>> MockClass = patcher.start()
>>> instance = MockClass()
>>> assert isinstance(instance, Original)
>>> patcher.stop()
The 'new_callable' argument is useful where you want to use an
alternative class to the default *note MagicMock: e8a. for the created
mock. For example, if you wanted a *note NonCallableMock: 40c9. to be
used:
>>> thing = object()
>>> with patch('__main__.thing', new_callable=NonCallableMock) as mock_thing:
... assert thing is mock_thing
... thing()
...
Traceback (most recent call last):
...
TypeError: 'NonCallableMock' object is not callable
Another use case might be to replace an object with an *note
io.StringIO: f22. instance:
>>> from io import StringIO
>>> def foo():
... print('Something')
...
>>> @patch('sys.stdout', new_callable=StringIO)
... def test(mock_stdout):
... foo()
... assert mock_stdout.getvalue() == 'Something\n'
...
>>> test()
When *note patch(): e8b. is creating a mock for you, it is common that
the first thing you need to do is to configure the mock. Some of that
configuration can be done in the call to patch. Any arbitrary keywords
you pass into the call will be used to set attributes on the created
mock:
>>> patcher = patch('__main__.thing', first='one', second='two')
>>> mock_thing = patcher.start()
>>> mock_thing.first
'one'
>>> mock_thing.second
'two'
As well as attributes on the created mock attributes, like the *note
return_value: 40cc. and *note side_effect: 40c4, of child mocks can also
be configured. These aren’t syntactically valid to pass in directly as
keyword arguments, but a dictionary with these as keys can still be
expanded into a *note patch(): e8b. call using ‘**’:
>>> config = {'method.return_value': 3, 'other.side_effect': KeyError}
>>> patcher = patch('__main__.thing', **config)
>>> mock_thing = patcher.start()
>>> mock_thing.method()
3
>>> mock_thing.other()
Traceback (most recent call last):
...
KeyError
By default, attempting to patch a function in a module (or a method or
an attribute in a class) that does not exist will fail with *note
AttributeError: 348.:
>>> @patch('sys.non_existing_attribute', 42)
... def test():
... assert sys.non_existing_attribute == 42
...
>>> test()
Traceback (most recent call last):
...
AttributeError: does not have the attribute 'non_existing_attribute'
but adding ‘create=True’ in the call to *note patch(): e8b. will make
the previous example work as expected:
>>> @patch('sys.non_existing_attribute', 42, create=True)
... def test(mock_stdout):
... assert sys.non_existing_attribute == 42
...
>>> test()
Changed in version 3.8: *note patch(): e8b. now returns an *note
AsyncMock: a4a. if the target is an async function.
File: python.info, Node: patch object, Next: patch dict, Prev: patch, Up: The patchers
5.27.6.9 patch.object
.....................
-- Function: patch.object (target, attribute, new=DEFAULT, spec=None,
create=False, spec_set=None, autospec=None, new_callable=None,
**kwargs)
patch the named member ('attribute') on an object ('target') with a
mock object.
*note patch.object(): 1598. can be used as a decorator, class
decorator or a context manager. Arguments 'new', 'spec', 'create',
'spec_set', 'autospec' and 'new_callable' have the same meaning as
for *note patch(): e8b. Like *note patch(): e8b, *note
patch.object(): 1598. takes arbitrary keyword arguments for
configuring the mock object it creates.
When used as a class decorator *note patch.object(): 1598. honours
‘patch.TEST_PREFIX’ for choosing which methods to wrap.
You can either call *note patch.object(): 1598. with three arguments or
two arguments. The three argument form takes the object to be patched,
the attribute name and the object to replace the attribute with.
When calling with the two argument form you omit the replacement object,
and a mock is created for you and passed in as an extra argument to the
decorated function:
>>> @patch.object(SomeClass, 'class_method')
... def test(mock_method):
... SomeClass.class_method(3)
... mock_method.assert_called_with(3)
...
>>> test()
'spec', 'create' and the other arguments to *note patch.object(): 1598.
have the same meaning as they do for *note patch(): e8b.
File: python.info, Node: patch dict, Next: patch multiple, Prev: patch object, Up: The patchers
5.27.6.10 patch.dict
....................
-- Function: patch.dict (in_dict, values=(), clear=False, **kwargs)
Patch a dictionary, or dictionary like object, and restore the
dictionary to its original state after the test.
'in_dict' can be a dictionary or a mapping like container. If it
is a mapping then it must at least support getting, setting and
deleting items plus iterating over keys.
'in_dict' can also be a string specifying the name of the
dictionary, which will then be fetched by importing it.
'values' can be a dictionary of values to set in the dictionary.
'values' can also be an iterable of ‘(key, value)’ pairs.
If 'clear' is true then the dictionary will be cleared before the
new values are set.
*note patch.dict(): 19f2. can also be called with arbitrary keyword
arguments to set values in the dictionary.
Changed in version 3.8: *note patch.dict(): 19f2. now returns the
patched dictionary when used as a context manager.
*note patch.dict(): 19f2. can be used as a context manager, decorator or
class decorator:
>>> foo = {}
>>> @patch.dict(foo, {'newkey': 'newvalue'})
... def test():
... assert foo == {'newkey': 'newvalue'}
...
>>> test()
>>> assert foo == {}
When used as a class decorator *note patch.dict(): 19f2. honours
‘patch.TEST_PREFIX’ (default to ‘'test'’) for choosing which methods to
wrap:
>>> import os
>>> import unittest
>>> from unittest.mock import patch
>>> @patch.dict('os.environ', {'newkey': 'newvalue'})
... class TestSample(unittest.TestCase):
... def test_sample(self):
... self.assertEqual(os.environ['newkey'], 'newvalue')
If you want to use a different prefix for your test, you can inform the
patchers of the different prefix by setting ‘patch.TEST_PREFIX’. For
more details about how to change the value of see *note TEST_PREFIX:
40f4.
*note patch.dict(): 19f2. can be used to add members to a dictionary, or
simply let a test change a dictionary, and ensure the dictionary is
restored when the test ends.
>>> foo = {}
>>> with patch.dict(foo, {'newkey': 'newvalue'}) as patched_foo:
... assert foo == {'newkey': 'newvalue'}
... assert patched_foo == {'newkey': 'newvalue'}
... # You can add, update or delete keys of foo (or patched_foo, it's the same dict)
... patched_foo['spam'] = 'eggs'
...
>>> assert foo == {}
>>> assert patched_foo == {}
>>> import os
>>> with patch.dict('os.environ', {'newkey': 'newvalue'}):
... print(os.environ['newkey'])
...
newvalue
>>> assert 'newkey' not in os.environ
Keywords can be used in the *note patch.dict(): 19f2. call to set values
in the dictionary:
>>> mymodule = MagicMock()
>>> mymodule.function.return_value = 'fish'
>>> with patch.dict('sys.modules', mymodule=mymodule):
... import mymodule
... mymodule.function('some', 'args')
...
'fish'
*note patch.dict(): 19f2. can be used with dictionary like objects that
aren’t actually dictionaries. At the very minimum they must support
item getting, setting, deleting and either iteration or membership test.
This corresponds to the magic methods *note __getitem__(): 285, *note
__setitem__(): 12be, *note __delitem__(): 12bf. and either *note
__iter__(): 1cc1. or *note __contains__(): 1f5e.
>>> class Container:
... def __init__(self):
... self.values = {}
... def __getitem__(self, name):
... return self.values[name]
... def __setitem__(self, name, value):
... self.values[name] = value
... def __delitem__(self, name):
... del self.values[name]
... def __iter__(self):
... return iter(self.values)
...
>>> thing = Container()
>>> thing['one'] = 1
>>> with patch.dict(thing, one=2, two=3):
... assert thing['one'] == 2
... assert thing['two'] == 3
...
>>> assert thing['one'] == 1
>>> assert list(thing) == ['one']
File: python.info, Node: patch multiple, Next: patch methods start and stop, Prev: patch dict, Up: The patchers
5.27.6.11 patch.multiple
........................
-- Function: patch.multiple (target, spec=None, create=False,
spec_set=None, autospec=None, new_callable=None, **kwargs)
Perform multiple patches in a single call. It takes the object to
be patched (either as an object or a string to fetch the object by
importing) and keyword arguments for the patches:
with patch.multiple(settings, FIRST_PATCH='one', SECOND_PATCH='two'):
...
Use *note DEFAULT: 40cb. as the value if you want *note
patch.multiple(): 40f6. to create mocks for you. In this case the
created mocks are passed into a decorated function by keyword, and
a dictionary is returned when *note patch.multiple(): 40f6. is used
as a context manager.
*note patch.multiple(): 40f6. can be used as a decorator, class
decorator or a context manager. The arguments 'spec', 'spec_set',
'create', 'autospec' and 'new_callable' have the same meaning as
for *note patch(): e8b. These arguments will be applied to 'all'
patches done by *note patch.multiple(): 40f6.
When used as a class decorator *note patch.multiple(): 40f6.
honours ‘patch.TEST_PREFIX’ for choosing which methods to wrap.
If you want *note patch.multiple(): 40f6. to create mocks for you, then
you can use *note DEFAULT: 40cb. as the value. If you use *note
patch.multiple(): 40f6. as a decorator then the created mocks are passed
into the decorated function by keyword.
>>> thing = object()
>>> other = object()
>>> @patch.multiple('__main__', thing=DEFAULT, other=DEFAULT)
... def test_function(thing, other):
... assert isinstance(thing, MagicMock)
... assert isinstance(other, MagicMock)
...
>>> test_function()
*note patch.multiple(): 40f6. can be nested with other ‘patch’
decorators, but put arguments passed by keyword 'after' any of the
standard arguments created by *note patch(): e8b.:
>>> @patch('sys.exit')
... @patch.multiple('__main__', thing=DEFAULT, other=DEFAULT)
... def test_function(mock_exit, other, thing):
... assert 'other' in repr(other)
... assert 'thing' in repr(thing)
... assert 'exit' in repr(mock_exit)
...
>>> test_function()
If *note patch.multiple(): 40f6. is used as a context manager, the value
returned by the context manager is a dictionary where created mocks are
keyed by name:
>>> with patch.multiple('__main__', thing=DEFAULT, other=DEFAULT) as values:
... assert 'other' in repr(values['other'])
... assert 'thing' in repr(values['thing'])
... assert values['thing'] is thing
... assert values['other'] is other
...
File: python.info, Node: patch methods start and stop, Next: patch builtins, Prev: patch multiple, Up: The patchers
5.27.6.12 patch methods: start and stop
.......................................
All the patchers have ‘start()’ and ‘stop()’ methods. These make it
simpler to do patching in ‘setUp’ methods or where you want to do
multiple patches without nesting decorators or with statements.
To use them call *note patch(): e8b, *note patch.object(): 1598. or
*note patch.dict(): 19f2. as normal and keep a reference to the returned
‘patcher’ object. You can then call ‘start()’ to put the patch in place
and ‘stop()’ to undo it.
If you are using *note patch(): e8b. to create a mock for you then it
will be returned by the call to ‘patcher.start’.
>>> patcher = patch('package.module.ClassName')
>>> from package import module
>>> original = module.ClassName
>>> new_mock = patcher.start()
>>> assert module.ClassName is not original
>>> assert module.ClassName is new_mock
>>> patcher.stop()
>>> assert module.ClassName is original
>>> assert module.ClassName is not new_mock
A typical use case for this might be for doing multiple patches in the
‘setUp’ method of a *note TestCase: 449.:
>>> class MyTest(unittest.TestCase):
... def setUp(self):
... self.patcher1 = patch('package.module.Class1')
... self.patcher2 = patch('package.module.Class2')
... self.MockClass1 = self.patcher1.start()
... self.MockClass2 = self.patcher2.start()
...
... def tearDown(self):
... self.patcher1.stop()
... self.patcher2.stop()
...
... def test_something(self):
... assert package.module.Class1 is self.MockClass1
... assert package.module.Class2 is self.MockClass2
...
>>> MyTest('test_something').run()
Caution: If you use this technique you must ensure that the
patching is “undone” by calling ‘stop’. This can be fiddlier than
you might think, because if an exception is raised in the ‘setUp’
then ‘tearDown’ is not called. *note
unittest.TestCase.addCleanup(): a81. makes this easier:
>>> class MyTest(unittest.TestCase):
... def setUp(self):
... patcher = patch('package.module.Class')
... self.MockClass = patcher.start()
... self.addCleanup(patcher.stop)
...
... def test_something(self):
... assert package.module.Class is self.MockClass
...
As an added bonus you no longer need to keep a reference to the
‘patcher’ object.
It is also possible to stop all patches which have been started by using
*note patch.stopall(): 40f9.
-- Function: patch.stopall ()
Stop all active patches. Only stops patches started with ‘start’.
File: python.info, Node: patch builtins, Next: TEST_PREFIX, Prev: patch methods start and stop, Up: The patchers
5.27.6.13 patch builtins
........................
You can patch any builtins within a module. The following example
patches builtin *note ord(): 1ef2.:
>>> @patch('__main__.ord')
... def test(mock_ord):
... mock_ord.return_value = 101
... print(ord('c'))
...
>>> test()
101
File: python.info, Node: TEST_PREFIX, Next: Nesting Patch Decorators, Prev: patch builtins, Up: The patchers
5.27.6.14 TEST_PREFIX
.....................
All of the patchers can be used as class decorators. When used in this
way they wrap every test method on the class. The patchers recognise
methods that start with ‘'test'’ as being test methods. This is the
same way that the *note unittest.TestLoader: 290. finds test methods by
default.
It is possible that you want to use a different prefix for your tests.
You can inform the patchers of the different prefix by setting
‘patch.TEST_PREFIX’:
>>> patch.TEST_PREFIX = 'foo'
>>> value = 3
>>>
>>> @patch('__main__.value', 'not three')
... class Thing:
... def foo_one(self):
... print(value)
... def foo_two(self):
... print(value)
...
>>>
>>> Thing().foo_one()
not three
>>> Thing().foo_two()
not three
>>> value
3
File: python.info, Node: Nesting Patch Decorators, Next: Where to patch, Prev: TEST_PREFIX, Up: The patchers
5.27.6.15 Nesting Patch Decorators
..................................
If you want to perform multiple patches then you can simply stack up the
decorators.
You can stack up multiple patch decorators using this pattern:
>>> @patch.object(SomeClass, 'class_method')
... @patch.object(SomeClass, 'static_method')
... def test(mock1, mock2):
... assert SomeClass.static_method is mock1
... assert SomeClass.class_method is mock2
... SomeClass.static_method('foo')
... SomeClass.class_method('bar')
... return mock1, mock2
...
>>> mock1, mock2 = test()
>>> mock1.assert_called_once_with('foo')
>>> mock2.assert_called_once_with('bar')
Note that the decorators are applied from the bottom upwards. This is
the standard way that Python applies decorators. The order of the
created mocks passed into your test function matches this order.
File: python.info, Node: Where to patch, Next: Patching Descriptors and Proxy Objects, Prev: Nesting Patch Decorators, Up: The patchers
5.27.6.16 Where to patch
........................
*note patch(): e8b. works by (temporarily) changing the object that a
'name' points to with another one. There can be many names pointing to
any individual object, so for patching to work you must ensure that you
patch the name used by the system under test.
The basic principle is that you patch where an object is 'looked up',
which is not necessarily the same place as where it is defined. A
couple of examples will help to clarify this.
Imagine we have a project that we want to test with the following
structure:
a.py
-> Defines SomeClass
b.py
-> from a import SomeClass
-> some_function instantiates SomeClass
Now we want to test ‘some_function’ but we want to mock out ‘SomeClass’
using *note patch(): e8b. The problem is that when we import module b,
which we will have to do when it imports ‘SomeClass’ from module a. If
we use *note patch(): e8b. to mock out ‘a.SomeClass’ then it will have
no effect on our test; module b already has a reference to the 'real'
‘SomeClass’ and it looks like our patching had no effect.
The key is to patch out ‘SomeClass’ where it is used (or where it is
looked up). In this case ‘some_function’ will actually look up
‘SomeClass’ in module b, where we have imported it. The patching should
look like:
@patch('b.SomeClass')
However, consider the alternative scenario where instead of ‘from a
import SomeClass’ module b does ‘import a’ and ‘some_function’ uses
‘a.SomeClass’. Both of these import forms are common. In this case the
class we want to patch is being looked up in the module and so we have
to patch ‘a.SomeClass’ instead:
@patch('a.SomeClass')
File: python.info, Node: Patching Descriptors and Proxy Objects, Prev: Where to patch, Up: The patchers
5.27.6.17 Patching Descriptors and Proxy Objects
................................................
Both *note patch: 40f0. and *note patch.object: 40f2. correctly patch
and restore descriptors: class methods, static methods and properties.
You should patch these on the 'class' rather than an instance. They
also work with 'some' objects that proxy attribute access, like the
django settings object(1).
---------- Footnotes ----------
(1)
https://web.archive.org/web/20200603181648/http://www.voidspace.org.uk/python/weblog/arch_d7_2010_12_04.shtml#e1198
File: python.info, Node: MagicMock and magic method support, Next: Helpers, Prev: The patchers, Up: unittest mock — mock object library
5.27.6.18 MagicMock and magic method support
............................................
* Menu:
* Mocking Magic Methods::
* Magic Mock::
File: python.info, Node: Mocking Magic Methods, Next: Magic Mock, Up: MagicMock and magic method support
5.27.6.19 Mocking Magic Methods
...............................
*note Mock: a4b. supports mocking the Python protocol methods, also
known as *note “magic methods”: 4101. This allows mock objects to
replace containers or other objects that implement Python protocols.
Because magic methods are looked up differently from normal methods (1),
this support has been specially implemented. This means that only
specific magic methods are supported. The supported list includes
'almost' all of them. If there are any missing that you need please let
us know.
You mock magic methods by setting the method you are interested in to a
function or a mock instance. If you are using a function then it 'must'
take ‘self’ as the first argument (2).
>>> def __str__(self):
... return 'fooble'
...
>>> mock = Mock()
>>> mock.__str__ = __str__
>>> str(mock)
'fooble'
>>> mock = Mock()
>>> mock.__str__ = Mock()
>>> mock.__str__.return_value = 'fooble'
>>> str(mock)
'fooble'
>>> mock = Mock()
>>> mock.__iter__ = Mock(return_value=iter([]))
>>> list(mock)
[]
One use case for this is for mocking objects used as context managers in
a *note with: 5ce. statement:
>>> mock = Mock()
>>> mock.__enter__ = Mock(return_value='foo')
>>> mock.__exit__ = Mock(return_value=False)
>>> with mock as m:
... assert m == 'foo'
...
>>> mock.__enter__.assert_called_with()
>>> mock.__exit__.assert_called_with(None, None, None)
Calls to magic methods do not appear in *note method_calls: 40d2, but
they are recorded in *note mock_calls: 40d0.
Note: If you use the 'spec' keyword argument to create a mock then
attempting to set a magic method that isn’t in the spec will raise
an *note AttributeError: 348.
The full list of supported magic methods is:
* ‘__hash__’, ‘__sizeof__’, ‘__repr__’ and ‘__str__’
* ‘__dir__’, ‘__format__’ and ‘__subclasses__’
* ‘__round__’, ‘__floor__’, ‘__trunc__’ and ‘__ceil__’
* Comparisons: ‘__lt__’, ‘__gt__’, ‘__le__’, ‘__ge__’, ‘__eq__’ and
‘__ne__’
* Container methods: ‘__getitem__’, ‘__setitem__’, ‘__delitem__’,
‘__contains__’, ‘__len__’, ‘__iter__’, ‘__reversed__’ and
‘__missing__’
* Context manager: ‘__enter__’, ‘__exit__’, ‘__aenter__’ and
‘__aexit__’
* Unary numeric methods: ‘__neg__’, ‘__pos__’ and ‘__invert__’
* The numeric methods (including right hand and in-place variants):
‘__add__’, ‘__sub__’, ‘__mul__’, ‘__matmul__’, ‘__truediv__’,
‘__floordiv__’, ‘__mod__’, ‘__divmod__’, ‘__lshift__’,
‘__rshift__’, ‘__and__’, ‘__xor__’, ‘__or__’, and ‘__pow__’
* Numeric conversion methods: ‘__complex__’, ‘__int__’, ‘__float__’
and ‘__index__’
* Descriptor methods: ‘__get__’, ‘__set__’ and ‘__delete__’
* Pickling: ‘__reduce__’, ‘__reduce_ex__’, ‘__getinitargs__’,
‘__getnewargs__’, ‘__getstate__’ and ‘__setstate__’
* File system path representation: ‘__fspath__’
* Asynchronous iteration methods: ‘__aiter__’ and ‘__anext__’
Changed in version 3.8: Added support for *note
os.PathLike.__fspath__(): c52.
Changed in version 3.8: Added support for ‘__aenter__’, ‘__aexit__’,
‘__aiter__’ and ‘__anext__’.
The following methods exist but are 'not' supported as they are either
in use by mock, can’t be set dynamically, or can cause problems:
* ‘__getattr__’, ‘__setattr__’, ‘__init__’ and ‘__new__’
* ‘__prepare__’, ‘__instancecheck__’, ‘__subclasscheck__’, ‘__del__’
---------- Footnotes ----------
(1) Magic methods 'should' be looked up on the class rather than the
instance. Different versions of Python are inconsistent about applying
this rule. The supported protocol methods should work with all
supported versions of Python.
(2) The function is basically hooked up to the class, but each ‘Mock’
instance is kept isolated from the others.
File: python.info, Node: Magic Mock, Prev: Mocking Magic Methods, Up: MagicMock and magic method support
5.27.6.20 Magic Mock
....................
There are two ‘MagicMock’ variants: *note MagicMock: e8a. and *note
NonCallableMagicMock: 40ca.
-- Class: unittest.mock.MagicMock (*args, **kw)
‘MagicMock’ is a subclass of *note Mock: a4b. with default
implementations of most of the *note magic methods: 4101. You can
use ‘MagicMock’ without having to configure the magic methods
yourself.
The constructor parameters have the same meaning as for *note Mock:
a4b.
If you use the 'spec' or 'spec_set' arguments then 'only' magic
methods that exist in the spec will be created.
-- Class: unittest.mock.NonCallableMagicMock (*args, **kw)
A non-callable version of *note MagicMock: e8a.
The constructor parameters have the same meaning as for *note
MagicMock: e8a, with the exception of 'return_value' and
'side_effect' which have no meaning on a non-callable mock.
The magic methods are setup with *note MagicMock: e8a. objects, so you
can configure them and use them in the usual way:
>>> mock = MagicMock()
>>> mock[3] = 'fish'
>>> mock.__setitem__.assert_called_with(3, 'fish')
>>> mock.__getitem__.return_value = 'result'
>>> mock[2]
'result'
By default many of the protocol methods are required to return objects
of a specific type. These methods are preconfigured with a default
return value, so that they can be used without you having to do anything
if you aren’t interested in the return value. You can still 'set' the
return value manually if you want to change the default.
Methods and their defaults:
* ‘__lt__’: *note NotImplemented: 7cd.
* ‘__gt__’: ‘NotImplemented’
* ‘__le__’: ‘NotImplemented’
* ‘__ge__’: ‘NotImplemented’
* ‘__int__’: ‘1’
* ‘__contains__’: ‘False’
* ‘__len__’: ‘0’
* ‘__iter__’: ‘iter([])’
* ‘__exit__’: ‘False’
* ‘__aexit__’: ‘False’
* ‘__complex__’: ‘1j’
* ‘__float__’: ‘1.0’
* ‘__bool__’: ‘True’
* ‘__index__’: ‘1’
* ‘__hash__’: default hash for the mock
* ‘__str__’: default str for the mock
* ‘__sizeof__’: default sizeof for the mock
For example:
>>> mock = MagicMock()
>>> int(mock)
1
>>> len(mock)
0
>>> list(mock)
[]
>>> object() in mock
False
The two equality methods, ‘__eq__()’ and ‘__ne__()’, are special. They
do the default equality comparison on identity, using the *note
side_effect: 40c4. attribute, unless you change their return value to
return something else:
>>> MagicMock() == 3
False
>>> MagicMock() != 3
True
>>> mock = MagicMock()
>>> mock.__eq__.return_value = True
>>> mock == 3
True
The return value of ‘MagicMock.__iter__()’ can be any iterable object
and isn’t required to be an iterator:
>>> mock = MagicMock()
>>> mock.__iter__.return_value = ['a', 'b', 'c']
>>> list(mock)
['a', 'b', 'c']
>>> list(mock)
['a', 'b', 'c']
If the return value 'is' an iterator, then iterating over it once will
consume it and subsequent iterations will result in an empty list:
>>> mock.__iter__.return_value = iter(['a', 'b', 'c'])
>>> list(mock)
['a', 'b', 'c']
>>> list(mock)
[]
‘MagicMock’ has all of the supported magic methods configured except for
some of the obscure and obsolete ones. You can still set these up if
you want.
Magic methods that are supported but not setup by default in ‘MagicMock’
are:
* ‘__subclasses__’
* ‘__dir__’
* ‘__format__’
* ‘__get__’, ‘__set__’ and ‘__delete__’
* ‘__reversed__’ and ‘__missing__’
* ‘__reduce__’, ‘__reduce_ex__’, ‘__getinitargs__’, ‘__getnewargs__’,
‘__getstate__’ and ‘__setstate__’
* ‘__getformat__’
File: python.info, Node: Helpers, Next: Order of precedence of side_effect return_value and wraps, Prev: MagicMock and magic method support, Up: unittest mock — mock object library
5.27.6.21 Helpers
.................
* Menu:
* sentinel::
* DEFAULT::
* call::
* create_autospec::
* ANY::
* FILTER_DIR::
* mock_open::
* Autospeccing::
* Sealing mocks::
File: python.info, Node: sentinel, Next: DEFAULT, Up: Helpers
5.27.6.22 sentinel
..................
-- Data: unittest.mock.sentinel
The ‘sentinel’ object provides a convenient way of providing unique
objects for your tests.
Attributes are created on demand when you access them by name.
Accessing the same attribute will always return the same object.
The objects returned have a sensible repr so that test failure
messages are readable.
Changed in version 3.7: The ‘sentinel’ attributes now preserve
their identity when they are *note copied: 25. or *note pickled:
a6.
Sometimes when testing you need to test that a specific object is passed
as an argument to another method, or returned. It can be common to
create named sentinel objects to test this. *note sentinel: ba0.
provides a convenient way of creating and testing the identity of
objects like this.
In this example we monkey patch ‘method’ to return
‘sentinel.some_object’:
>>> real = ProductionClass()
>>> real.method = Mock(name="method")
>>> real.method.return_value = sentinel.some_object
>>> result = real.method()
>>> assert result is sentinel.some_object
>>> result
sentinel.some_object
File: python.info, Node: DEFAULT, Next: call, Prev: sentinel, Up: Helpers
5.27.6.23 DEFAULT
.................
-- Data: unittest.mock.DEFAULT
The *note DEFAULT: 40cb. object is a pre-created sentinel (actually
‘sentinel.DEFAULT’). It can be used by *note side_effect: 40c4.
functions to indicate that the normal return value should be used.
File: python.info, Node: call, Next: create_autospec, Prev: DEFAULT, Up: Helpers
5.27.6.24 call
..............
-- Function: unittest.mock.call (*args, **kwargs)
*note call(): 19cd. is a helper object for making simpler
assertions, for comparing with *note call_args: 197f, *note
call_args_list: 40d8, *note mock_calls: 40d0. and *note
method_calls: 40d2. *note call(): 19cd. can also be used with
*note assert_has_calls(): 16a8.
>>> m = MagicMock(return_value=None)
>>> m(1, 2, a='foo', b='bar')
>>> m()
>>> m.call_args_list == [call(1, 2, a='foo', b='bar'), call()]
True
-- Method: call.call_list ()
For a call object that represents multiple calls, *note
call_list(): 4107. returns a list of all the intermediate calls as
well as the final call.
‘call_list’ is particularly useful for making assertions on “chained
calls”. A chained call is multiple calls on a single line of code.
This results in multiple entries in *note mock_calls: 40d0. on a mock.
Manually constructing the sequence of calls can be tedious.
*note call_list(): 4107. can construct the sequence of calls from the
same chained call:
>>> m = MagicMock()
>>> m(1).method(arg='foo').other('bar')(2.0)
>>> kall = call(1).method(arg='foo').other('bar')(2.0)
>>> kall.call_list()
[call(1),
call().method(arg='foo'),
call().method().other('bar'),
call().method().other()(2.0)]
>>> m.mock_calls == kall.call_list()
True
A ‘call’ object is either a tuple of (positional args, keyword args) or
(name, positional args, keyword args) depending on how it was
constructed. When you construct them yourself this isn’t particularly
interesting, but the ‘call’ objects that are in the *note
Mock.call_args: 197f, *note Mock.call_args_list: 40d8. and *note
Mock.mock_calls: 40d0. attributes can be introspected to get at the
individual arguments they contain.
The ‘call’ objects in *note Mock.call_args: 197f. and *note
Mock.call_args_list: 40d8. are two-tuples of (positional args, keyword
args) whereas the ‘call’ objects in *note Mock.mock_calls: 40d0, along
with ones you construct yourself, are three-tuples of (name, positional
args, keyword args).
You can use their “tupleness” to pull out the individual arguments for
more complex introspection and assertions. The positional arguments are
a tuple (an empty tuple if there are no positional arguments) and the
keyword arguments are a dictionary:
>>> m = MagicMock(return_value=None)
>>> m(1, 2, 3, arg='one', arg2='two')
>>> kall = m.call_args
>>> kall.args
(1, 2, 3)
>>> kall.kwargs
{'arg': 'one', 'arg2': 'two'}
>>> kall.args is kall[0]
True
>>> kall.kwargs is kall[1]
True
>>> m = MagicMock()
>>> m.foo(4, 5, 6, arg='two', arg2='three')
>>> kall = m.mock_calls[0]
>>> name, args, kwargs = kall
>>> name
'foo'
>>> args
(4, 5, 6)
>>> kwargs
{'arg': 'two', 'arg2': 'three'}
>>> name is m.mock_calls[0][0]
True
File: python.info, Node: create_autospec, Next: ANY, Prev: call, Up: Helpers
5.27.6.25 create_autospec
.........................
-- Function: unittest.mock.create_autospec (spec, spec_set=False,
instance=False, **kwargs)
Create a mock object using another object as a spec. Attributes on
the mock will use the corresponding attribute on the 'spec' object
as their spec.
Functions or methods being mocked will have their arguments checked
to ensure that they are called with the correct signature.
If 'spec_set' is ‘True’ then attempting to set attributes that
don’t exist on the spec object will raise an *note AttributeError:
348.
If a class is used as a spec then the return value of the mock (the
instance of the class) will have the same spec. You can use a
class as the spec for an instance object by passing
‘instance=True’. The returned mock will only be callable if
instances of the mock are callable.
*note create_autospec(): 1608. also takes arbitrary keyword
arguments that are passed to the constructor of the created mock.
See *note Autospeccing: 40c7. for examples of how to use auto-speccing
with *note create_autospec(): 1608. and the 'autospec' argument to *note
patch(): e8b.
Changed in version 3.8: *note create_autospec(): 1608. now returns an
*note AsyncMock: a4a. if the target is an async function.
File: python.info, Node: ANY, Next: FILTER_DIR, Prev: create_autospec, Up: Helpers
5.27.6.26 ANY
.............
-- Data: unittest.mock.ANY
Sometimes you may need to make assertions about 'some' of the arguments
in a call to mock, but either not care about some of the arguments or
want to pull them individually out of *note call_args: 197f. and make
more complex assertions on them.
To ignore certain arguments you can pass in objects that compare equal
to 'everything'. Calls to *note assert_called_with(): 1a20. and *note
assert_called_once_with(): 40ce. will then succeed no matter what was
passed in.
>>> mock = Mock(return_value=None)
>>> mock('foo', bar=object())
>>> mock.assert_called_once_with('foo', bar=ANY)
*note ANY: 19cb. can also be used in comparisons with call lists like
*note mock_calls: 40d0.:
>>> m = MagicMock(return_value=None)
>>> m(1)
>>> m(1, 2)
>>> m(object())
>>> m.mock_calls == [call(1), call(1, 2), ANY]
True
*note ANY: 19cb. is not limited to comparisons with call objects and so
can also be used in test assertions:
class TestStringMethods(unittest.TestCase):
def test_split(self):
s = 'hello world'
self.assertEqual(s.split(), ['hello', ANY])
File: python.info, Node: FILTER_DIR, Next: mock_open, Prev: ANY, Up: Helpers
5.27.6.27 FILTER_DIR
....................
-- Data: unittest.mock.FILTER_DIR
*note FILTER_DIR: 40d4. is a module level variable that controls the way
mock objects respond to *note dir(): 62e. The default is ‘True’, which
uses the filtering described below, to only show useful members. If you
dislike this filtering, or need to switch it off for diagnostic
purposes, then set ‘mock.FILTER_DIR = False’.
With filtering on, ‘dir(some_mock)’ shows only useful attributes and
will include any dynamically created attributes that wouldn’t normally
be shown. If the mock was created with a 'spec' (or 'autospec' of
course) then all the attributes from the original are shown, even if
they haven’t been accessed yet:
>>> dir(Mock())
['assert_any_call',
'assert_called',
'assert_called_once',
'assert_called_once_with',
'assert_called_with',
'assert_has_calls',
'assert_not_called',
'attach_mock',
...
>>> from urllib import request
>>> dir(Mock(spec=request))
['AbstractBasicAuthHandler',
'AbstractDigestAuthHandler',
'AbstractHTTPHandler',
'BaseHandler',
...
Many of the not-very-useful (private to *note Mock: a4b. rather than the
thing being mocked) underscore and double underscore prefixed attributes
have been filtered from the result of calling *note dir(): 62e. on a
*note Mock: a4b. If you dislike this behaviour you can switch it off by
setting the module level switch *note FILTER_DIR: 40d4.:
>>> from unittest import mock
>>> mock.FILTER_DIR = False
>>> dir(mock.Mock())
['_NonCallableMock__get_return_value',
'_NonCallableMock__get_side_effect',
'_NonCallableMock__return_value_doc',
'_NonCallableMock__set_return_value',
'_NonCallableMock__set_side_effect',
'__call__',
'__class__',
...
Alternatively you can just use ‘vars(my_mock)’ (instance members) and
‘dir(type(my_mock))’ (type members) to bypass the filtering irrespective
of *note FILTER_DIR: 40d4.
File: python.info, Node: mock_open, Next: Autospeccing, Prev: FILTER_DIR, Up: Helpers
5.27.6.28 mock_open
...................
-- Function: unittest.mock.mock_open (mock=None, read_data=None)
A helper function to create a mock to replace the use of *note
open(): 517. It works for *note open(): 517. called directly or
used as a context manager.
The 'mock' argument is the mock object to configure. If ‘None’
(the default) then a *note MagicMock: e8a. will be created for you,
with the API limited to methods or attributes available on standard
file handles.
'read_data' is a string for the *note read(): e1b, *note
readline(): 131c, and *note readlines(): 2333. methods of the file
handle to return. Calls to those methods will take data from
'read_data' until it is depleted. The mock of these methods is
pretty simplistic: every time the 'mock' is called, the 'read_data'
is rewound to the start. If you need more control over the data
that you are feeding to the tested code you will need to customize
this mock for yourself. When that is insufficient, one of the
in-memory filesystem packages on PyPI(1) can offer a realistic
filesystem for testing.
Changed in version 3.4: Added *note readline(): 131c. and *note
readlines(): 2333. support. The mock of *note read(): e1b. changed
to consume 'read_data' rather than returning it on each call.
Changed in version 3.5: 'read_data' is now reset on each call to
the 'mock'.
Changed in version 3.8: Added *note __iter__(): 1cc1. to
implementation so that iteration (such as in for loops) correctly
consumes 'read_data'.
Using *note open(): 517. as a context manager is a great way to ensure
your file handles are closed properly and is becoming common:
with open('/some/path', 'w') as f:
f.write('something')
The issue is that even if you mock out the call to *note open(): 517. it
is the 'returned object' that is used as a context manager (and has
*note __enter__(): 5c4. and *note __exit__(): 12f3. called).
Mocking context managers with a *note MagicMock: e8a. is common enough
and fiddly enough that a helper function is useful.
>>> m = mock_open()
>>> with patch('__main__.open', m):
... with open('foo', 'w') as h:
... h.write('some stuff')
...
>>> m.mock_calls
[call('foo', 'w'),
call().__enter__(),
call().write('some stuff'),
call().__exit__(None, None, None)]
>>> m.assert_called_once_with('foo', 'w')
>>> handle = m()
>>> handle.write.assert_called_once_with('some stuff')
And for reading files:
>>> with patch('__main__.open', mock_open(read_data='bibble')) as m:
... with open('foo') as h:
... result = h.read()
...
>>> m.assert_called_once_with('foo')
>>> assert result == 'bibble'
---------- Footnotes ----------
(1) https://pypi.org
File: python.info, Node: Autospeccing, Next: Sealing mocks, Prev: mock_open, Up: Helpers
5.27.6.29 Autospeccing
......................
Autospeccing is based on the existing ‘spec’ feature of mock. It limits
the api of mocks to the api of an original object (the spec), but it is
recursive (implemented lazily) so that attributes of mocks only have the
same api as the attributes of the spec. In addition mocked functions /
methods have the same call signature as the original so they raise a
*note TypeError: 534. if they are called incorrectly.
Before I explain how auto-speccing works, here’s why it is needed.
*note Mock: a4b. is a very powerful and flexible object, but it suffers
from a flaw which is general to mocking. If you refactor some of your
code, rename members and so on, any tests for code that is still using
the 'old api' but uses mocks instead of the real objects will still
pass. This means your tests can all pass even though your code is
broken.
Changed in version 3.5: Before 3.5, tests with a typo in the word assert
would silently pass when they should raise an error. You can still
achieve this behavior by passing ‘unsafe=True’ to Mock.
Note that this is another reason why you need integration tests as well
as unit tests. Testing everything in isolation is all fine and dandy,
but if you don’t test how your units are “wired together” there is still
lots of room for bugs that tests might have caught.
*note unittest.mock: 107. already provides a feature to help with this,
called speccing. If you use a class or instance as the ‘spec’ for a
mock then you can only access attributes on the mock that exist on the
real class:
>>> from urllib import request
>>> mock = Mock(spec=request.Request)
>>> mock.assret_called_with # Intentional typo!
Traceback (most recent call last):
...
AttributeError: Mock object has no attribute 'assret_called_with'
The spec only applies to the mock itself, so we still have the same
issue with any methods on the mock:
>>> mock.header_items()
>>> mock.header_items.assret_called_with() # Intentional typo!
Auto-speccing solves this problem. You can either pass ‘autospec=True’
to *note patch(): e8b. / *note patch.object(): 1598. or use the *note
create_autospec(): 1608. function to create a mock with a spec. If you
use the ‘autospec=True’ argument to *note patch(): e8b. then the object
that is being replaced will be used as the spec object. Because the
speccing is done “lazily” (the spec is created as attributes on the mock
are accessed) you can use it with very complex or deeply nested objects
(like modules that import modules that import modules) without a big
performance hit.
Here’s an example of it in use:
>>> from urllib import request
>>> patcher = patch('__main__.request', autospec=True)
>>> mock_request = patcher.start()
>>> request is mock_request
True
>>> mock_request.Request
You can see that ‘request.Request’ has a spec. ‘request.Request’ takes
two arguments in the constructor (one of which is 'self'). Here’s what
happens if we try to call it incorrectly:
>>> req = request.Request()
Traceback (most recent call last):
...
TypeError: () takes at least 2 arguments (1 given)
The spec also applies to instantiated classes (i.e. the return value of
specced mocks):
>>> req = request.Request('foo')
>>> req
‘Request’ objects are not callable, so the return value of instantiating
our mocked out ‘request.Request’ is a non-callable mock. With the spec
in place any typos in our asserts will raise the correct error:
>>> req.add_header('spam', 'eggs')
>>> req.add_header.assret_called_with # Intentional typo!
Traceback (most recent call last):
...
AttributeError: Mock object has no attribute 'assret_called_with'
>>> req.add_header.assert_called_with('spam', 'eggs')
In many cases you will just be able to add ‘autospec=True’ to your
existing *note patch(): e8b. calls and then be protected against bugs
due to typos and api changes.
As well as using 'autospec' through *note patch(): e8b. there is a *note
create_autospec(): 1608. for creating autospecced mocks directly:
>>> from urllib import request
>>> mock_request = create_autospec(request)
>>> mock_request.Request('foo', 'bar')
This isn’t without caveats and limitations however, which is why it is
not the default behaviour. In order to know what attributes are
available on the spec object, autospec has to introspect (access
attributes) the spec. As you traverse attributes on the mock a
corresponding traversal of the original object is happening under the
hood. If any of your specced objects have properties or descriptors
that can trigger code execution then you may not be able to use
autospec. On the other hand it is much better to design your objects so
that introspection is safe (1).
A more serious problem is that it is common for instance attributes to
be created in the *note __init__(): 6ac. method and not to exist on the
class at all. 'autospec' can’t know about any dynamically created
attributes and restricts the api to visible attributes.
>>> class Something:
... def __init__(self):
... self.a = 33
...
>>> with patch('__main__.Something', autospec=True):
... thing = Something()
... thing.a
...
Traceback (most recent call last):
...
AttributeError: Mock object has no attribute 'a'
There are a few different ways of resolving this problem. The easiest,
but not necessarily the least annoying, way is to simply set the
required attributes on the mock after creation. Just because 'autospec'
doesn’t allow you to fetch attributes that don’t exist on the spec it
doesn’t prevent you setting them:
>>> with patch('__main__.Something', autospec=True):
... thing = Something()
... thing.a = 33
...
There is a more aggressive version of both 'spec' and 'autospec' that
'does' prevent you setting non-existent attributes. This is useful if
you want to ensure your code only 'sets' valid attributes too, but
obviously it prevents this particular scenario:
>>> with patch('__main__.Something', autospec=True, spec_set=True):
... thing = Something()
... thing.a = 33
...
Traceback (most recent call last):
...
AttributeError: Mock object has no attribute 'a'
Probably the best way of solving the problem is to add class attributes
as default values for instance members initialised in *note __init__():
6ac. Note that if you are only setting default attributes in
‘__init__()’ then providing them via class attributes (shared between
instances of course) is faster too. e.g.
class Something:
a = 33
This brings up another issue. It is relatively common to provide a
default value of ‘None’ for members that will later be an object of a
different type. ‘None’ would be useless as a spec because it wouldn’t
let you access 'any' attributes or methods on it. As ‘None’ is 'never'
going to be useful as a spec, and probably indicates a member that will
normally of some other type, autospec doesn’t use a spec for members
that are set to ‘None’. These will just be ordinary mocks (well -
MagicMocks):
>>> class Something:
... member = None
...
>>> mock = create_autospec(Something)
>>> mock.member.foo.bar.baz()
If modifying your production classes to add defaults isn’t to your
liking then there are more options. One of these is simply to use an
instance as the spec rather than the class. The other is to create a
subclass of the production class and add the defaults to the subclass
without affecting the production class. Both of these require you to
use an alternative object as the spec. Thankfully *note patch(): e8b.
supports this - you can simply pass the alternative object as the
'autospec' argument:
>>> class Something:
... def __init__(self):
... self.a = 33
...
>>> class SomethingForTest(Something):
... a = 33
...
>>> p = patch('__main__.Something', autospec=SomethingForTest)
>>> mock = p.start()
>>> mock.a
---------- Footnotes ----------
(1) This only applies to classes or already instantiated objects.
Calling a mocked class to create a mock instance 'does not' create a
real instance. It is only attribute lookups - along with calls to *note
dir(): 62e. - that are done.
File: python.info, Node: Sealing mocks, Prev: Autospeccing, Up: Helpers
5.27.6.30 Sealing mocks
.......................
-- Function: unittest.mock.seal (mock)
Seal will disable the automatic creation of mocks when accessing an
attribute of the mock being sealed or any of its attributes that
are already mocks recursively.
If a mock instance with a name or a spec is assigned to an
attribute it won’t be considered in the sealing chain. This allows
one to prevent seal from fixing part of the mock object.
>>> mock = Mock()
>>> mock.submock.attribute1 = 2
>>> mock.not_submock = mock.Mock(name="sample_name")
>>> seal(mock)
>>> mock.new_attribute # This will raise AttributeError.
>>> mock.submock.attribute2 # This will raise AttributeError.
>>> mock.not_submock.attribute2 # This won't raise.
Added in version 3.7.
File: python.info, Node: Order of precedence of side_effect return_value and wraps, Prev: Helpers, Up: unittest mock — mock object library
5.27.6.31 Order of precedence of ‘side_effect’, ‘return_value’ and 'wraps'
..........................................................................
The order of their precedence is:
1. *note side_effect: 40c4.
2. *note return_value: 40cc.
3. 'wraps'
If all three are set, mock will return the value from *note side_effect:
40c4, ignoring *note return_value: 40cc. and the wrapped object
altogether. If any two are set, the one with the higher precedence will
return the value. Regardless of the order of which was set first, the
order of precedence remains unchanged.
>>> from unittest.mock import Mock
>>> class Order:
... @staticmethod
... def get_value():
... return "third"
...
>>> order_mock = Mock(spec=Order, wraps=Order)
>>> order_mock.get_value.side_effect = ["first"]
>>> order_mock.get_value.return_value = "second"
>>> order_mock.get_value()
'first'
As ‘None’ is the default value of *note side_effect: 40c4, if you
reassign its value back to ‘None’, the order of precedence will be
checked between *note return_value: 40cc. and the wrapped object,
ignoring *note side_effect: 40c4.
>>> order_mock.get_value.side_effect = None
>>> order_mock.get_value()
'second'
If the value being returned by *note side_effect: 40c4. is *note
DEFAULT: 40cb, it is ignored and the order of precedence moves to the
successor to obtain the value to return.
>>> from unittest.mock import DEFAULT
>>> order_mock.get_value.side_effect = [DEFAULT]
>>> order_mock.get_value()
'second'
When *note Mock: a4b. wraps an object, the default value of *note
return_value: 40cc. will be *note DEFAULT: 40cb.
>>> order_mock = Mock(spec=Order, wraps=Order)
>>> order_mock.return_value
sentinel.DEFAULT
>>> order_mock.get_value.return_value
sentinel.DEFAULT
The order of precedence will ignore this value and it will move to the
last successor which is the wrapped object.
As the real call is being made to the wrapped object, creating an
instance of this mock will return the real instance of the class. The
positional arguments, if any, required by the wrapped object must be
passed.
>>> order_mock_instance = order_mock()
>>> isinstance(order_mock_instance, Order)
True
>>> order_mock_instance.get_value()
'third'
>>> order_mock.get_value.return_value = DEFAULT
>>> order_mock.get_value()
'third'
>>> order_mock.get_value.return_value = "second"
>>> order_mock.get_value()
'second'
But if you assign ‘None’ to it, this will not be ignored as it is an
explicit assignment. So, the order of precedence will not move to the
wrapped object.
>>> order_mock.get_value.return_value = None
>>> order_mock.get_value() is None
True
Even if you set all three at once when initializing the mock, the order
of precedence remains the same:
>>> order_mock = Mock(spec=Order, wraps=Order,
... **{"get_value.side_effect": ["first"],
... "get_value.return_value": "second"}
... )
...
>>> order_mock.get_value()
'first'
>>> order_mock.get_value.side_effect = None
>>> order_mock.get_value()
'second'
>>> order_mock.get_value.return_value = DEFAULT
>>> order_mock.get_value()
'third'
If *note side_effect: 40c4. is exhausted, the order of precedence will
not cause a value to be obtained from the successors. Instead,
‘StopIteration’ exception is raised.
>>> order_mock = Mock(spec=Order, wraps=Order)
>>> order_mock.get_value.side_effect = ["first side effect value",
... "another side effect value"]
>>> order_mock.get_value.return_value = "second"
>>> order_mock.get_value()
'first side effect value'
>>> order_mock.get_value()
'another side effect value'
>>> order_mock.get_value()
Traceback (most recent call last):
...
StopIteration
File: python.info, Node: unittest mock — getting started, Next: test — Regression tests package for Python, Prev: unittest mock — mock object library, Up: Development Tools
5.27.7 ‘unittest.mock’ — getting started
----------------------------------------
Added in version 3.3.
* Menu:
* Using Mock::
* Patch Decorators::
* Further Examples::
File: python.info, Node: Using Mock, Next: Patch Decorators, Up: unittest mock — getting started
5.27.7.1 Using Mock
...................
* Menu:
* Mock Patching Methods::
* Mock for Method Calls on an Object::
* Mocking Classes::
* Naming your mocks::
* Tracking all Calls::
* Setting Return Values and Attributes::
* Raising exceptions with mocks::
* Side effect functions and iterables::
* Mocking asynchronous iterators::
* Mocking asynchronous context manager::
* Creating a Mock from an Existing Object::
* Using side_effect to return per file content::
File: python.info, Node: Mock Patching Methods, Next: Mock for Method Calls on an Object, Up: Using Mock
5.27.7.2 Mock Patching Methods
..............................
Common uses for *note Mock: a4b. objects include:
* Patching methods
* Recording method calls on objects
You might want to replace a method on an object to check that it is
called with the correct arguments by another part of the system:
>>> real = SomeClass()
>>> real.method = MagicMock(name='method')
>>> real.method(3, 4, 5, key='value')
Once our mock has been used (‘real.method’ in this example) it has
methods and attributes that allow you to make assertions about how it
has been used.
Note: In most of these examples the *note Mock: a4b. and *note
MagicMock: e8a. classes are interchangeable. As the ‘MagicMock’ is
the more capable class it makes a sensible one to use by default.
Once the mock has been called its *note called: 40d6. attribute is set
to ‘True’. More importantly we can use the *note assert_called_with():
1a20. or *note assert_called_once_with(): 40ce. method to check that it
was called with the correct arguments.
This example tests that calling ‘ProductionClass().method’ results in a
call to the ‘something’ method:
>>> class ProductionClass:
... def method(self):
... self.something(1, 2, 3)
... def something(self, a, b, c):
... pass
...
>>> real = ProductionClass()
>>> real.something = MagicMock()
>>> real.method()
>>> real.something.assert_called_once_with(1, 2, 3)
File: python.info, Node: Mock for Method Calls on an Object, Next: Mocking Classes, Prev: Mock Patching Methods, Up: Using Mock
5.27.7.3 Mock for Method Calls on an Object
...........................................
In the last example we patched a method directly on an object to check
that it was called correctly. Another common use case is to pass an
object into a method (or some part of the system under test) and then
check that it is used in the correct way.
The simple ‘ProductionClass’ below has a ‘closer’ method. If it is
called with an object then it calls ‘close’ on it.
>>> class ProductionClass:
... def closer(self, something):
... something.close()
...
So to test it we need to pass in an object with a ‘close’ method and
check that it was called correctly.
>>> real = ProductionClass()
>>> mock = Mock()
>>> real.closer(mock)
>>> mock.close.assert_called_with()
We don’t have to do any work to provide the ‘close’ method on our mock.
Accessing close creates it. So, if ‘close’ hasn’t already been called
then accessing it in the test will create it, but *note
assert_called_with(): 1a20. will raise a failure exception.
File: python.info, Node: Mocking Classes, Next: Naming your mocks, Prev: Mock for Method Calls on an Object, Up: Using Mock
5.27.7.4 Mocking Classes
........................
A common use case is to mock out classes instantiated by your code under
test. When you patch a class, then that class is replaced with a mock.
Instances are created by 'calling the class'. This means you access the
“mock instance” by looking at the return value of the mocked class.
In the example below we have a function ‘some_function’ that
instantiates ‘Foo’ and calls a method on it. The call to *note patch():
e8b. replaces the class ‘Foo’ with a mock. The ‘Foo’ instance is the
result of calling the mock, so it is configured by modifying the mock
*note return_value: 40cc.
>>> def some_function():
... instance = module.Foo()
... return instance.method()
...
>>> with patch('module.Foo') as mock:
... instance = mock.return_value
... instance.method.return_value = 'the result'
... result = some_function()
... assert result == 'the result'
File: python.info, Node: Naming your mocks, Next: Tracking all Calls, Prev: Mocking Classes, Up: Using Mock
5.27.7.5 Naming your mocks
..........................
It can be useful to give your mocks a name. The name is shown in the
repr of the mock and can be helpful when the mock appears in test
failure messages. The name is also propagated to attributes or methods
of the mock:
>>> mock = MagicMock(name='foo')
>>> mock
>>> mock.method
File: python.info, Node: Tracking all Calls, Next: Setting Return Values and Attributes, Prev: Naming your mocks, Up: Using Mock
5.27.7.6 Tracking all Calls
...........................
Often you want to track more than a single call to a method. The *note
mock_calls: 40d0. attribute records all calls to child attributes of the
mock - and also to their children.
>>> mock = MagicMock()
>>> mock.method()
>>> mock.attribute.method(10, x=53)
>>> mock.mock_calls
[call.method(), call.attribute.method(10, x=53)]
If you make an assertion about ‘mock_calls’ and any unexpected methods
have been called, then the assertion will fail. This is useful because
as well as asserting that the calls you expected have been made, you are
also checking that they were made in the right order and with no
additional calls:
You use the *note call: 19cd. object to construct lists for comparing
with ‘mock_calls’:
>>> expected = [call.method(), call.attribute.method(10, x=53)]
>>> mock.mock_calls == expected
True
However, parameters to calls that return mocks are not recorded, which
means it is not possible to track nested calls where the parameters used
to create ancestors are important:
>>> m = Mock()
>>> m.factory(important=True).deliver()
>>> m.mock_calls[-1] == call.factory(important=False).deliver()
True
File: python.info, Node: Setting Return Values and Attributes, Next: Raising exceptions with mocks, Prev: Tracking all Calls, Up: Using Mock
5.27.7.7 Setting Return Values and Attributes
.............................................
Setting the return values on a mock object is trivially easy:
>>> mock = Mock()
>>> mock.return_value = 3
>>> mock()
3
Of course you can do the same for methods on the mock:
>>> mock = Mock()
>>> mock.method.return_value = 3
>>> mock.method()
3
The return value can also be set in the constructor:
>>> mock = Mock(return_value=3)
>>> mock()
3
If you need an attribute setting on your mock, just do it:
>>> mock = Mock()
>>> mock.x = 3
>>> mock.x
3
Sometimes you want to mock up a more complex situation, like for example
‘mock.connection.cursor().execute("SELECT 1")’. If we wanted this call
to return a list, then we have to configure the result of the nested
call.
We can use *note call: 19cd. to construct the set of calls in a “chained
call” like this for easy assertion afterwards:
>>> mock = Mock()
>>> cursor = mock.connection.cursor.return_value
>>> cursor.execute.return_value = ['foo']
>>> mock.connection.cursor().execute("SELECT 1")
['foo']
>>> expected = call.connection.cursor().execute("SELECT 1").call_list()
>>> mock.mock_calls
[call.connection.cursor(), call.connection.cursor().execute('SELECT 1')]
>>> mock.mock_calls == expected
True
It is the call to ‘.call_list()’ that turns our call object into a list
of calls representing the chained calls.
File: python.info, Node: Raising exceptions with mocks, Next: Side effect functions and iterables, Prev: Setting Return Values and Attributes, Up: Using Mock
5.27.7.8 Raising exceptions with mocks
......................................
A useful attribute is *note side_effect: 40c4. If you set this to an
exception class or instance then the exception will be raised when the
mock is called.
>>> mock = Mock(side_effect=Exception('Boom!'))
>>> mock()
Traceback (most recent call last):
...
Exception: Boom!
File: python.info, Node: Side effect functions and iterables, Next: Mocking asynchronous iterators, Prev: Raising exceptions with mocks, Up: Using Mock
5.27.7.9 Side effect functions and iterables
............................................
‘side_effect’ can also be set to a function or an iterable. The use
case for ‘side_effect’ as an iterable is where your mock is going to be
called several times, and you want each call to return a different
value. When you set ‘side_effect’ to an iterable every call to the mock
returns the next value from the iterable:
>>> mock = MagicMock(side_effect=[4, 5, 6])
>>> mock()
4
>>> mock()
5
>>> mock()
6
For more advanced use cases, like dynamically varying the return values
depending on what the mock is called with, ‘side_effect’ can be a
function. The function will be called with the same arguments as the
mock. Whatever the function returns is what the call returns:
>>> vals = {(1, 2): 1, (2, 3): 2}
>>> def side_effect(*args):
... return vals[args]
...
>>> mock = MagicMock(side_effect=side_effect)
>>> mock(1, 2)
1
>>> mock(2, 3)
2
File: python.info, Node: Mocking asynchronous iterators, Next: Mocking asynchronous context manager, Prev: Side effect functions and iterables, Up: Using Mock
5.27.7.10 Mocking asynchronous iterators
........................................
Since Python 3.8, ‘AsyncMock’ and ‘MagicMock’ have support to mock *note
Asynchronous Iterators: d79. through ‘__aiter__’. The *note
return_value: 40cc. attribute of ‘__aiter__’ can be used to set the
return values to be used for iteration.
>>> mock = MagicMock() # AsyncMock also works here
>>> mock.__aiter__.return_value = [1, 2, 3]
>>> async def main():
... return [i async for i in mock]
...
>>> asyncio.run(main())
[1, 2, 3]
File: python.info, Node: Mocking asynchronous context manager, Next: Creating a Mock from an Existing Object, Prev: Mocking asynchronous iterators, Up: Using Mock
5.27.7.11 Mocking asynchronous context manager
..............................................
Since Python 3.8, ‘AsyncMock’ and ‘MagicMock’ have support to mock *note
Asynchronous Context Managers: 5f8. through ‘__aenter__’ and
‘__aexit__’. By default, ‘__aenter__’ and ‘__aexit__’ are ‘AsyncMock’
instances that return an async function.
>>> class AsyncContextManager:
... async def __aenter__(self):
... return self
... async def __aexit__(self, exc_type, exc, tb):
... pass
...
>>> mock_instance = MagicMock(AsyncContextManager()) # AsyncMock also works here
>>> async def main():
... async with mock_instance as result:
... pass
...
>>> asyncio.run(main())
>>> mock_instance.__aenter__.assert_awaited_once()
>>> mock_instance.__aexit__.assert_awaited_once()
File: python.info, Node: Creating a Mock from an Existing Object, Next: Using side_effect to return per file content, Prev: Mocking asynchronous context manager, Up: Using Mock
5.27.7.12 Creating a Mock from an Existing Object
.................................................
One problem with over use of mocking is that it couples your tests to
the implementation of your mocks rather than your real code. Suppose
you have a class that implements ‘some_method’. In a test for another
class, you provide a mock of this object that 'also' provides
‘some_method’. If later you refactor the first class, so that it no
longer has ‘some_method’ - then your tests will continue to pass even
though your code is now broken!
*note Mock: a4b. allows you to provide an object as a specification for
the mock, using the 'spec' keyword argument. Accessing methods /
attributes on the mock that don’t exist on your specification object
will immediately raise an attribute error. If you change the
implementation of your specification, then tests that use that class
will start failing immediately without you having to instantiate the
class in those tests.
>>> mock = Mock(spec=SomeClass)
>>> mock.old_method()
Traceback (most recent call last):
...
AttributeError: Mock object has no attribute 'old_method'. Did you mean: 'class_method'?
Using a specification also enables a smarter matching of calls made to
the mock, regardless of whether some parameters were passed as
positional or named arguments:
>>> def f(a, b, c): pass
...
>>> mock = Mock(spec=f)
>>> mock(1, 2, 3)
>>> mock.assert_called_with(a=1, b=2, c=3)
If you want this smarter matching to also work with method calls on the
mock, you can use *note auto-speccing: 40c7.
If you want a stronger form of specification that prevents the setting
of arbitrary attributes as well as the getting of them then you can use
'spec_set' instead of 'spec'.
File: python.info, Node: Using side_effect to return per file content, Prev: Creating a Mock from an Existing Object, Up: Using Mock
5.27.7.13 Using side_effect to return per file content
......................................................
*note mock_open(): 1709. is used to patch *note open(): 517. method.
*note side_effect: 40c4. can be used to return a new Mock object per
call. This can be used to return different contents per file stored in
a dictionary:
DEFAULT = "default"
data_dict = {"file1": "data1",
"file2": "data2"}
def open_side_effect(name):
return mock_open(read_data=data_dict.get(name, DEFAULT))()
with patch("builtins.open", side_effect=open_side_effect):
with open("file1") as file1:
assert file1.read() == "data1"
with open("file2") as file2:
assert file2.read() == "data2"
with open("file3") as file2:
assert file2.read() == "default"
File: python.info, Node: Patch Decorators, Next: Further Examples, Prev: Using Mock, Up: unittest mock — getting started
5.27.7.14 Patch Decorators
..........................
Note: With *note patch(): e8b. it matters that you patch objects in
the namespace where they are looked up. This is normally
straightforward, but for a quick guide read *note where to patch:
40c5.
A common need in tests is to patch a class attribute or a module
attribute, for example patching a builtin or patching a class in a
module to test that it is instantiated. Modules and classes are
effectively global, so patching on them has to be undone after the test
or the patch will persist into other tests and cause hard to diagnose
problems.
mock provides three convenient decorators for this: *note patch(): e8b,
*note patch.object(): 1598. and *note patch.dict(): 19f2. ‘patch’ takes
a single string, of the form ‘package.module.Class.attribute’ to specify
the attribute you are patching. It also optionally takes a value that
you want the attribute (or class or whatever) to be replaced with.
‘patch.object’ takes an object and the name of the attribute you would
like patched, plus optionally the value to patch it with.
‘patch.object’:
>>> original = SomeClass.attribute
>>> @patch.object(SomeClass, 'attribute', sentinel.attribute)
... def test():
... assert SomeClass.attribute == sentinel.attribute
...
>>> test()
>>> assert SomeClass.attribute == original
>>> @patch('package.module.attribute', sentinel.attribute)
... def test():
... from package.module import attribute
... assert attribute is sentinel.attribute
...
>>> test()
If you are patching a module (including *note builtins: 12.) then use
*note patch(): e8b. instead of *note patch.object(): 1598.:
>>> mock = MagicMock(return_value=sentinel.file_handle)
>>> with patch('builtins.open', mock):
... handle = open('filename', 'r')
...
>>> mock.assert_called_with('filename', 'r')
>>> assert handle == sentinel.file_handle, "incorrect file handle returned"
The module name can be ‘dotted’, in the form ‘package.module’ if needed:
>>> @patch('package.module.ClassName.attribute', sentinel.attribute)
... def test():
... from package.module import ClassName
... assert ClassName.attribute == sentinel.attribute
...
>>> test()
A nice pattern is to actually decorate test methods themselves:
>>> class MyTest(unittest.TestCase):
... @patch.object(SomeClass, 'attribute', sentinel.attribute)
... def test_something(self):
... self.assertEqual(SomeClass.attribute, sentinel.attribute)
...
>>> original = SomeClass.attribute
>>> MyTest('test_something').test_something()
>>> assert SomeClass.attribute == original
If you want to patch with a Mock, you can use *note patch(): e8b. with
only one argument (or *note patch.object(): 1598. with two arguments).
The mock will be created for you and passed into the test function /
method:
>>> class MyTest(unittest.TestCase):
... @patch.object(SomeClass, 'static_method')
... def test_something(self, mock_method):
... SomeClass.static_method()
... mock_method.assert_called_with()
...
>>> MyTest('test_something').test_something()
You can stack up multiple patch decorators using this pattern:
>>> class MyTest(unittest.TestCase):
... @patch('package.module.ClassName1')
... @patch('package.module.ClassName2')
... def test_something(self, MockClass2, MockClass1):
... self.assertIs(package.module.ClassName1, MockClass1)
... self.assertIs(package.module.ClassName2, MockClass2)
...
>>> MyTest('test_something').test_something()
When you nest patch decorators the mocks are passed in to the decorated
function in the same order they applied (the normal 'Python' order that
decorators are applied). This means from the bottom up, so in the
example above the mock for ‘test_module.ClassName2’ is passed in first.
There is also *note patch.dict(): 19f2. for setting values in a
dictionary just during a scope and restoring the dictionary to its
original state when the test ends:
>>> foo = {'key': 'value'}
>>> original = foo.copy()
>>> with patch.dict(foo, {'newkey': 'newvalue'}, clear=True):
... assert foo == {'newkey': 'newvalue'}
...
>>> assert foo == original
‘patch’, ‘patch.object’ and ‘patch.dict’ can all be used as context
managers.
Where you use *note patch(): e8b. to create a mock for you, you can get
a reference to the mock using the “as” form of the with statement:
>>> class ProductionClass:
... def method(self):
... pass
...
>>> with patch.object(ProductionClass, 'method') as mock_method:
... mock_method.return_value = None
... real = ProductionClass()
... real.method(1, 2, 3)
...
>>> mock_method.assert_called_with(1, 2, 3)
As an alternative ‘patch’, ‘patch.object’ and ‘patch.dict’ can be used
as class decorators. When used in this way it is the same as applying
the decorator individually to every method whose name starts with
“test”.
File: python.info, Node: Further Examples, Prev: Patch Decorators, Up: unittest mock — getting started
5.27.7.15 Further Examples
..........................
Here are some more examples for some slightly more advanced scenarios.
* Menu:
* Mocking chained calls::
* Partial mocking::
* Mocking a Generator Method::
* Applying the same patch to every test method::
* Mocking Unbound Methods::
* Checking multiple calls with mock::
* Coping with mutable arguments::
* Nesting Patches::
* Mocking a dictionary with MagicMock::
* Mock subclasses and their attributes::
* Mocking imports with patch.dict: Mocking imports with patch dict.
* Tracking order of calls and less verbose call assertions::
* More complex argument matching::
File: python.info, Node: Mocking chained calls, Next: Partial mocking, Up: Further Examples
5.27.7.16 Mocking chained calls
...............................
Mocking chained calls is actually straightforward with mock once you
understand the *note return_value: 40cc. attribute. When a mock is
called for the first time, or you fetch its ‘return_value’ before it has
been called, a new *note Mock: a4b. is created.
This means that you can see how the object returned from a call to a
mocked object has been used by interrogating the ‘return_value’ mock:
>>> mock = Mock()
>>> mock().foo(a=2, b=3)
>>> mock.return_value.foo.assert_called_with(a=2, b=3)
From here it is a simple step to configure and then make assertions
about chained calls. Of course another alternative is writing your code
in a more testable way in the first place…
So, suppose we have some code that looks a little bit like this:
>>> class Something:
... def __init__(self):
... self.backend = BackendProvider()
... def method(self):
... response = self.backend.get_endpoint('foobar').create_call('spam', 'eggs').start_call()
... # more code
Assuming that ‘BackendProvider’ is already well tested, how do we test
‘method()’? Specifically, we want to test that the code section ‘# more
code’ uses the response object in the correct way.
As this chain of calls is made from an instance attribute we can monkey
patch the ‘backend’ attribute on a ‘Something’ instance. In this
particular case we are only interested in the return value from the
final call to ‘start_call’ so we don’t have much configuration to do.
Let’s assume the object it returns is ‘file-like’, so we’ll ensure that
our response object uses the builtin *note open(): 517. as its ‘spec’.
To do this we create a mock instance as our mock backend and create a
mock response object for it. To set the response as the return value
for that final ‘start_call’ we could do this:
mock_backend.get_endpoint.return_value.create_call.return_value.start_call.return_value = mock_response
We can do that in a slightly nicer way using the *note configure_mock():
40cd. method to directly set the return value for us:
>>> something = Something()
>>> mock_response = Mock(spec=open)
>>> mock_backend = Mock()
>>> config = {'get_endpoint.return_value.create_call.return_value.start_call.return_value': mock_response}
>>> mock_backend.configure_mock(**config)
With these we monkey patch the “mock backend” in place and can make the
real call:
>>> something.backend = mock_backend
>>> something.method()
Using *note mock_calls: 40d0. we can check the chained call with a
single assert. A chained call is several calls in one line of code, so
there will be several entries in ‘mock_calls’. We can use *note
call.call_list(): 4107. to create this list of calls for us:
>>> chained = call.get_endpoint('foobar').create_call('spam', 'eggs').start_call()
>>> call_list = chained.call_list()
>>> assert mock_backend.mock_calls == call_list
File: python.info, Node: Partial mocking, Next: Mocking a Generator Method, Prev: Mocking chained calls, Up: Further Examples
5.27.7.17 Partial mocking
.........................
In some tests I wanted to mock out a call to *note
datetime.date.today(): 1495. to return a known date, but I didn’t want
to prevent the code under test from creating new date objects.
Unfortunately *note datetime.date: 1cd. is written in C, and so I
couldn’t just monkey-patch out the static *note datetime.date.today():
1495. method.
I found a simple way of doing this that involved effectively wrapping
the date class with a mock, but passing through calls to the constructor
to the real class (and returning real instances).
The *note patch decorator: e8b. is used here to mock out the ‘date’
class in the module under test. The *note side_effect: 40c4. attribute
on the mock date class is then set to a lambda function that returns a
real date. When the mock date class is called a real date will be
constructed and returned by ‘side_effect’.
>>> from datetime import date
>>> with patch('mymodule.date') as mock_date:
... mock_date.today.return_value = date(2010, 10, 8)
... mock_date.side_effect = lambda *args, **kw: date(*args, **kw)
...
... assert mymodule.date.today() == date(2010, 10, 8)
... assert mymodule.date(2009, 6, 8) == date(2009, 6, 8)
Note that we don’t patch *note datetime.date: 1cd. globally, we patch
‘date’ in the module that 'uses' it. See *note where to patch: 40c5.
When ‘date.today()’ is called a known date is returned, but calls to the
‘date(...)’ constructor still return normal dates. Without this you can
find yourself having to calculate an expected result using exactly the
same algorithm as the code under test, which is a classic testing
anti-pattern.
Calls to the date constructor are recorded in the ‘mock_date’ attributes
(‘call_count’ and friends) which may also be useful for your tests.
An alternative way of dealing with mocking dates, or other builtin
classes, is discussed in this blog entry(1).
---------- Footnotes ----------
(1)
https://williambert.online/2011/07/how-to-unit-testing-in-django-with-mocking-and-patching/
File: python.info, Node: Mocking a Generator Method, Next: Applying the same patch to every test method, Prev: Partial mocking, Up: Further Examples
5.27.7.18 Mocking a Generator Method
....................................
A Python generator is a function or method that uses the *note yield:
9cd. statement to return a series of values when iterated over (1).
A generator method / function is called to return the generator object.
It is the generator object that is then iterated over. The protocol
method for iteration is *note __iter__(): 1cc1, so we can mock this
using a *note MagicMock: e8a.
Here’s an example class with an “iter” method implemented as a
generator:
>>> class Foo:
... def iter(self):
... for i in [1, 2, 3]:
... yield i
...
>>> foo = Foo()
>>> list(foo.iter())
[1, 2, 3]
How would we mock this class, and in particular its “iter” method?
To configure the values returned from the iteration (implicit in the
call to *note list: 60d.), we need to configure the object returned by
the call to ‘foo.iter()’.
>>> mock_foo = MagicMock()
>>> mock_foo.iter.return_value = iter([1, 2, 3])
>>> list(mock_foo.iter())
[1, 2, 3]
---------- Footnotes ----------
(1) There are also generator expressions and more advanced uses
(http://www.dabeaz.com/coroutines/index.html) of generators, but we
aren’t concerned about them here. A very good introduction to
generators and how powerful they are is: Generator Tricks for Systems
Programmers (http://www.dabeaz.com/generators/).
File: python.info, Node: Applying the same patch to every test method, Next: Mocking Unbound Methods, Prev: Mocking a Generator Method, Up: Further Examples
5.27.7.19 Applying the same patch to every test method
......................................................
If you want several patches in place for multiple test methods the
obvious way is to apply the patch decorators to every method. This can
feel like unnecessary repetition. Instead, you can use *note patch():
e8b. (in all its various forms) as a class decorator. This applies the
patches to all test methods on the class. A test method is identified
by methods whose names start with ‘test’:
>>> @patch('mymodule.SomeClass')
... class MyTest(unittest.TestCase):
...
... def test_one(self, MockSomeClass):
... self.assertIs(mymodule.SomeClass, MockSomeClass)
...
... def test_two(self, MockSomeClass):
... self.assertIs(mymodule.SomeClass, MockSomeClass)
...
... def not_a_test(self):
... return 'something'
...
>>> MyTest('test_one').test_one()
>>> MyTest('test_two').test_two()
>>> MyTest('test_two').not_a_test()
'something'
An alternative way of managing patches is to use the *note patch
methods; start and stop: 40f8. These allow you to move the patching
into your ‘setUp’ and ‘tearDown’ methods.
>>> class MyTest(unittest.TestCase):
... def setUp(self):
... self.patcher = patch('mymodule.foo')
... self.mock_foo = self.patcher.start()
...
... def test_foo(self):
... self.assertIs(mymodule.foo, self.mock_foo)
...
... def tearDown(self):
... self.patcher.stop()
...
>>> MyTest('test_foo').run()
If you use this technique you must ensure that the patching is “undone”
by calling ‘stop’. This can be fiddlier than you might think, because
if an exception is raised in the setUp then tearDown is not called.
*note unittest.TestCase.addCleanup(): a81. makes this easier:
>>> class MyTest(unittest.TestCase):
... def setUp(self):
... patcher = patch('mymodule.foo')
... self.addCleanup(patcher.stop)
... self.mock_foo = patcher.start()
...
... def test_foo(self):
... self.assertIs(mymodule.foo, self.mock_foo)
...
>>> MyTest('test_foo').run()
File: python.info, Node: Mocking Unbound Methods, Next: Checking multiple calls with mock, Prev: Applying the same patch to every test method, Up: Further Examples
5.27.7.20 Mocking Unbound Methods
.................................
Whilst writing tests today I needed to patch an 'unbound method'
(patching the method on the class rather than on the instance). I
needed self to be passed in as the first argument because I want to make
asserts about which objects were calling this particular method. The
issue is that you can’t patch with a mock for this, because if you
replace an unbound method with a mock it doesn’t become a bound method
when fetched from the instance, and so it doesn’t get self passed in.
The workaround is to patch the unbound method with a real function
instead. The *note patch(): e8b. decorator makes it so simple to patch
out methods with a mock that having to create a real function becomes a
nuisance.
If you pass ‘autospec=True’ to patch then it does the patching with a
'real' function object. This function object has the same signature as
the one it is replacing, but delegates to a mock under the hood. You
still get your mock auto-created in exactly the same way as before.
What it means though, is that if you use it to patch out an unbound
method on a class the mocked function will be turned into a bound method
if it is fetched from an instance. It will have ‘self’ passed in as the
first argument, which is exactly what I wanted:
>>> class Foo:
... def foo(self):
... pass
...
>>> with patch.object(Foo, 'foo', autospec=True) as mock_foo:
... mock_foo.return_value = 'foo'
... foo = Foo()
... foo.foo()
...
'foo'
>>> mock_foo.assert_called_once_with(foo)
If we don’t use ‘autospec=True’ then the unbound method is patched out
with a Mock instance instead, and isn’t called with ‘self’.
File: python.info, Node: Checking multiple calls with mock, Next: Coping with mutable arguments, Prev: Mocking Unbound Methods, Up: Further Examples
5.27.7.21 Checking multiple calls with mock
...........................................
mock has a nice API for making assertions about how your mock objects
are used.
>>> mock = Mock()
>>> mock.foo_bar.return_value = None
>>> mock.foo_bar('baz', spam='eggs')
>>> mock.foo_bar.assert_called_with('baz', spam='eggs')
If your mock is only being called once you can use the *note
assert_called_once_with(): 40ce. method that also asserts that the *note
call_count: 40d7. is one.
>>> mock.foo_bar.assert_called_once_with('baz', spam='eggs')
>>> mock.foo_bar()
>>> mock.foo_bar.assert_called_once_with('baz', spam='eggs')
Traceback (most recent call last):
...
AssertionError: Expected 'foo_bar' to be called once. Called 2 times.
Calls: [call('baz', spam='eggs'), call()].
Both ‘assert_called_with’ and ‘assert_called_once_with’ make assertions
about the 'most recent' call. If your mock is going to be called
several times, and you want to make assertions about 'all' those calls
you can use *note call_args_list: 40d8.:
>>> mock = Mock(return_value=None)
>>> mock(1, 2, 3)
>>> mock(4, 5, 6)
>>> mock()
>>> mock.call_args_list
[call(1, 2, 3), call(4, 5, 6), call()]
The *note call: 19cd. helper makes it easy to make assertions about
these calls. You can build up a list of expected calls and compare it
to ‘call_args_list’. This looks remarkably similar to the repr of the
‘call_args_list’:
>>> expected = [call(1, 2, 3), call(4, 5, 6), call()]
>>> mock.call_args_list == expected
True
File: python.info, Node: Coping with mutable arguments, Next: Nesting Patches, Prev: Checking multiple calls with mock, Up: Further Examples
5.27.7.22 Coping with mutable arguments
.......................................
Another situation is rare, but can bite you, is when your mock is called
with mutable arguments. ‘call_args’ and ‘call_args_list’ store
'references' to the arguments. If the arguments are mutated by the code
under test then you can no longer make assertions about what the values
were when the mock was called.
Here’s some example code that shows the problem. Imagine the following
functions defined in ‘mymodule’:
def frob(val):
pass
def grob(val):
"First frob and then clear val"
frob(val)
val.clear()
When we try to test that ‘grob’ calls ‘frob’ with the correct argument
look what happens:
>>> with patch('mymodule.frob') as mock_frob:
... val = {6}
... mymodule.grob(val)
...
>>> val
set()
>>> mock_frob.assert_called_with({6})
Traceback (most recent call last):
...
AssertionError: Expected: (({6},), {})
Called with: ((set(),), {})
One possibility would be for mock to copy the arguments you pass in.
This could then cause problems if you do assertions that rely on object
identity for equality.
Here’s one solution that uses the *note side_effect: 40c4.
functionality. If you provide a ‘side_effect’ function for a mock then
‘side_effect’ will be called with the same args as the mock. This gives
us an opportunity to copy the arguments and store them for later
assertions. In this example I’m using 'another' mock to store the
arguments so that I can use the mock methods for doing the assertion.
Again a helper function sets this up for me.
>>> from copy import deepcopy
>>> from unittest.mock import Mock, patch, DEFAULT
>>> def copy_call_args(mock):
... new_mock = Mock()
... def side_effect(*args, **kwargs):
... args = deepcopy(args)
... kwargs = deepcopy(kwargs)
... new_mock(*args, **kwargs)
... return DEFAULT
... mock.side_effect = side_effect
... return new_mock
...
>>> with patch('mymodule.frob') as mock_frob:
... new_mock = copy_call_args(mock_frob)
... val = {6}
... mymodule.grob(val)
...
>>> new_mock.assert_called_with({6})
>>> new_mock.call_args
call({6})
‘copy_call_args’ is called with the mock that will be called. It
returns a new mock that we do the assertion on. The ‘side_effect’
function makes a copy of the args and calls our ‘new_mock’ with the
copy.
Note: If your mock is only going to be used once there is an easier
way of checking arguments at the point they are called. You can
simply do the checking inside a ‘side_effect’ function.
>>> def side_effect(arg):
... assert arg == {6}
...
>>> mock = Mock(side_effect=side_effect)
>>> mock({6})
>>> mock(set())
Traceback (most recent call last):
...
AssertionError
An alternative approach is to create a subclass of *note Mock: a4b. or
*note MagicMock: e8a. that copies (using *note copy.deepcopy(): b6e.)
the arguments. Here’s an example implementation:
>>> from copy import deepcopy
>>> class CopyingMock(MagicMock):
... def __call__(self, /, *args, **kwargs):
... args = deepcopy(args)
... kwargs = deepcopy(kwargs)
... return super().__call__(*args, **kwargs)
...
>>> c = CopyingMock(return_value=None)
>>> arg = set()
>>> c(arg)
>>> arg.add(1)
>>> c.assert_called_with(set())
>>> c.assert_called_with(arg)
Traceback (most recent call last):
...
AssertionError: expected call not found.
Expected: mock({1})
Actual: mock(set())
>>> c.foo
When you subclass ‘Mock’ or ‘MagicMock’ all dynamically created
attributes, and the ‘return_value’ will use your subclass automatically.
That means all children of a ‘CopyingMock’ will also have the type
‘CopyingMock’.
File: python.info, Node: Nesting Patches, Next: Mocking a dictionary with MagicMock, Prev: Coping with mutable arguments, Up: Further Examples
5.27.7.23 Nesting Patches
.........................
Using patch as a context manager is nice, but if you do multiple patches
you can end up with nested with statements indenting further and further
to the right:
>>> class MyTest(unittest.TestCase):
...
... def test_foo(self):
... with patch('mymodule.Foo') as mock_foo:
... with patch('mymodule.Bar') as mock_bar:
... with patch('mymodule.Spam') as mock_spam:
... assert mymodule.Foo is mock_foo
... assert mymodule.Bar is mock_bar
... assert mymodule.Spam is mock_spam
...
>>> original = mymodule.Foo
>>> MyTest('test_foo').test_foo()
>>> assert mymodule.Foo is original
With unittest ‘cleanup’ functions and the *note patch methods; start and
stop: 40f8. we can achieve the same effect without the nested
indentation. A simple helper method, ‘create_patch’, puts the patch in
place and returns the created mock for us:
>>> class MyTest(unittest.TestCase):
...
... def create_patch(self, name):
... patcher = patch(name)
... thing = patcher.start()
... self.addCleanup(patcher.stop)
... return thing
...
... def test_foo(self):
... mock_foo = self.create_patch('mymodule.Foo')
... mock_bar = self.create_patch('mymodule.Bar')
... mock_spam = self.create_patch('mymodule.Spam')
...
... assert mymodule.Foo is mock_foo
... assert mymodule.Bar is mock_bar
... assert mymodule.Spam is mock_spam
...
>>> original = mymodule.Foo
>>> MyTest('test_foo').run()
>>> assert mymodule.Foo is original
File: python.info, Node: Mocking a dictionary with MagicMock, Next: Mock subclasses and their attributes, Prev: Nesting Patches, Up: Further Examples
5.27.7.24 Mocking a dictionary with MagicMock
.............................................
You may want to mock a dictionary, or other container object, recording
all access to it whilst having it still behave like a dictionary.
We can do this with *note MagicMock: e8a, which will behave like a
dictionary, and using *note side_effect: 40c4. to delegate dictionary
access to a real underlying dictionary that is under our control.
When the *note __getitem__(): 285. and *note __setitem__(): 12be.
methods of our ‘MagicMock’ are called (normal dictionary access) then
‘side_effect’ is called with the key (and in the case of ‘__setitem__’
the value too). We can also control what is returned.
After the ‘MagicMock’ has been used we can use attributes like *note
call_args_list: 40d8. to assert about how the dictionary was used:
>>> my_dict = {'a': 1, 'b': 2, 'c': 3}
>>> def getitem(name):
... return my_dict[name]
...
>>> def setitem(name, val):
... my_dict[name] = val
...
>>> mock = MagicMock()
>>> mock.__getitem__.side_effect = getitem
>>> mock.__setitem__.side_effect = setitem
Note: An alternative to using ‘MagicMock’ is to use ‘Mock’ and
'only' provide the magic methods you specifically want:
>>> mock = Mock()
>>> mock.__getitem__ = Mock(side_effect=getitem)
>>> mock.__setitem__ = Mock(side_effect=setitem)
A 'third' option is to use ‘MagicMock’ but passing in ‘dict’ as the
'spec' (or 'spec_set') argument so that the ‘MagicMock’ created
only has dictionary magic methods available:
>>> mock = MagicMock(spec_set=dict)
>>> mock.__getitem__.side_effect = getitem
>>> mock.__setitem__.side_effect = setitem
With these side effect functions in place, the ‘mock’ will behave like a
normal dictionary but recording the access. It even raises a *note
KeyError: 33f. if you try to access a key that doesn’t exist.
>>> mock['a']
1
>>> mock['c']
3
>>> mock['d']
Traceback (most recent call last):
...
KeyError: 'd'
>>> mock['b'] = 'fish'
>>> mock['d'] = 'eggs'
>>> mock['b']
'fish'
>>> mock['d']
'eggs'
After it has been used you can make assertions about the access using
the normal mock methods and attributes:
>>> mock.__getitem__.call_args_list
[call('a'), call('c'), call('d'), call('b'), call('d')]
>>> mock.__setitem__.call_args_list
[call('b', 'fish'), call('d', 'eggs')]
>>> my_dict
{'a': 1, 'b': 'fish', 'c': 3, 'd': 'eggs'}
File: python.info, Node: Mock subclasses and their attributes, Next: Mocking imports with patch dict, Prev: Mocking a dictionary with MagicMock, Up: Further Examples
5.27.7.25 Mock subclasses and their attributes
..............................................
There are various reasons why you might want to subclass *note Mock:
a4b. One reason might be to add helper methods. Here’s a silly
example:
>>> class MyMock(MagicMock):
... def has_been_called(self):
... return self.called
...
>>> mymock = MyMock(return_value=None)
>>> mymock
>>> mymock.has_been_called()
False
>>> mymock()
>>> mymock.has_been_called()
True
The standard behaviour for ‘Mock’ instances is that attributes and the
return value mocks are of the same type as the mock they are accessed
on. This ensures that ‘Mock’ attributes are ‘Mocks’ and ‘MagicMock’
attributes are ‘MagicMocks’ (1). So if you’re subclassing to add helper
methods then they’ll also be available on the attributes and return
value mock of instances of your subclass.
>>> mymock.foo
>>> mymock.foo.has_been_called()
False
>>> mymock.foo()
>>> mymock.foo.has_been_called()
True
Sometimes this is inconvenient. For example, one user(2) is subclassing
mock to created a Twisted adaptor(3). Having this applied to attributes
too actually causes errors.
‘Mock’ (in all its flavours) uses a method called ‘_get_child_mock’ to
create these “sub-mocks” for attributes and return values. You can
prevent your subclass being used for attributes by overriding this
method. The signature is that it takes arbitrary keyword arguments
(‘**kwargs’) which are then passed onto the mock constructor:
>>> class Subclass(MagicMock):
... def _get_child_mock(self, /, **kwargs):
... return MagicMock(**kwargs)
...
>>> mymock = Subclass()
>>> mymock.foo
>>> assert isinstance(mymock, Subclass)
>>> assert not isinstance(mymock.foo, Subclass)
>>> assert not isinstance(mymock(), Subclass)
---------- Footnotes ----------
(1) An exception to this rule are the non-callable mocks. Attributes
use the callable variant because otherwise non-callable mocks couldn’t
have callable methods.
(2) https://code.google.com/archive/p/mock/issues/105
(3)
https://twisted.org/documents/11.0.0/api/twisted.python.components.html
File: python.info, Node: Mocking imports with patch dict, Next: Tracking order of calls and less verbose call assertions, Prev: Mock subclasses and their attributes, Up: Further Examples
5.27.7.26 Mocking imports with patch.dict
.........................................
One situation where mocking can be hard is where you have a local import
inside a function. These are harder to mock because they aren’t using
an object from the module namespace that we can patch out.
Generally local imports are to be avoided. They are sometimes done to
prevent circular dependencies, for which there is 'usually' a much
better way to solve the problem (refactor the code) or to prevent “up
front costs” by delaying the import. This can also be solved in better
ways than an unconditional local import (store the module as a class or
module attribute and only do the import on first use).
That aside there is a way to use ‘mock’ to affect the results of an
import. Importing fetches an 'object' from the *note sys.modules: 1550.
dictionary. Note that it fetches an 'object', which need not be a
module. Importing a module for the first time results in a module
object being put in ‘sys.modules’, so usually when you import something
you get a module back. This need not be the case however.
This means you can use *note patch.dict(): 19f2. to 'temporarily' put a
mock in place in *note sys.modules: 1550. Any imports whilst this patch
is active will fetch the mock. When the patch is complete (the
decorated function exits, the with statement body is complete or
‘patcher.stop()’ is called) then whatever was there previously will be
restored safely.
Here’s an example that mocks out the ‘fooble’ module.
>>> import sys
>>> mock = Mock()
>>> with patch.dict('sys.modules', {'fooble': mock}):
... import fooble
... fooble.blob()
...
>>> assert 'fooble' not in sys.modules
>>> mock.blob.assert_called_once_with()
As you can see the ‘import fooble’ succeeds, but on exit there is no
‘fooble’ left in *note sys.modules: 1550.
This also works for the ‘from module import name’ form:
>>> mock = Mock()
>>> with patch.dict('sys.modules', {'fooble': mock}):
... from fooble import blob
... blob.blip()
...
>>> mock.blob.blip.assert_called_once_with()
With slightly more work you can also mock package imports:
>>> mock = Mock()
>>> modules = {'package': mock, 'package.module': mock.module}
>>> with patch.dict('sys.modules', modules):
... from package.module import fooble
... fooble()
...
>>> mock.module.fooble.assert_called_once_with()
File: python.info, Node: Tracking order of calls and less verbose call assertions, Next: More complex argument matching, Prev: Mocking imports with patch dict, Up: Further Examples
5.27.7.27 Tracking order of calls and less verbose call assertions
..................................................................
The *note Mock: a4b. class allows you to track the 'order' of method
calls on your mock objects through the *note method_calls: 40d2.
attribute. This doesn’t allow you to track the order of calls between
separate mock objects, however we can use *note mock_calls: 40d0. to
achieve the same effect.
Because mocks track calls to child mocks in ‘mock_calls’, and accessing
an arbitrary attribute of a mock creates a child mock, we can create our
separate mocks from a parent one. Calls to those child mock will then
all be recorded, in order, in the ‘mock_calls’ of the parent:
>>> manager = Mock()
>>> mock_foo = manager.foo
>>> mock_bar = manager.bar
>>> mock_foo.something()
>>> mock_bar.other.thing()
>>> manager.mock_calls
[call.foo.something(), call.bar.other.thing()]
We can then assert about the calls, including the order, by comparing
with the ‘mock_calls’ attribute on the manager mock:
>>> expected_calls = [call.foo.something(), call.bar.other.thing()]
>>> manager.mock_calls == expected_calls
True
If ‘patch’ is creating, and putting in place, your mocks then you can
attach them to a manager mock using the *note attach_mock(): 160f.
method. After attaching calls will be recorded in ‘mock_calls’ of the
manager.
>>> manager = MagicMock()
>>> with patch('mymodule.Class1') as MockClass1:
... with patch('mymodule.Class2') as MockClass2:
... manager.attach_mock(MockClass1, 'MockClass1')
... manager.attach_mock(MockClass2, 'MockClass2')
... MockClass1().foo()
... MockClass2().bar()
>>> manager.mock_calls
[call.MockClass1(),
call.MockClass1().foo(),
call.MockClass2(),
call.MockClass2().bar()]
If many calls have been made, but you’re only interested in a particular
sequence of them then an alternative is to use the *note
assert_has_calls(): 16a8. method. This takes a list of calls
(constructed with the *note call: 19cd. object). If that sequence of
calls are in *note mock_calls: 40d0. then the assert succeeds.
>>> m = MagicMock()
>>> m().foo().bar().baz()
>>> m.one().two().three()
>>> calls = call.one().two().three().call_list()
>>> m.assert_has_calls(calls)
Even though the chained call ‘m.one().two().three()’ aren’t the only
calls that have been made to the mock, the assert still succeeds.
Sometimes a mock may have several calls made to it, and you are only
interested in asserting about 'some' of those calls. You may not even
care about the order. In this case you can pass ‘any_order=True’ to
‘assert_has_calls’:
>>> m = MagicMock()
>>> m(1), m.two(2, 3), m.seven(7), m.fifty('50')
(...)
>>> calls = [call.fifty('50'), call(1), call.seven(7)]
>>> m.assert_has_calls(calls, any_order=True)
File: python.info, Node: More complex argument matching, Prev: Tracking order of calls and less verbose call assertions, Up: Further Examples
5.27.7.28 More complex argument matching
........................................
Using the same basic concept as *note ANY: 19cb. we can implement
matchers to do more complex assertions on objects used as arguments to
mocks.
Suppose we expect some object to be passed to a mock that by default
compares equal based on object identity (which is the Python default for
user defined classes). To use *note assert_called_with(): 1a20. we
would need to pass in the exact same object. If we are only interested
in some of the attributes of this object then we can create a matcher
that will check these attributes for us.
You can see in this example how a ‘standard’ call to
‘assert_called_with’ isn’t sufficient:
>>> class Foo:
... def __init__(self, a, b):
... self.a, self.b = a, b
...
>>> mock = Mock(return_value=None)
>>> mock(Foo(1, 2))
>>> mock.assert_called_with(Foo(1, 2))
Traceback (most recent call last):
...
AssertionError: expected call not found.
Expected: mock(<__main__.Foo object at 0x...>)
Actual: mock(<__main__.Foo object at 0x...>)
A comparison function for our ‘Foo’ class might look something like
this:
>>> def compare(self, other):
... if not type(self) == type(other):
... return False
... if self.a != other.a:
... return False
... if self.b != other.b:
... return False
... return True
...
And a matcher object that can use comparison functions like this for its
equality operation would look something like this:
>>> class Matcher:
... def __init__(self, compare, some_obj):
... self.compare = compare
... self.some_obj = some_obj
... def __eq__(self, other):
... return self.compare(self.some_obj, other)
...
Putting all this together:
>>> match_foo = Matcher(compare, Foo(1, 2))
>>> mock.assert_called_with(match_foo)
The ‘Matcher’ is instantiated with our compare function and the ‘Foo’
object we want to compare against. In ‘assert_called_with’ the
‘Matcher’ equality method will be called, which compares the object the
mock was called with against the one we created our matcher with. If
they match then ‘assert_called_with’ passes, and if they don’t an *note
AssertionError: 6a5. is raised:
>>> match_wrong = Matcher(compare, Foo(3, 4))
>>> mock.assert_called_with(match_wrong)
Traceback (most recent call last):
...
AssertionError: Expected: ((,), {})
Called with: ((,), {})
With a bit of tweaking you could have the comparison function raise the
*note AssertionError: 6a5. directly and provide a more useful failure
message.
As of version 1.5, the Python testing library PyHamcrest(1) provides
similar functionality, that may be useful here, in the form of its
equality matcher (hamcrest.library.integration.match_equality(2)).
---------- Footnotes ----------
(1) https://pyhamcrest.readthedocs.io/
(2)
https://pyhamcrest.readthedocs.io/en/release-1.8/integration/#module-hamcrest.library.integration.match_equality
File: python.info, Node: test — Regression tests package for Python, Next: test support — Utilities for the Python test suite, Prev: unittest mock — getting started, Up: Development Tools
5.27.8 ‘test’ — Regression tests package for Python
---------------------------------------------------
Note: The *note test: e2. package is meant for internal use by
Python only. It is documented for the benefit of the core
developers of Python. Any use of this package outside of Python’s
standard library is discouraged as code mentioned here can change
or be removed without notice between releases of Python.
__________________________________________________________________
The *note test: e2. package contains all regression tests for Python as
well as the modules *note test.support: e4. and *note test.regrtest: e3.
*note test.support: e4. is used to enhance your tests while *note
test.regrtest: e3. drives the testing suite.
Each module in the *note test: e2. package whose name starts with
‘test_’ is a testing suite for a specific module or feature. All new
tests should be written using the *note unittest: 106. or *note doctest:
3a. module. Some older tests are written using a “traditional” testing
style that compares output printed to ‘sys.stdout’; this style of test
is considered deprecated.
See also
........
Module *note unittest: 106.
Writing PyUnit regression tests.
Module *note doctest: 3a.
Tests embedded in documentation strings.
* Menu:
* Writing Unit Tests for the test package::
* Running tests using the command-line interface::
File: python.info, Node: Writing Unit Tests for the test package, Next: Running tests using the command-line interface, Up: test — Regression tests package for Python
5.27.8.1 Writing Unit Tests for the ‘test’ package
..................................................
It is preferred that tests that use the *note unittest: 106. module
follow a few guidelines. One is to name the test module by starting it
with ‘test_’ and end it with the name of the module being tested. The
test methods in the test module should start with ‘test_’ and end with a
description of what the method is testing. This is needed so that the
methods are recognized by the test driver as test methods. Also, no
documentation string for the method should be included. A comment (such
as ‘# Tests function returns only True or False’) should be used to
provide documentation for test methods. This is done because
documentation strings get printed out if they exist and thus what test
is being run is not stated.
A basic boilerplate is often used:
import unittest
from test import support
class MyTestCase1(unittest.TestCase):
# Only use setUp() and tearDown() if necessary
def setUp(self):
... code to execute in preparation for tests ...
def tearDown(self):
... code to execute to clean up after tests ...
def test_feature_one(self):
# Test feature one.
... testing code ...
def test_feature_two(self):
# Test feature two.
... testing code ...
... more test methods ...
class MyTestCase2(unittest.TestCase):
... same structure as MyTestCase1 ...
... more test classes ...
if __name__ == '__main__':
unittest.main()
This code pattern allows the testing suite to be run by *note
test.regrtest: e3, on its own as a script that supports the *note
unittest: 106. CLI, or via the ‘python -m unittest’ CLI.
The goal for regression testing is to try to break code. This leads to
a few guidelines to be followed:
* The testing suite should exercise all classes, functions, and
constants. This includes not just the external API that is to be
presented to the outside world but also “private” code.
* Whitebox testing (examining the code being tested when the tests
are being written) is preferred. Blackbox testing (testing only
the published user interface) is not complete enough to make sure
all boundary and edge cases are tested.
* Make sure all possible values are tested including invalid ones.
This makes sure that not only all valid values are acceptable but
also that improper values are handled correctly.
* Exhaust as many code paths as possible. Test where branching
occurs and thus tailor input to make sure as many different paths
through the code are taken.
* Add an explicit test for any bugs discovered for the tested code.
This will make sure that the error does not crop up again if the
code is changed in the future.
* Make sure to clean up after your tests (such as close and remove
all temporary files).
* If a test is dependent on a specific condition of the operating
system then verify the condition already exists before attempting
the test.
* Import as few modules as possible and do it as soon as possible.
This minimizes external dependencies of tests and also minimizes
possible anomalous behavior from side-effects of importing a
module.
* Try to maximize code reuse. On occasion, tests will vary by
something as small as what type of input is used. Minimize code
duplication by subclassing a basic test class with a class that
specifies the input:
class TestFuncAcceptsSequencesMixin:
func = mySuperWhammyFunction
def test_func(self):
self.func(self.arg)
class AcceptLists(TestFuncAcceptsSequencesMixin, unittest.TestCase):
arg = [1, 2, 3]
class AcceptStrings(TestFuncAcceptsSequencesMixin, unittest.TestCase):
arg = 'abc'
class AcceptTuples(TestFuncAcceptsSequencesMixin, unittest.TestCase):
arg = (1, 2, 3)
When using this pattern, remember that all classes that inherit
from *note unittest.TestCase: 449. are run as tests. The
‘TestFuncAcceptsSequencesMixin’ class in the example above does not
have any data and so can’t be run by itself, thus it does not
inherit from *note unittest.TestCase: 449.
See also
........
Test Driven Development
A book by Kent Beck on writing tests before code.
File: python.info, Node: Running tests using the command-line interface, Prev: Writing Unit Tests for the test package, Up: test — Regression tests package for Python
5.27.8.2 Running tests using the command-line interface
.......................................................
The *note test: e2. package can be run as a script to drive Python’s
regression test suite, thanks to the *note -m: 5dd. option: ‘python -m
test’. Under the hood, it uses *note test.regrtest: e3.; the call
‘python -m test.regrtest’ used in previous Python versions still works.
Running the script by itself automatically starts running all regression
tests in the *note test: e2. package. It does this by finding all
modules in the package whose name starts with ‘test_’, importing them,
and executing the function ‘test_main()’ if present or loading the tests
via unittest.TestLoader.loadTestsFromModule if ‘test_main’ does not
exist. The names of tests to execute may also be passed to the script.
Specifying a single regression test (‘python -m test test_spam’) will
minimize output and only print whether the test passed or failed.
Running *note test: e2. directly allows what resources are available for
tests to use to be set. You do this by using the ‘-u’ command-line
option. Specifying ‘all’ as the value for the ‘-u’ option enables all
possible resources: ‘python -m test -uall’. If all but one resource is
desired (a more common case), a comma-separated list of resources that
are not desired may be listed after ‘all’. The command ‘python -m test
-uall,-audio,-largefile’ will run *note test: e2. with all resources
except the ‘audio’ and ‘largefile’ resources. For a list of all
resources and more command-line options, run ‘python -m test -h’.
Some other ways to execute the regression tests depend on what platform
the tests are being executed on. On Unix, you can run ‘make test’ at
the top-level directory where Python was built. On Windows, executing
‘rt.bat’ from your ‘PCbuild’ directory will run all regression tests.
File: python.info, Node: test support — Utilities for the Python test suite, Next: test support socket_helper — Utilities for socket tests, Prev: test — Regression tests package for Python, Up: Development Tools
5.27.9 ‘test.support’ — Utilities for the Python test suite
-----------------------------------------------------------
The *note test.support: e4. module provides support for Python’s
regression test suite.
Note: *note test.support: e4. is not a public module. It is
documented here to help Python developers write tests. The API of
this module is subject to change without backwards compatibility
concerns between releases.
This module defines the following exceptions:
-- Exception: test.support.TestFailed
Exception to be raised when a test fails. This is deprecated in
favor of *note unittest: 106.-based tests and *note
unittest.TestCase: 449.’s assertion methods.
-- Exception: test.support.ResourceDenied
Subclass of *note unittest.SkipTest: fdf. Raised when a resource
(such as a network connection) is not available. Raised by the
*note requires(): 4137. function.
The *note test.support: e4. module defines the following constants:
-- Data: test.support.verbose
‘True’ when verbose output is enabled. Should be checked when more
detailed information is desired about a running test. 'verbose' is
set by *note test.regrtest: e3.
-- Data: test.support.is_jython
‘True’ if the running interpreter is Jython.
-- Data: test.support.is_android
‘True’ if the system is Android.
-- Data: test.support.unix_shell
Path for shell if not on Windows; otherwise ‘None’.
-- Data: test.support.LOOPBACK_TIMEOUT
Timeout in seconds for tests using a network server listening on
the network local loopback interface like ‘127.0.0.1’.
The timeout is long enough to prevent test failure: it takes into
account that the client and the server can run in different threads
or even different processes.
The timeout should be long enough for *note connect(): da2, *note
recv(): da3. and *note send(): da6. methods of *note socket.socket:
da0.
Its default value is 5 seconds.
See also *note INTERNET_TIMEOUT: 19b4.
-- Data: test.support.INTERNET_TIMEOUT
Timeout in seconds for network requests going to the internet.
The timeout is short enough to prevent a test to wait for too long
if the internet request is blocked for whatever reason.
Usually, a timeout using *note INTERNET_TIMEOUT: 19b4. should not
mark a test as failed, but skip the test instead: see *note
transient_internet(): 413c.
Its default value is 1 minute.
See also *note LOOPBACK_TIMEOUT: 19b3.
-- Data: test.support.SHORT_TIMEOUT
Timeout in seconds to mark a test as failed if the test takes “too
long”.
The timeout value depends on the regrtest ‘--timeout’ command line
option.
If a test using *note SHORT_TIMEOUT: 18ae. starts to fail randomly
on slow buildbots, use *note LONG_TIMEOUT: 19b5. instead.
Its default value is 30 seconds.
-- Data: test.support.LONG_TIMEOUT
Timeout in seconds to detect when a test hangs.
It is long enough to reduce the risk of test failure on the slowest
Python buildbots. It should not be used to mark a test as failed
if the test takes “too long”. The timeout value depends on the
regrtest ‘--timeout’ command line option.
Its default value is 5 minutes.
See also *note LOOPBACK_TIMEOUT: 19b3, *note INTERNET_TIMEOUT:
19b4. and *note SHORT_TIMEOUT: 18ae.
-- Data: test.support.PGO
Set when tests can be skipped when they are not useful for PGO.
-- Data: test.support.PIPE_MAX_SIZE
A constant that is likely larger than the underlying OS pipe buffer
size, to make writes blocking.
-- Data: test.support.Py_DEBUG
‘True’ if Python was built with the *note Py_DEBUG: 4140. macro
defined, that is, if Python was *note built in debug mode: 1fb.
Added in version 3.12.
-- Data: test.support.SOCK_MAX_SIZE
A constant that is likely larger than the underlying OS socket
buffer size, to make writes blocking.
-- Data: test.support.TEST_SUPPORT_DIR
Set to the top level directory that contains *note test.support:
e4.
-- Data: test.support.TEST_HOME_DIR
Set to the top level directory for the test package.
-- Data: test.support.TEST_DATA_DIR
Set to the ‘data’ directory within the test package.
-- Data: test.support.MAX_Py_ssize_t
Set to *note sys.maxsize: 11b7. for big memory tests.
-- Data: test.support.max_memuse
Set by *note set_memlimit(): 4147. as the memory limit for big
memory tests. Limited by *note MAX_Py_ssize_t: 4145.
-- Data: test.support.real_max_memuse
Set by *note set_memlimit(): 4147. as the memory limit for big
memory tests. Not limited by *note MAX_Py_ssize_t: 4145.
-- Data: test.support.MISSING_C_DOCSTRINGS
Set to ‘True’ if Python is built without docstrings (the
‘WITH_DOC_STRINGS’ macro is not defined). See the *note configure
-without-doc-strings: 1db1. option.
See also the *note HAVE_DOCSTRINGS: 414a. variable.
-- Data: test.support.HAVE_DOCSTRINGS
Set to ‘True’ if function docstrings are available. See the *note
python -OO: db4. option, which strips docstrings of functions
implemented in Python.
See also the *note MISSING_C_DOCSTRINGS: 4149. variable.
-- Data: test.support.TEST_HTTP_URL
Define the URL of a dedicated HTTP server for the network tests.
-- Data: test.support.ALWAYS_EQ
Object that is equal to anything. Used to test mixed type
comparison.
-- Data: test.support.NEVER_EQ
Object that is not equal to anything (even to *note ALWAYS_EQ:
414c.). Used to test mixed type comparison.
-- Data: test.support.LARGEST
Object that is greater than anything (except itself). Used to test
mixed type comparison.
-- Data: test.support.SMALLEST
Object that is less than anything (except itself). Used to test
mixed type comparison.
The *note test.support: e4. module defines the following functions:
-- Function: test.support.busy_retry (timeout, err_msg=None, /, *,
error=True)
Run the loop body until ‘break’ stops the loop.
After 'timeout' seconds, raise an *note AssertionError: 6a5. if
'error' is true, or just stop the loop if 'error' is false.
Example:
for _ in support.busy_retry(support.SHORT_TIMEOUT):
if check():
break
Example of error=False usage:
for _ in support.busy_retry(support.SHORT_TIMEOUT, error=False):
if check():
break
else:
raise RuntimeError('my custom error')
-- Function: test.support.sleeping_retry (timeout, err_msg=None, /, *,
init_delay=0.010, max_delay=1.0, error=True)
Wait strategy that applies exponential backoff.
Run the loop body until ‘break’ stops the loop. Sleep at each loop
iteration, but not at the first iteration. The sleep delay is
doubled at each iteration (up to 'max_delay' seconds).
See *note busy_retry(): 4150. documentation for the parameters
usage.
Example raising an exception after SHORT_TIMEOUT seconds:
for _ in support.sleeping_retry(support.SHORT_TIMEOUT):
if check():
break
Example of error=False usage:
for _ in support.sleeping_retry(support.SHORT_TIMEOUT, error=False):
if check():
break
else:
raise RuntimeError('my custom error')
-- Function: test.support.is_resource_enabled (resource)
Return ‘True’ if 'resource' is enabled and available. The list of
available resources is only set when *note test.regrtest: e3. is
executing the tests.
-- Function: test.support.python_is_optimized ()
Return ‘True’ if Python was not built with ‘-O0’ or ‘-Og’.
-- Function: test.support.with_pymalloc ()
Return ‘_testcapi.WITH_PYMALLOC’.
-- Function: test.support.requires (resource, msg=None)
Raise *note ResourceDenied: 4136. if 'resource' is not available.
'msg' is the argument to *note ResourceDenied: 4136. if it is
raised. Always returns ‘True’ if called by a function whose
‘__name__’ is ‘'__main__'’. Used when tests are executed by *note
test.regrtest: e3.
-- Function: test.support.sortdict (dict)
Return a repr of 'dict' with keys sorted.
-- Function: test.support.findfile (filename, subdir=None)
Return the path to the file named 'filename'. If no match is found
'filename' is returned. This does not equal a failure since it
could be the path to the file.
Setting 'subdir' indicates a relative path to use to find the file
rather than looking directly in the path directories.
-- Function: test.support.get_pagesize ()
Get size of a page in bytes.
Added in version 3.12.
-- Function: test.support.setswitchinterval (interval)
Set the *note sys.setswitchinterval(): 94a. to the given
'interval'. Defines a minimum interval for Android systems to
prevent the system from hanging.
-- Function: test.support.check_impl_detail (**guards)
Use this check to guard CPython’s implementation-specific tests or
to run them only on the implementations guarded by the arguments.
This function returns ‘True’ or ‘False’ depending on the host
platform. Example usage:
check_impl_detail() # Only on CPython (default).
check_impl_detail(jython=True) # Only on Jython.
check_impl_detail(cpython=False) # Everywhere except CPython.
-- Function: test.support.set_memlimit (limit)
Set the values for *note max_memuse: 4146. and *note
real_max_memuse: 4148. for big memory tests.
-- Function: test.support.record_original_stdout (stdout)
Store the value from 'stdout'. It is meant to hold the stdout at
the time the regrtest began.
-- Function: test.support.get_original_stdout ()
Return the original stdout set by *note record_original_stdout():
415a. or ‘sys.stdout’ if it’s not set.
-- Function: test.support.args_from_interpreter_flags ()
Return a list of command line arguments reproducing the current
settings in ‘sys.flags’ and ‘sys.warnoptions’.
-- Function: test.support.optim_args_from_interpreter_flags ()
Return a list of command line arguments reproducing the current
optimization settings in ‘sys.flags’.
-- Function: test.support.captured_stdin ()
-- Function: test.support.captured_stdout ()
-- Function: test.support.captured_stderr ()
A context managers that temporarily replaces the named stream with
*note io.StringIO: f22. object.
Example use with output streams:
with captured_stdout() as stdout, captured_stderr() as stderr:
print("hello")
print("error", file=sys.stderr)
assert stdout.getvalue() == "hello\n"
assert stderr.getvalue() == "error\n"
Example use with input stream:
with captured_stdin() as stdin:
stdin.write('hello\n')
stdin.seek(0)
# call test code that consumes from sys.stdin
captured = input()
self.assertEqual(captured, "hello")
-- Function: test.support.disable_faulthandler ()
A context manager that temporary disables *note faulthandler: 58.
-- Function: test.support.gc_collect ()
Force as many objects as possible to be collected. This is needed
because timely deallocation is not guaranteed by the garbage
collector. This means that ‘__del__’ methods may be called later
than expected and weakrefs may remain alive for longer than
expected.
-- Function: test.support.disable_gc ()
A context manager that disables the garbage collector on entry. On
exit, the garbage collector is restored to its prior state.
-- Function: test.support.swap_attr (obj, attr, new_val)
Context manager to swap out an attribute with a new object.
Usage:
with swap_attr(obj, "attr", 5):
...
This will set ‘obj.attr’ to 5 for the duration of the ‘with’ block,
restoring the old value at the end of the block. If ‘attr’ doesn’t
exist on ‘obj’, it will be created and then deleted at the end of
the block.
The old value (or ‘None’ if it doesn’t exist) will be assigned to
the target of the “as” clause, if there is one.
-- Function: test.support.swap_item (obj, attr, new_val)
Context manager to swap out an item with a new object.
Usage:
with swap_item(obj, "item", 5):
...
This will set ‘obj["item"]’ to 5 for the duration of the ‘with’
block, restoring the old value at the end of the block. If ‘item’
doesn’t exist on ‘obj’, it will be created and then deleted at the
end of the block.
The old value (or ‘None’ if it doesn’t exist) will be assigned to
the target of the “as” clause, if there is one.
-- Function: test.support.flush_std_streams ()
Call the ‘flush()’ method on *note sys.stdout: ad6. and then on
*note sys.stderr: 939. It can be used to make sure that the logs
order is consistent before writing into stderr.
Added in version 3.11.
-- Function: test.support.print_warning (msg)
Print a warning into *note sys.__stderr__: 4167. Format the
message as: ‘f"Warning -- {msg}"’. If 'msg' is made of multiple
lines, add ‘"Warning -- "’ prefix to each line.
Added in version 3.9.
-- Function: test.support.wait_process (pid, *, exitcode, timeout=None)
Wait until process 'pid' completes and check that the process exit
code is 'exitcode'.
Raise an *note AssertionError: 6a5. if the process exit code is not
equal to 'exitcode'.
If the process runs longer than 'timeout' seconds (*note
SHORT_TIMEOUT: 18ae. by default), kill the process and raise an
*note AssertionError: 6a5. The timeout feature is not available on
Windows.
Added in version 3.9.
-- Function: test.support.calcobjsize (fmt)
Return the size of the *note PyObject: 334. whose structure members
are defined by 'fmt'. The returned value includes the size of the
Python object header and alignment.
-- Function: test.support.calcvobjsize (fmt)
Return the size of the *note PyVarObject: 416a. whose structure
members are defined by 'fmt'. The returned value includes the size
of the Python object header and alignment.
-- Function: test.support.checksizeof (test, o, size)
For testcase 'test', assert that the ‘sys.getsizeof’ for 'o' plus
the GC header size equals 'size'.
-- Function: @test.support.anticipate_failure (condition)
A decorator to conditionally mark tests with *note
unittest.expectedFailure(): 4083. Any use of this decorator should
have an associated comment identifying the relevant tracker issue.
-- Function: test.support.system_must_validate_cert (f)
A decorator that skips the decorated test on TLS certification
validation failures.
-- Function: @test.support.run_with_locale (catstr, *locales)
A decorator for running a function in a different locale, correctly
resetting it after it has finished. 'catstr' is the locale
category as a string (for example ‘"LC_ALL"’). The 'locales'
passed will be tried sequentially, and the first valid locale will
be used.
-- Function: @test.support.run_with_tz (tz)
A decorator for running a function in a specific timezone,
correctly resetting it after it has finished.
-- Function: @test.support.requires_freebsd_version (*min_version)
Decorator for the minimum version when running test on FreeBSD. If
the FreeBSD version is less than the minimum, the test is skipped.
-- Function: @test.support.requires_linux_version (*min_version)
Decorator for the minimum version when running test on Linux. If
the Linux version is less than the minimum, the test is skipped.
-- Function: @test.support.requires_mac_version (*min_version)
Decorator for the minimum version when running test on macOS. If
the macOS version is less than the minimum, the test is skipped.
-- Function: @test.support.requires_gil_enabled
Decorator for skipping tests on the free-threaded build. If the
*note GIL: 159. is disabled, the test is skipped.
-- Function: @test.support.requires_IEEE_754
Decorator for skipping tests on non-IEEE 754 platforms.
-- Function: @test.support.requires_zlib
Decorator for skipping tests if *note zlib: 133. doesn’t exist.
-- Function: @test.support.requires_gzip
Decorator for skipping tests if *note gzip: 67. doesn’t exist.
-- Function: @test.support.requires_bz2
Decorator for skipping tests if *note bz2: 13. doesn’t exist.
-- Function: @test.support.requires_lzma
Decorator for skipping tests if *note lzma: 8a. doesn’t exist.
-- Function: @test.support.requires_resource (resource)
Decorator for skipping tests if 'resource' is not available.
-- Function: @test.support.requires_docstrings
Decorator for only running the test if *note HAVE_DOCSTRINGS: 414a.
-- Function: @test.support.requires_limited_api
Decorator for only running the test if *note Limited C API: 391. is
available.
-- Function: @test.support.cpython_only
Decorator for tests only applicable to CPython.
-- Function: @test.support.impl_detail (msg=None, **guards)
Decorator for invoking *note check_impl_detail(): 4159. on
'guards'. If that returns ‘False’, then uses 'msg' as the reason
for skipping the test.
-- Function: @test.support.no_tracing
Decorator to temporarily turn off tracing for the duration of the
test.
-- Function: @test.support.refcount_test
Decorator for tests which involve reference counting. The
decorator does not run the test if it is not run by CPython. Any
trace function is unset for the duration of the test to prevent
unexpected refcounts caused by the trace function.
-- Function: @test.support.bigmemtest (size, memuse, dry_run=True)
Decorator for bigmem tests.
'size' is a requested size for the test (in arbitrary,
test-interpreted units.) 'memuse' is the number of bytes per unit
for the test, or a good estimate of it. For example, a test that
needs two byte buffers, of 4 GiB each, could be decorated with
‘@bigmemtest(size=_4G, memuse=2)’.
The 'size' argument is normally passed to the decorated test method
as an extra argument. If 'dry_run' is ‘True’, the value passed to
the test method may be less than the requested value. If 'dry_run'
is ‘False’, it means the test doesn’t support dummy runs when ‘-M’
is not specified.
-- Function: @test.support.bigaddrspacetest
Decorator for tests that fill the address space.
-- Function: test.support.check_syntax_error (testcase, statement,
errtext='', *, lineno=None, offset=None)
Test for syntax errors in 'statement' by attempting to compile
'statement'. 'testcase' is the *note unittest: 106. instance for
the test. 'errtext' is the regular expression which should match
the string representation of the raised *note SyntaxError: 18d. If
'lineno' is not ‘None’, compares to the line of the exception. If
'offset' is not ‘None’, compares to the offset of the exception.
-- Function: test.support.open_urlresource (url, *args, **kw)
Open 'url'. If open fails, raises *note TestFailed: 4135.
-- Function: test.support.reap_children ()
Use this at the end of ‘test_main’ whenever sub-processes are
started. This will help ensure that no extra children (zombies)
stick around to hog resources and create problems when looking for
refleaks.
-- Function: test.support.get_attribute (obj, name)
Get an attribute, raising *note unittest.SkipTest: fdf. if *note
AttributeError: 348. is raised.
-- Function: test.support.catch_unraisable_exception ()
Context manager catching unraisable exception using *note
sys.unraisablehook(): 1f9.
Storing the exception value (‘cm.unraisable.exc_value’) creates a
reference cycle. The reference cycle is broken explicitly when the
context manager exits.
Storing the object (‘cm.unraisable.object’) can resurrect it if it
is set to an object which is being finalized. Exiting the context
manager clears the stored object.
Usage:
with support.catch_unraisable_exception() as cm:
# code creating an "unraisable exception"
...
# check the unraisable exception: use cm.unraisable
...
# cm.unraisable attribute no longer exists at this point
# (to break a reference cycle)
Added in version 3.8.
-- Function: test.support.load_package_tests (pkg_dir, loader,
standard_tests, pattern)
Generic implementation of the *note unittest: 106. ‘load_tests’
protocol for use in test packages. 'pkg_dir' is the root directory
of the package; 'loader', 'standard_tests', and 'pattern' are the
arguments expected by ‘load_tests’. In simple cases, the test
package’s ‘__init__.py’ can be the following:
import os
from test.support import load_package_tests
def load_tests(*args):
return load_package_tests(os.path.dirname(__file__), *args)
-- Function: test.support.detect_api_mismatch (ref_api, other_api, *,
ignore=())
Returns the set of attributes, functions or methods of 'ref_api'
not found on 'other_api', except for a defined list of items to be
ignored in this check specified in 'ignore'.
By default this skips private attributes beginning with ‘_’ but
includes all magic methods, i.e. those starting and ending in
‘__’.
Added in version 3.5.
-- Function: test.support.patch (test_instance, object_to_patch,
attr_name, new_value)
Override 'object_to_patch.attr_name' with 'new_value'. Also add
cleanup procedure to 'test_instance' to restore 'object_to_patch'
for 'attr_name'. The 'attr_name' should be a valid attribute for
'object_to_patch'.
-- Function: test.support.run_in_subinterp (code)
Run 'code' in subinterpreter. Raise *note unittest.SkipTest: fdf.
if *note tracemalloc: ff. is enabled.
-- Function: test.support.check_free_after_iterating (test, iter, cls,
args=())
Assert instances of 'cls' are deallocated after iterating.
-- Function: test.support.missing_compiler_executable (cmd_names=[])
Check for the existence of the compiler executables whose names are
listed in 'cmd_names' or all the compiler executables when
'cmd_names' is empty and return the first missing executable or
‘None’ when none is found missing.
-- Function: test.support.check__all__ (test_case, module,
name_of_module=None, extra=(), not_exported=())
Assert that the ‘__all__’ variable of 'module' contains all public
names.
The module’s public names (its API) are detected automatically
based on whether they match the public name convention and were
defined in 'module'.
The 'name_of_module' argument can specify (as a string or tuple
thereof) what module(s) an API could be defined in order to be
detected as a public API. One case for this is when 'module'
imports part of its public API from other modules, possibly a C
backend (like ‘csv’ and its ‘_csv’).
The 'extra' argument can be a set of names that wouldn’t otherwise
be automatically detected as “public”, like objects without a
proper *note __module__: 2276. attribute. If provided, it will be
added to the automatically detected ones.
The 'not_exported' argument can be a set of names that must not be
treated as part of the public API even though their names indicate
otherwise.
Example use:
import bar
import foo
import unittest
from test import support
class MiscTestCase(unittest.TestCase):
def test__all__(self):
support.check__all__(self, foo)
class OtherTestCase(unittest.TestCase):
def test__all__(self):
extra = {'BAR_CONST', 'FOO_CONST'}
not_exported = {'baz'} # Undocumented name.
# bar imports part of its API from _bar.
support.check__all__(self, bar, ('bar', '_bar'),
extra=extra, not_exported=not_exported)
Added in version 3.6.
-- Function: test.support.skip_if_broken_multiprocessing_synchronize ()
Skip tests if the ‘multiprocessing.synchronize’ module is missing,
if there is no available semaphore implementation, or if creating a
lock raises an *note OSError: 413.
Added in version 3.10.
-- Function: test.support.check_disallow_instantiation (test_case, tp,
*args, **kwds)
Assert that type 'tp' cannot be instantiated using 'args' and
'kwds'.
Added in version 3.10.
-- Function: test.support.adjust_int_max_str_digits (max_digits)
This function returns a context manager that will change the global
*note sys.set_int_max_str_digits(): 17be. setting for the duration
of the context to allow execution of test code that needs a
different limit on the number of digits when converting between an
integer and string.
Added in version 3.11.
The *note test.support: e4. module defines the following classes:
-- Class: test.support.SuppressCrashReport
A context manager used to try to prevent crash dialog popups on
tests that are expected to crash a subprocess.
On Windows, it disables Windows Error Reporting dialogs using
SetErrorMode(1).
On UNIX, *note resource.setrlimit(): 151d. is used to set *note
resource.RLIMIT_CORE: 418f.’s soft limit to 0 to prevent coredump
file creation.
On both platforms, the old value is restored by *note __exit__():
12f3.
-- Class: test.support.SaveSignals
Class to save and restore signal handlers registered by the Python
signal handler.
-- Method: save (self)
Save the signal handlers to a dictionary mapping signal
numbers to the current signal handler.
-- Method: restore (self)
Set the signal numbers from the *note save(): 4191. dictionary
to the saved handler.
-- Class: test.support.Matcher
-- Method: matches (self, d, **kwargs)
Try to match a single dict with the supplied arguments.
-- Method: match_value (self, k, dv, v)
Try to match a single stored value ('dv') with a supplied
value ('v').
---------- Footnotes ----------
(1)
https://msdn.microsoft.com/en-us/library/windows/desktop/ms680621.aspx
File: python.info, Node: test support socket_helper — Utilities for socket tests, Next: test support script_helper — Utilities for the Python execution tests, Prev: test support — Utilities for the Python test suite, Up: Development Tools
5.27.10 ‘test.support.socket_helper’ — Utilities for socket tests
-----------------------------------------------------------------
The *note test.support.socket_helper: e9. module provides support for
socket tests.
Added in version 3.9.
-- Data: test.support.socket_helper.IPV6_ENABLED
Set to ‘True’ if IPv6 is enabled on this host, ‘False’ otherwise.
-- Function: test.support.socket_helper.find_unused_port
(family=socket.AF_INET, socktype=socket.SOCK_STREAM)
Returns an unused port that should be suitable for binding. This
is achieved by creating a temporary socket with the same family and
type as the ‘sock’ parameter (default is *note AF_INET: 181c, *note
SOCK_STREAM: 12e6.), and binding it to the specified host address
(defaults to ‘0.0.0.0’) with the port set to 0, eliciting an unused
ephemeral port from the OS. The temporary socket is then closed and
deleted, and the ephemeral port is returned.
Either this method or *note bind_port(): 4199. should be used for
any tests where a server socket needs to be bound to a particular
port for the duration of the test. Which one to use depends on
whether the calling code is creating a Python socket, or if an
unused port needs to be provided in a constructor or passed to an
external program (i.e. the ‘-accept’ argument to openssl’s
s_server mode). Always prefer *note bind_port(): 4199. over *note
find_unused_port(): 4198. where possible. Using a hard coded port
is discouraged since it can make multiple instances of the test
impossible to run simultaneously, which is a problem for buildbots.
-- Function: test.support.socket_helper.bind_port (sock, host=HOST)
Bind the socket to a free port and return the port number. Relies
on ephemeral ports in order to ensure we are using an unbound port.
This is important as many tests may be running simultaneously,
especially in a buildbot environment. This method raises an
exception if the ‘sock.family’ is *note AF_INET: 181c. and
‘sock.type’ is *note SOCK_STREAM: 12e6, and the socket has
‘SO_REUSEADDR’ or ‘SO_REUSEPORT’ set on it. Tests should never set
these socket options for TCP/IP sockets. The only case for setting
these options is testing multicasting via multiple UDP sockets.
Additionally, if the ‘SO_EXCLUSIVEADDRUSE’ socket option is
available (i.e. on Windows), it will be set on the socket. This
will prevent anyone else from binding to our host/port for the
duration of the test.
-- Function: test.support.socket_helper.bind_unix_socket (sock, addr)
Bind a Unix socket, raising *note unittest.SkipTest: fdf. if *note
PermissionError: d42. is raised.
-- Function: @test.support.socket_helper.skip_unless_bind_unix_socket
A decorator for running tests that require a functional ‘bind()’
for Unix sockets.
-- Function: test.support.socket_helper.transient_internet
(resource_name, *, timeout=30.0, errnos=())
A context manager that raises *note ResourceDenied: 4136. when
various issues with the internet connection manifest themselves as
exceptions.
File: python.info, Node: test support script_helper — Utilities for the Python execution tests, Next: test support bytecode_helper — Support tools for testing correct bytecode generation, Prev: test support socket_helper — Utilities for socket tests, Up: Development Tools
5.27.11 ‘test.support.script_helper’ — Utilities for the Python execution tests
-------------------------------------------------------------------------------
The *note test.support.script_helper: e8. module provides support for
Python’s script execution tests.
-- Function:
test.support.script_helper.interpreter_requires_environment ()
Return ‘True’ if ‘sys.executable interpreter’ requires environment
variables in order to be able to run at all.
This is designed to be used with ‘@unittest.skipIf()’ to annotate
tests that need to use an ‘assert_python*()’ function to launch an
isolated mode (‘-I’) or no environment mode (‘-E’) sub-interpreter
process.
A normal build & test does not run into this situation but it can
happen when trying to run the standard library test suite from an
interpreter that doesn’t have an obvious home with Python’s current
home finding logic.
Setting *note PYTHONHOME: 3b8. is one way to get most of the
testsuite to run in that situation. *note PYTHONPATH: 1016. or
‘PYTHONUSERSITE’ are other common environment variables that might
impact whether or not the interpreter can start.
-- Function: test.support.script_helper.run_python_until_end (*args,
**env_vars)
Set up the environment based on 'env_vars' for running the
interpreter in a subprocess. The values can include ‘__isolated’,
‘__cleanenv’, ‘__cwd’, and ‘TERM’.
Changed in version 3.9: The function no longer strips whitespaces
from 'stderr'.
-- Function: test.support.script_helper.assert_python_ok (*args,
**env_vars)
Assert that running the interpreter with 'args' and optional
environment variables 'env_vars' succeeds (‘rc == 0’) and return a
‘(return code, stdout, stderr)’ tuple.
If the '__cleanenv' keyword-only parameter is set, 'env_vars' is
used as a fresh environment.
Python is started in isolated mode (command line option ‘-I’),
except if the '__isolated' keyword-only parameter is set to
‘False’.
Changed in version 3.9: The function no longer strips whitespaces
from 'stderr'.
-- Function: test.support.script_helper.assert_python_failure (*args,
**env_vars)
Assert that running the interpreter with 'args' and optional
environment variables 'env_vars' fails (‘rc != 0’) and return a
‘(return code, stdout, stderr)’ tuple.
See *note assert_python_ok(): 419f. for more options.
Changed in version 3.9: The function no longer strips whitespaces
from 'stderr'.
-- Function: test.support.script_helper.spawn_python (*args,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **kw)
Run a Python subprocess with the given arguments.
'kw' is extra keyword args to pass to *note subprocess.Popen():
199. Returns a *note subprocess.Popen: 199. object.
-- Function: test.support.script_helper.kill_python (p)
Run the given *note subprocess.Popen: 199. process until completion
and return stdout.
-- Function: test.support.script_helper.make_script (script_dir,
script_basename, source, omit_suffix=False)
Create script containing 'source' in path 'script_dir' and
'script_basename'. If 'omit_suffix' is ‘False’, append ‘.py’ to
the name. Return the full script path.
-- Function: test.support.script_helper.make_zip_script (zip_dir,
zip_basename, script_name, name_in_zip=None)
Create zip file at 'zip_dir' and 'zip_basename' with extension
‘zip’ which contains the files in 'script_name'. 'name_in_zip' is
the archive name. Return a tuple containing ‘(full path, full path
of archive name)’.
-- Function: test.support.script_helper.make_pkg (pkg_dir,
init_source='')
Create a directory named 'pkg_dir' containing an ‘__init__’ file
with 'init_source' as its contents.
-- Function: test.support.script_helper.make_zip_pkg (zip_dir,
zip_basename, pkg_name, script_basename, source, depth=1,
compiled=False)
Create a zip package directory with a path of 'zip_dir' and
'zip_basename' containing an empty ‘__init__’ file and a file
'script_basename' containing the 'source'. If 'compiled' is
‘True’, both source files will be compiled and added to the zip
package. Return a tuple of the full zip path and the archive name
for the zip file.
File: python.info, Node: test support bytecode_helper — Support tools for testing correct bytecode generation, Next: test support threading_helper — Utilities for threading tests, Prev: test support script_helper — Utilities for the Python execution tests, Up: Development Tools
5.27.12 ‘test.support.bytecode_helper’ — Support tools for testing correct bytecode generation
----------------------------------------------------------------------------------------------
The *note test.support.bytecode_helper: e5. module provides support for
testing and inspecting bytecode generation.
Added in version 3.9.
The module defines the following class:
-- Class: test.support.bytecode_helper.BytecodeTestCase
(unittest.TestCase)
This class has custom assertion methods for inspecting bytecode.
-- Method: BytecodeTestCase.get_disassembly_as_string (co)
Return the disassembly of 'co' as string.
-- Method: BytecodeTestCase.assertInBytecode (x, opname,
argval=_UNSPECIFIED)
Return instr if 'opname' is found, otherwise throws *note
AssertionError: 6a5.
-- Method: BytecodeTestCase.assertNotInBytecode (x, opname,
argval=_UNSPECIFIED)
Throws *note AssertionError: 6a5. if 'opname' is found.
File: python.info, Node: test support threading_helper — Utilities for threading tests, Next: test support os_helper — Utilities for os tests, Prev: test support bytecode_helper — Support tools for testing correct bytecode generation, Up: Development Tools
5.27.13 ‘test.support.threading_helper’ — Utilities for threading tests
-----------------------------------------------------------------------
The *note test.support.threading_helper: ea. module provides support for
threading tests.
Added in version 3.10.
-- Function: test.support.threading_helper.join_thread (thread,
timeout=None)
Join a 'thread' within 'timeout'. Raise an *note AssertionError:
6a5. if thread is still alive after 'timeout' seconds.
-- Function: @test.support.threading_helper.reap_threads
Decorator to ensure the threads are cleaned up even if the test
fails.
-- Function: test.support.threading_helper.start_threads (threads,
unlock=None)
Context manager to start 'threads', which is a sequence of threads.
'unlock' is a function called after the threads are started, even
if an exception was raised; an example would be *note
threading.Event.set(): 311d. ‘start_threads’ will attempt to join
the started threads upon exit.
-- Function: test.support.threading_helper.threading_cleanup
(*original_values)
Cleanup up threads not specified in 'original_values'. Designed to
emit a warning if a test leaves running threads in the background.
-- Function: test.support.threading_helper.threading_setup ()
Return current thread count and copy of dangling threads.
-- Function: test.support.threading_helper.wait_threads_exit
(timeout=None)
Context manager to wait until all threads created in the ‘with’
statement exit.
-- Function: test.support.threading_helper.catch_threading_exception ()
Context manager catching *note threading.Thread: 94c. exception
using *note threading.excepthook(): 847.
Attributes set when an exception is caught:
* ‘exc_type’
* ‘exc_value’
* ‘exc_traceback’
* ‘thread’
See *note threading.excepthook(): 847. documentation.
These attributes are deleted at the context manager exit.
Usage:
with threading_helper.catch_threading_exception() as cm:
# code spawning a thread which raises an exception
...
# check the thread exception, use cm attributes:
# exc_type, exc_value, exc_traceback, thread
...
# exc_type, exc_value, exc_traceback, thread attributes of cm no longer
# exists at this point
# (to avoid reference cycles)
Added in version 3.8.
File: python.info, Node: test support os_helper — Utilities for os tests, Next: test support import_helper — Utilities for import tests, Prev: test support threading_helper — Utilities for threading tests, Up: Development Tools
5.27.14 ‘test.support.os_helper’ — Utilities for os tests
---------------------------------------------------------
The *note test.support.os_helper: e7. module provides support for os
tests.
Added in version 3.10.
-- Data: test.support.os_helper.FS_NONASCII
A non-ASCII character encodable by *note os.fsencode(): c55.
-- Data: test.support.os_helper.SAVEDCWD
Set to *note os.getcwd(): 159b.
-- Data: test.support.os_helper.TESTFN
Set to a name that is safe to use as the name of a temporary file.
Any temporary file that is created should be closed and unlinked
(removed).
-- Data: test.support.os_helper.TESTFN_NONASCII
Set to a filename containing the *note FS_NONASCII: 41b5.
character, if it exists. This guarantees that if the filename
exists, it can be encoded and decoded with the default filesystem
encoding. This allows tests that require a non-ASCII filename to
be easily skipped on platforms where they can’t work.
-- Data: test.support.os_helper.TESTFN_UNENCODABLE
Set to a filename (str type) that should not be able to be encoded
by file system encoding in strict mode. It may be ‘None’ if it’s
not possible to generate such a filename.
-- Data: test.support.os_helper.TESTFN_UNDECODABLE
Set to a filename (bytes type) that should not be able to be
decoded by file system encoding in strict mode. It may be ‘None’
if it’s not possible to generate such a filename.
-- Data: test.support.os_helper.TESTFN_UNICODE
Set to a non-ASCII name for a temporary file.
-- Class: test.support.os_helper.EnvironmentVarGuard
Class used to temporarily set or unset environment variables.
Instances can be used as a context manager and have a complete
dictionary interface for querying/modifying the underlying
‘os.environ’. After exit from the context manager all changes to
environment variables done through this instance will be rolled
back.
Changed in version 3.1: Added dictionary interface.
-- Class: test.support.os_helper.FakePath (path)
Simple *note path-like object: 2a2. It implements the *note
__fspath__(): c52. method which just returns the 'path' argument.
If 'path' is an exception, it will be raised in ‘__fspath__()’.
-- Method: EnvironmentVarGuard.set (envvar, value)
Temporarily set the environment variable ‘envvar’ to the value of
‘value’.
-- Method: EnvironmentVarGuard.unset (envvar)
Temporarily unset the environment variable ‘envvar’.
-- Function: test.support.os_helper.can_symlink ()
Return ‘True’ if the OS supports symbolic links, ‘False’ otherwise.
-- Function: test.support.os_helper.can_xattr ()
Return ‘True’ if the OS supports xattr, ‘False’ otherwise.
-- Function: test.support.os_helper.change_cwd (path, quiet=False)
A context manager that temporarily changes the current working
directory to 'path' and yields the directory.
If 'quiet' is ‘False’, the context manager raises an exception on
error. Otherwise, it issues only a warning and keeps the current
working directory the same.
-- Function: test.support.os_helper.create_empty_file (filename)
Create an empty file with 'filename'. If it already exists,
truncate it.
-- Function: test.support.os_helper.fd_count ()
Count the number of open file descriptors.
-- Function: test.support.os_helper.fs_is_case_insensitive (directory)
Return ‘True’ if the file system for 'directory' is
case-insensitive.
-- Function: test.support.os_helper.make_bad_fd ()
Create an invalid file descriptor by opening and closing a
temporary file, and returning its descriptor.
-- Function: test.support.os_helper.rmdir (filename)
Call *note os.rmdir(): 10e8. on 'filename'. On Windows platforms,
this is wrapped with a wait loop that checks for the existence of
the file, which is needed due to antivirus programs that can hold
files open and prevent deletion.
-- Function: test.support.os_helper.rmtree (path)
Call *note shutil.rmtree(): 2fb. on 'path' or call *note
os.lstat(): 49d. and *note os.rmdir(): 10e8. to remove a path and
its contents. As with *note rmdir(): 41c6, on Windows platforms
this is wrapped with a wait loop that checks for the existence of
the files.
-- Function: @test.support.os_helper.skip_unless_symlink
A decorator for running tests that require support for symbolic
links.
-- Function: @test.support.os_helper.skip_unless_xattr
A decorator for running tests that require support for xattr.
-- Function: test.support.os_helper.temp_cwd (name='tempcwd',
quiet=False)
A context manager that temporarily creates a new directory and
changes the current working directory (CWD).
The context manager creates a temporary directory in the current
directory with name 'name' before temporarily changing the current
working directory. If 'name' is ‘None’, the temporary directory is
created using *note tempfile.mkdtemp(): 221.
If 'quiet' is ‘False’ and it is not possible to create or change
the CWD, an error is raised. Otherwise, only a warning is raised
and the original CWD is used.
-- Function: test.support.os_helper.temp_dir (path=None, quiet=False)
A context manager that creates a temporary directory at 'path' and
yields the directory.
If 'path' is ‘None’, the temporary directory is created using *note
tempfile.mkdtemp(): 221. If 'quiet' is ‘False’, the context
manager raises an exception on error. Otherwise, if 'path' is
specified and cannot be created, only a warning is issued.
-- Function: test.support.os_helper.temp_umask (umask)
A context manager that temporarily sets the process umask.
-- Function: test.support.os_helper.unlink (filename)
Call *note os.unlink(): 10ea. on 'filename'. As with *note
rmdir(): 41c6, on Windows platforms, this is wrapped with a wait
loop that checks for the existence of the file.
File: python.info, Node: test support import_helper — Utilities for import tests, Next: test support warnings_helper — Utilities for warnings tests, Prev: test support os_helper — Utilities for os tests, Up: Development Tools
5.27.15 ‘test.support.import_helper’ — Utilities for import tests
-----------------------------------------------------------------
The *note test.support.import_helper: e6. module provides support for
import tests.
Added in version 3.10.
-- Function: test.support.import_helper.forget (module_name)
Remove the module named 'module_name' from ‘sys.modules’ and delete
any byte-compiled files of the module.
-- Function: test.support.import_helper.import_fresh_module (name,
fresh=(), blocked=(), deprecated=False)
This function imports and returns a fresh copy of the named Python
module by removing the named module from ‘sys.modules’ before doing
the import. Note that unlike ‘reload()’, the original module is
not affected by this operation.
'fresh' is an iterable of additional module names that are also
removed from the ‘sys.modules’ cache before doing the import.
'blocked' is an iterable of module names that are replaced with
‘None’ in the module cache during the import to ensure that
attempts to import them raise *note ImportError: 415.
The named module and any modules named in the 'fresh' and 'blocked'
parameters are saved before starting the import and then reinserted
into ‘sys.modules’ when the fresh import is complete.
Module and package deprecation messages are suppressed during this
import if 'deprecated' is ‘True’.
This function will raise *note ImportError: 415. if the named
module cannot be imported.
Example use:
# Get copies of the warnings module for testing without affecting the
# version being used by the rest of the test suite. One copy uses the
# C implementation, the other is forced to use the pure Python fallback
# implementation
py_warnings = import_fresh_module('warnings', blocked=['_warnings'])
c_warnings = import_fresh_module('warnings', fresh=['_warnings'])
Added in version 3.1.
-- Function: test.support.import_helper.import_module (name,
deprecated=False, *, required_on=())
This function imports and returns the named module. Unlike a
normal import, this function raises *note unittest.SkipTest: fdf.
if the module cannot be imported.
Module and package deprecation messages are suppressed during this
import if 'deprecated' is ‘True’. If a module is required on a
platform but optional for others, set 'required_on' to an iterable
of platform prefixes which will be compared against *note
sys.platform: a8d.
Added in version 3.1.
-- Function: test.support.import_helper.modules_setup ()
Return a copy of *note sys.modules: 1550.
-- Function: test.support.import_helper.modules_cleanup (oldmodules)
Remove modules except for 'oldmodules' and ‘encodings’ in order to
preserve internal cache.
-- Function: test.support.import_helper.unload (name)
Delete 'name' from ‘sys.modules’.
-- Function: test.support.import_helper.make_legacy_pyc (source)
Move a PEP 3147(1)/ PEP 488(2) pyc file to its legacy pyc location
and return the file system path to the legacy pyc file. The
'source' value is the file system path to the source file. It does
not need to exist, however the PEP 3147/488 pyc file must exist.
-- Class: test.support.import_helper.CleanImport (*module_names)
A context manager to force import to return a new module reference.
This is useful for testing module-level behaviors, such as the
emission of a *note DeprecationWarning: 1a5. on import. Example
usage:
with CleanImport('foo'):
importlib.import_module('foo') # New reference.
-- Class: test.support.import_helper.DirsOnSysPath (*paths)
A context manager to temporarily add directories to *note sys.path:
3b0.
This makes a copy of *note sys.path: 3b0, appends any directories
given as positional arguments, then reverts *note sys.path: 3b0. to
the copied settings when the context ends.
Note that 'all' *note sys.path: 3b0. modifications in the body of
the context manager, including replacement of the object, will be
reverted at the end of the block.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3147/
(2) https://peps.python.org/pep-0488/
File: python.info, Node: test support warnings_helper — Utilities for warnings tests, Prev: test support import_helper — Utilities for import tests, Up: Development Tools
5.27.16 ‘test.support.warnings_helper’ — Utilities for warnings tests
---------------------------------------------------------------------
The *note test.support.warnings_helper: eb. module provides support for
warnings tests.
Added in version 3.10.
-- Function: test.support.warnings_helper.ignore_warnings (*, category)
Suppress warnings that are instances of 'category', which must be
*note Warning: 22b3. or a subclass. Roughly equivalent to *note
warnings.catch_warnings(): 581. with *note
warnings.simplefilter('ignore', category=category): 6bd. For
example:
@warning_helper.ignore_warnings(category=DeprecationWarning)
def test_suppress_warning():
# do something
Added in version 3.8.
-- Function: test.support.warnings_helper.check_no_resource_warning
(testcase)
Context manager to check that no *note ResourceWarning: 246. was
raised. You must remove the object which may emit *note
ResourceWarning: 246. before the end of the context manager.
-- Function: test.support.warnings_helper.check_syntax_warning
(testcase, statement, errtext='', *, lineno=1, offset=None)
Test for syntax warning in 'statement' by attempting to compile
'statement'. Test also that the *note SyntaxWarning: 461. is
emitted only once, and that it will be converted to a *note
SyntaxError: 18d. when turned into error. 'testcase' is the *note
unittest: 106. instance for the test. 'errtext' is the regular
expression which should match the string representation of the
emitted *note SyntaxWarning: 461. and raised *note SyntaxError:
18d. If 'lineno' is not ‘None’, compares to the line of the
warning and exception. If 'offset' is not ‘None’, compares to the
offset of the exception.
Added in version 3.8.
-- Function: test.support.warnings_helper.check_warnings (*filters,
quiet=True)
A convenience wrapper for *note warnings.catch_warnings(): 581.
that makes it easier to test that a warning was correctly raised.
It is approximately equivalent to calling
‘warnings.catch_warnings(record=True)’ with *note
warnings.simplefilter(): 6bd. set to ‘always’ and with the option
to automatically validate the results that are recorded.
‘check_warnings’ accepts 2-tuples of the form ‘("message regexp",
WarningCategory)’ as positional arguments. If one or more
'filters' are provided, or if the optional keyword argument 'quiet'
is ‘False’, it checks to make sure the warnings are as expected:
each specified filter must match at least one of the warnings
raised by the enclosed code or the test fails, and if any warnings
are raised that do not match any of the specified filters the test
fails. To disable the first of these checks, set 'quiet' to
‘True’.
If no arguments are specified, it defaults to:
check_warnings(("", Warning), quiet=True)
In this case all warnings are caught and no errors are raised.
On entry to the context manager, a ‘WarningRecorder’ instance is
returned. The underlying warnings list from *note
catch_warnings(): 581. is available via the recorder object’s *note
warnings: 112. attribute. As a convenience, the attributes of the
object representing the most recent warning can also be accessed
directly through the recorder object (see example below). If no
warning has been raised, then any of the attributes that would
otherwise be expected on an object representing a warning will
return ‘None’.
The recorder object also has a ‘reset()’ method, which clears the
warnings list.
The context manager is designed to be used like this:
with check_warnings(("assertion is always true", SyntaxWarning),
("", UserWarning)):
exec('assert(False, "Hey!")')
warnings.warn(UserWarning("Hide me!"))
In this case if either warning was not raised, or some other
warning was raised, *note check_warnings(): 41db. would raise an
error.
When a test needs to look more deeply into the warnings, rather
than just checking whether or not they occurred, code like this can
be used:
with check_warnings(quiet=True) as w:
warnings.warn("foo")
assert str(w.args[0]) == "foo"
warnings.warn("bar")
assert str(w.args[0]) == "bar"
assert str(w.warnings[0].args[0]) == "foo"
assert str(w.warnings[1].args[0]) == "bar"
w.reset()
assert len(w.warnings) == 0
Here all warnings will be caught, and the test code tests the
captured warnings directly.
Changed in version 3.2: New optional arguments 'filters' and
'quiet'.
-- Class: test.support.warnings_helper.WarningsRecorder
Class used to record warnings for unit tests. See documentation of
*note check_warnings(): 41db. above for more details.
File: python.info, Node: Debugging and Profiling, Next: Software Packaging and Distribution, Prev: Development Tools, Up: The Python Standard Library
5.28 Debugging and Profiling
============================
These libraries help you with Python development: the debugger enables
you to step through code, analyze stack frames and set breakpoints etc.,
and the profilers run code and give you a detailed breakdown of
execution times, allowing you to identify bottlenecks in your programs.
Auditing events provide visibility into runtime behaviors that would
otherwise require intrusive debugging or patching.
* Menu:
* Audit events table::
* bdb — Debugger framework::
* faulthandler — Dump the Python traceback::
* pdb — The Python Debugger::
* The Python Profilers::
* timeit — Measure execution time of small code snippets::
* trace — Trace or track Python statement execution::
* tracemalloc — Trace memory allocations::
File: python.info, Node: Audit events table, Next: bdb — Debugger framework, Up: Debugging and Profiling
5.28.1 Audit events table
-------------------------
This table contains all events raised by *note sys.audit(): 15b9. or
*note PySys_Audit(): 36a. calls throughout the CPython runtime and the
standard library. These calls were added in 3.8 or later (see PEP
578(1)).
See *note sys.addaudithook(): 41e1. and *note PySys_AddAuditHook():
41e2. for information on handling these events.
'CPython implementation detail:' This table is generated from the
CPython documentation, and may not represent events raised by other
implementations. See your runtime specific documentation for actual
events raised.
Audit event Arguments References
-------------------------------------------------------------------------------------------------------------------
_thread.start_new_thread ‘function’, ‘args’, ‘kwargs’ *note [1]: 41e3.
array.__new__ ‘typecode’, ‘initializer’ *note [1]: 1a0.
builtins.breakpoint ‘breakpointhook’ *note [1]: 236.
builtins.id ‘id’ *note [1]: 13ee.
builtins.input ‘prompt’ *note [1]: 12cb.
builtins.input/result ‘result’ *note [1]: 12cb.
code.__new__ ‘code’, ‘filename’, ‘name’, ‘argcount’, *note [1]: 2e3.
‘posonlyargcount’, ‘kwonlyargcount’, ‘nlocals’,
‘stacksize’, ‘flags’
compile ‘source’, ‘filename’ *note [1]: 192.
cpython.PyInterpreterState_Clear *note [1]: 41e4.
cpython.PyInterpreterState_New *note [1]: 41e5.
cpython._PySys_ClearAuditHooks *note [1]: d15.
cpython.run_command ‘command’ *note [1]: 5dc.
cpython.run_file ‘filename’ *note [1]: 41e6.
cpython.run_interactivehook ‘hook’ *note [1]: fc2.
cpython.run_module ‘module-name’ *note [1]: 5dd.
cpython.run_startup ‘filename’ *note [1]: fc3.
cpython.run_stdin *note [1]: 41e7.*note [2]: 41e8.*note [3]: 41e9.
ctypes.addressof ‘obj’ *note [1]: 2e47.
ctypes.call_function ‘func_pointer’, ‘arguments’ *note [1]: 2e3f.
ctypes.cdata ‘address’ *note [1]: 2e59.
ctypes.cdata/buffer ‘pointer’, ‘size’, ‘offset’ *note [1]: 2e57.*note [2]: 2e58.
ctypes.create_string_buffer ‘init’, ‘size’ *note [1]: 2dfc.
ctypes.create_unicode_buffer ‘init’, ‘size’ *note [1]: 19ce.
ctypes.dlopen ‘name’ *note [1]: 2e3d.
ctypes.dlsym ‘library’, ‘name’ *note [1]: 2e3d.
ctypes.dlsym/handle ‘handle’, ‘name’ *note [1]: 2e3d.
ctypes.get_errno *note [1]: 2e37.
ctypes.get_last_error *note [1]: 2e39.
ctypes.set_errno ‘errno’ *note [1]: 2e38.
ctypes.set_exception ‘code’ *note [1]: 2e3f.
ctypes.set_last_error ‘error’ *note [1]: 2e3a.
ctypes.string_at ‘ptr’, ‘size’ *note [1]: 2e50.
ctypes.wstring_at ‘ptr’, ‘size’ *note [1]: 2e52.
ensurepip.bootstrap ‘root’ *note [1]: 41ea.
exec ‘code_object’ *note [1]: 180.*note [2]: 17f.
fcntl.fcntl ‘fd’, ‘cmd’, ‘arg’ *note [1]: 344b.
fcntl.flock ‘fd’, ‘operation’ *note [1]: 41eb.
fcntl.ioctl ‘fd’, ‘request’, ‘arg’ *note [1]: 1443.
fcntl.lockf ‘fd’, ‘cmd’, ‘len’, ‘start’, ‘whence’ *note [1]: 13b8.
ftplib.connect ‘self’, ‘host’, ‘port’ *note [1]: 3a84.
ftplib.sendcmd ‘self’, ‘cmd’ *note [1]: 3a89.*note [2]: 3a8a.
function.__new__ ‘code’ *note [1]: 881.
gc.get_objects ‘generation’ *note [1]: 7ff.
gc.get_referents ‘objs’ *note [1]: 801.
gc.get_referrers ‘objs’ *note [1]: 800.
glob.glob ‘pathname’, ‘recursive’ *note [1]: 2a1.*note [2]: 803.
glob.glob/2 ‘pathname’, ‘recursive’, ‘root_dir’, ‘dir_fd’ *note [1]: 2a1.*note [2]: 803.
http.client.connect ‘self’, ‘host’, ‘port’ *note [1]: 3a67.
http.client.send ‘self’, ‘data’ *note [1]: 3a6c.
imaplib.open ‘self’, ‘host’, ‘port’ *note [1]: 904.
imaplib.send ‘self’, ‘data’ *note [1]: 3adb.
import ‘module’, ‘filename’, ‘sys.path’, ‘sys.meta_path’, *note [1]: 5de.
‘sys.path_hooks’
marshal.dumps ‘value’, ‘version’ *note [1]: 28c1.
marshal.load *note [1]: 1a12.
marshal.loads ‘bytes’ *note [1]: 1a12.
mmap.__new__ ‘fileno’, ‘length’, ‘access’, ‘offset’ *note [1]: 20d.
msvcrt.get_osfhandle ‘fd’ *note [1]: 41ec.
msvcrt.locking ‘fd’, ‘mode’, ‘nbytes’ *note [1]: 41ed.
msvcrt.open_osfhandle ‘handle’, ‘flags’ *note [1]: 41ee.
object.__delattr__ ‘obj’, ‘name’ *note [1]: 1f2e.
object.__getattr__ ‘obj’, ‘name’ *note [1]: bd2.
object.__setattr__ ‘obj’, ‘name’, ‘value’ *note [1]: 1f2d.
open ‘path’, ‘mode’, ‘flags’ *note [1]: 517.*note [2]: 518.*note [3]: d91.
os.add_dll_directory ‘path’ *note [1]: 9e9.
os.chdir ‘path’ *note [1]: 609.*note [2]: d88.
os.chflags ‘path’, ‘flags’ *note [1]: 10e2.*note [2]: 2b86.
os.chmod ‘path’, ‘mode’, ‘dir_fd’ *note [1]: 21d.*note [2]: 21e.*note [3]: 21c.
os.chown ‘path’, ‘uid’, ‘gid’, ‘dir_fd’ *note [1]: 10e3.*note [2]: d89.*note [3]: 2b87.
os.exec ‘path’, ‘args’, ‘env’ *note [1]: 2bc3.
os.fork *note [1]: 197.
os.forkpty *note [1]: 198.
os.fwalk ‘top’, ‘topdown’, ‘onerror’, ‘follow_symlinks’, ‘dir_fd’ *note [1]: b5b.
os.getxattr ‘path’, ‘attribute’ *note [1]: 10f8.
os.kill ‘pid’, ‘sig’ *note [1]: 1353.
os.killpg ‘pgid’, ‘sig’ *note [1]: 19d5.
os.link ‘src’, ‘dst’, ‘src_dir_fd’, ‘dst_dir_fd’ *note [1]: 10e4.
os.listdir ‘path’ *note [1]: 10ee.
os.listdrives *note [1]: 499.
os.listmounts ‘volume’ *note [1]: 49b.
os.listvolumes *note [1]: 49a.
os.listxattr ‘path’ *note [1]: 10f9.
os.lockf ‘fd’, ‘cmd’, ‘len’ *note [1]: 1107.
os.mkdir ‘path’, ‘mode’, ‘dir_fd’ *note [1]: 220.*note [2]: 21f.
os.posix_spawn ‘path’, ‘argv’, ‘env’ *note [1]: 222.*note [2]: 1a2e.
os.putenv ‘key’, ‘value’ *note [1]: 91b.
os.remove ‘path’, ‘dir_fd’ *note [1]: 10e5.*note [2]: 2b8b.*note [3]: 10ea.
os.removexattr ‘path’, ‘attribute’ *note [1]: 10fa.
os.rename ‘src’, ‘dst’, ‘src_dir_fd’, ‘dst_dir_fd’ *note [1]: 10e6.*note [2]: 2b8c.*note [3]: 10e7.
os.rmdir ‘path’, ‘dir_fd’ *note [1]: 10e8.
os.scandir ‘path’ *note [1]: a5e.
os.setxattr ‘path’, ‘attribute’, ‘value’, ‘flags’ *note [1]: 10fb.
os.spawn ‘mode’, ‘path’, ‘args’, ‘env’ *note [1]: 1731.
os.startfile ‘path’, ‘operation’ *note [1]: 18cb.
os.startfile/2 ‘path’, ‘operation’, ‘arguments’, ‘cwd’, ‘show_cmd’ *note [1]: 18cb.
os.symlink ‘src’, ‘dst’, ‘dir_fd’ *note [1]: 10e9.
os.system ‘command’ *note [1]: 1426.
os.truncate ‘fd’, ‘length’ *note [1]: d8e.*note [2]: e3b.
os.unsetenv ‘key’ *note [1]: 91a.
os.utime ‘path’, ‘times’, ‘ns’, ‘dir_fd’ *note [1]: 10eb.
os.walk ‘top’, ‘topdown’, ‘onerror’, ‘followlinks’ *note [1]: 4a5.
pathlib.Path.glob ‘self’, ‘pattern’ *note [1]: 22f.
pathlib.Path.rglob ‘self’, ‘pattern’ *note [1]: 230.
pdb.Pdb *note [1]: 921.
pickle.find_class ‘module’, ‘name’ *note [1]: 289f.
poplib.connect ‘self’, ‘host’, ‘port’ *note [1]: 923.*note [2]: 924.
poplib.putline ‘self’, ‘line’ *note [1]: 923.*note [2]: 924.
pty.spawn ‘argv’ *note [1]: f7d.
resource.prlimit ‘pid’, ‘resource’, ‘limits’ *note [1]: f84.
resource.setrlimit ‘resource’, ‘limits’ *note [1]: 151d.
setopencodehook *note [1]: 1729.
shutil.chown ‘path’, ‘user’, ‘group’ *note [1]: 242.
shutil.copyfile ‘src’, ‘dst’ *note [1]: a59.*note [2]: a5a.*note [3]: a58.
shutil.copymode ‘src’, ‘dst’ *note [1]: a59.*note [2]: 16ab.
shutil.copystat ‘src’, ‘dst’ *note [1]: a5a.*note [2]: 1122.
shutil.copytree ‘src’, ‘dst’ *note [1]: a28.
shutil.make_archive ‘base_name’, ‘format’, ‘root_dir’, ‘base_dir’ *note [1]: 4af.
shutil.move ‘src’, ‘dst’ *note [1]: a5b.
shutil.rmtree ‘path’, ‘dir_fd’ *note [1]: 2fb.
shutil.unpack_archive ‘filename’, ‘extract_dir’, ‘format’ *note [1]: 467.
signal.pthread_kill ‘thread_id’, ‘signalnum’ *note [1]: 1125.
smtplib.connect ‘self’, ‘host’, ‘port’ *note [1]: 19c9.
smtplib.send ‘self’, ‘data’ *note [1]: 92d.
socket.__new__ ‘self’, ‘family’, ‘type’, ‘protocol’ *note [1]: da0.
socket.bind ‘self’, ‘address’ *note [1]: 1504.
socket.connect ‘self’, ‘address’ *note [1]: da2.*note [2]: 1503.
socket.getaddrinfo ‘host’, ‘port’, ‘family’, ‘type’, ‘protocol’ *note [1]: 1752.
socket.gethostbyaddr ‘ip_address’ *note [1]: 18cc.
socket.gethostbyname ‘hostname’ *note [1]: 3440.*note [2]: 18cd.
socket.gethostname *note [1]: 2b31.
socket.getnameinfo ‘sockaddr’ *note [1]: 16c3.
socket.getservbyname ‘servicename’, ‘protocolname’ *note [1]: 3442.
socket.getservbyport ‘port’, ‘protocolname’ *note [1]: 3443.
socket.sendmsg ‘self’, ‘address’ *note [1]: 472.
socket.sendto ‘self’, ‘address’ *note [1]: da8.
socket.sethostname ‘name’ *note [1]: 112f.
sqlite3.connect ‘database’ *note [1]: 2aa.
sqlite3.connect/handle ‘connection_handle’ *note [1]: 2aa.
sqlite3.enable_load_extension ‘connection’, ‘enabled’ *note [1]: 83a.
sqlite3.load_extension ‘connection’, ‘path’ *note [1]: 4b4.
subprocess.Popen ‘executable’, ‘args’, ‘cwd’, ‘env’ *note [1]: 199.
sys._current_exceptions *note [1]: 4be.
sys._current_frames *note [1]: 13fe.
sys._getframe ‘frame’ *note [1]: 6dc.
sys._getframemodulename ‘depth’ *note [1]: 176e.
sys.addaudithook *note [1]: 41e2.*note [2]: 41e1.
sys.excepthook ‘hook’, ‘type’, ‘value’, ‘traceback’ *note [1]: 807.
sys.monitoring.register_callback ‘func’ *note [1]: 41ef.
sys.set_asyncgen_hooks_finalizer *note [1]: 1626.
sys.set_asyncgen_hooks_firstiter *note [1]: 1626.
sys.setprofile *note [1]: 14c9.
sys.settrace *note [1]: 14ca.
sys.unraisablehook ‘hook’, ‘unraisable’ *note [1]: 1f9.
syslog.closelog *note [1]: 53c.
syslog.openlog ‘ident’, ‘logoption’, ‘facility’ *note [1]: 53b.
syslog.setlogmask ‘maskpri’ *note [1]: 41f0.
syslog.syslog ‘priority’, ‘message’ *note [1]: 53d.
tempfile.mkdtemp ‘fullpath’ *note [1]: 1227.*note [2]: 221.
tempfile.mkstemp ‘fullpath’ *note [1]: 4c2.*note [2]: 2853.*note [3]: 2854.
time.sleep ‘secs’ *note [1]: 41f1.
urllib.Request ‘fullurl’, ‘data’, ‘headers’, ‘method’ *note [1]: 295.
webbrowser.open ‘url’ *note [1]: 394a.
winreg.ConnectRegistry ‘computer_name’, ‘key’ *note [1]: 41f2.
winreg.CreateKey ‘key’, ‘sub_key’, ‘access’ *note [1]: 41f3.*note [2]: 134e.
winreg.DeleteKey ‘key’, ‘sub_key’, ‘access’ *note [1]: 41f4.*note [2]: 134f.
winreg.DeleteValue ‘key’, ‘value’ *note [1]: 41f5.
winreg.DisableReflectionKey ‘key’ *note [1]: 1350.
winreg.EnableReflectionKey ‘key’ *note [1]: 1351.
winreg.EnumKey ‘key’, ‘index’ *note [1]: 41f6.
winreg.EnumValue ‘key’, ‘index’ *note [1]: 41f7.
winreg.ExpandEnvironmentStrings ‘str’ *note [1]: 13c4.
winreg.LoadKey ‘key’, ‘sub_key’, ‘file_name’ *note [1]: 41f8.
winreg.OpenKey ‘key’, ‘sub_key’, ‘access’ *note [1]: 41f9.
winreg.OpenKey/result ‘key’ *note [1]: 41f3.*note [2]: 134e.*note [3]: 41f9.
winreg.PyHKEY.Detach ‘key’ *note [1]: 41fa.
winreg.QueryInfoKey ‘key’ *note [1]: 41fb.
winreg.QueryReflectionKey ‘key’ *note [1]: 1352.
winreg.QueryValue ‘key’, ‘sub_key’, ‘value_name’ *note [1]: 41fc.*note [2]: 41fd.
winreg.SaveKey ‘key’, ‘file_name’ *note [1]: 41fe.
winreg.SetValue ‘key’, ‘sub_key’, ‘type’, ‘value’ *note [1]: 41ff.*note [2]: 178d.
The following events are raised internally and do not correspond to any
public API of CPython:
Audit event Arguments
-------------------------------------------------------------------------------
_winapi.CreateFile ‘file_name’, ‘desired_access’, ‘share_mode’,
‘creation_disposition’,
‘flags_and_attributes’
_winapi.CreateJunction ‘src_path’, ‘dst_path’
_winapi.CreateNamedPipe ‘name’, ‘open_mode’, ‘pipe_mode’
_winapi.CreatePipe
_winapi.CreateProcess ‘application_name’, ‘command_line’,
‘current_directory’
_winapi.OpenProcess ‘process_id’, ‘desired_access’
_winapi.TerminateProcess ‘handle’, ‘exit_code’
ctypes.PyObj_FromPtr ‘obj’
---------- Footnotes ----------
(1) https://peps.python.org/pep-0578/
File: python.info, Node: bdb — Debugger framework, Next: faulthandler — Dump the Python traceback, Prev: Audit events table, Up: Debugging and Profiling
5.28.2 ‘bdb’ — Debugger framework
---------------------------------
'Source code:' Lib/bdb.py(1)
__________________________________________________________________
The *note bdb: f. module handles basic debugger functions, like setting
breakpoints or managing execution via the debugger.
The following exception is defined:
-- Exception: bdb.BdbQuit
Exception raised by the *note Bdb: 12f5. class for quitting the
debugger.
The *note bdb: f. module also defines two classes:
-- Class: bdb.Breakpoint (self, file, line, temporary=False, cond=None,
funcname=None)
This class implements temporary breakpoints, ignore counts,
disabling and (re-)enabling, and conditionals.
Breakpoints are indexed by number through a list called *note
bpbynumber: 4204. and by ‘(file, line)’ pairs through *note bplist:
4205. The former points to a single instance of class *note
Breakpoint: 4203. The latter points to a list of such instances
since there may be more than one breakpoint per line.
When creating a breakpoint, its associated *note file name: 4206.
should be in canonical form. If a *note funcname: 4207. is
defined, a breakpoint *note hit: 4208. will be counted when the
first line of that function is executed. A *note conditional:
4209. breakpoint always counts a *note hit: 4208.
*note Breakpoint: 4203. instances have the following methods:
-- Method: deleteMe ()
Delete the breakpoint from the list associated to a file/line.
If it is the last breakpoint in that position, it also deletes
the entry for the file/line.
-- Method: enable ()
Mark the breakpoint as enabled.
-- Method: disable ()
Mark the breakpoint as disabled.
-- Method: bpformat ()
Return a string with all the information about the breakpoint,
nicely formatted:
* Breakpoint number.
* Temporary status (del or keep).
* File/line position.
* Break condition.
* Number of times to ignore.
* Number of times hit.
Added in version 3.2.
-- Method: bpprint (out=None)
Print the output of *note bpformat(): 420d. to the file 'out',
or if it is ‘None’, to standard output.
*note Breakpoint: 4203. instances have the following attributes:
-- Attribute: file
File name of the *note Breakpoint: 4203.
-- Attribute: line
Line number of the *note Breakpoint: 4203. within *note file:
4206.
-- Attribute: temporary
‘True’ if a *note Breakpoint: 4203. at (file, line) is
temporary.
-- Attribute: cond
Condition for evaluating a *note Breakpoint: 4203. at (file,
line).
-- Attribute: funcname
Function name that defines whether a *note Breakpoint: 4203.
is hit upon entering the function.
-- Attribute: enabled
‘True’ if *note Breakpoint: 4203. is enabled.
-- Attribute: bpbynumber
Numeric index for a single instance of a *note Breakpoint:
4203.
-- Attribute: bplist
Dictionary of *note Breakpoint: 4203. instances indexed by
(*note file: 4206, *note line: 420f.) tuples.
-- Attribute: ignore
Number of times to ignore a *note Breakpoint: 4203.
-- Attribute: hits
Count of the number of times a *note Breakpoint: 4203. has
been hit.
-- Class: bdb.Bdb (skip=None)
The *note Bdb: 12f5. class acts as a generic Python debugger base
class.
This class takes care of the details of the trace facility; a
derived class should implement user interaction. The standard
debugger class (*note pdb.Pdb: 921.) is an example.
The 'skip' argument, if given, must be an iterable of glob-style
module name patterns. The debugger will not step into frames that
originate in a module that matches one of these patterns. Whether
a frame is considered to originate in a certain module is
determined by the ‘__name__’ in the frame globals.
Changed in version 3.1: Added the 'skip' parameter.
The following methods of *note Bdb: 12f5. normally don’t need to be
overridden.
-- Method: canonic (filename)
Return canonical form of 'filename'.
For real file names, the canonical form is an
operating-system-dependent, *note case-normalized: 27e3. *note
absolute path: 130e. A 'filename' with angle brackets, such
as ‘""’ generated in interactive mode, is returned
unchanged.
-- Method: reset ()
Set the ‘botframe’, ‘stopframe’, ‘returnframe’ and *note
quitting: 4215. attributes with values ready to start
debugging.
-- Method: trace_dispatch (frame, event, arg)
This function is installed as the trace function of debugged
frames. Its return value is the new trace function (in most
cases, that is, itself).
The default implementation decides how to dispatch a frame,
depending on the type of event (passed as a string) that is
about to be executed. 'event' can be one of the following:
* ‘"line"’: A new line of code is going to be executed.
* ‘"call"’: A function is about to be called, or another
code block entered.
* ‘"return"’: A function or other code block is about to
return.
* ‘"exception"’: An exception has occurred.
* ‘"c_call"’: A C function is about to be called.
* ‘"c_return"’: A C function has returned.
* ‘"c_exception"’: A C function has raised an exception.
For the Python events, specialized functions (see below) are
called. For the C events, no action is taken.
The 'arg' parameter depends on the previous event.
See the documentation for *note sys.settrace(): 14ca. for more
information on the trace function. For more information on
code and frame objects, refer to *note The standard type
hierarchy: 1ee7.
-- Method: dispatch_line (frame)
If the debugger should stop on the current line, invoke the
*note user_line(): 4218. method (which should be overridden in
subclasses). Raise a *note BdbQuit: 4202. exception if the
*note quitting: 4215. flag is set (which can be set from *note
user_line(): 4218.). Return a reference to the *note
trace_dispatch(): 4216. method for further tracing in that
scope.
-- Method: dispatch_call (frame, arg)
If the debugger should stop on this function call, invoke the
*note user_call(): 421a. method (which should be overridden in
subclasses). Raise a *note BdbQuit: 4202. exception if the
*note quitting: 4215. flag is set (which can be set from *note
user_call(): 421a.). Return a reference to the *note
trace_dispatch(): 4216. method for further tracing in that
scope.
-- Method: dispatch_return (frame, arg)
If the debugger should stop on this function return, invoke
the *note user_return(): 421c. method (which should be
overridden in subclasses). Raise a *note BdbQuit: 4202.
exception if the *note quitting: 4215. flag is set (which can
be set from *note user_return(): 421c.). Return a reference
to the *note trace_dispatch(): 4216. method for further
tracing in that scope.
-- Method: dispatch_exception (frame, arg)
If the debugger should stop at this exception, invokes the
*note user_exception(): 421e. method (which should be
overridden in subclasses). Raise a *note BdbQuit: 4202.
exception if the *note quitting: 4215. flag is set (which can
be set from *note user_exception(): 421e.). Return a
reference to the *note trace_dispatch(): 4216. method for
further tracing in that scope.
Normally derived classes don’t override the following methods, but
they may if they want to redefine the definition of stopping and
breakpoints.
-- Method: is_skipped_line (module_name)
Return ‘True’ if 'module_name' matches any skip pattern.
-- Method: stop_here (frame)
Return ‘True’ if 'frame' is below the starting frame in the
stack.
-- Method: break_here (frame)
Return ‘True’ if there is an effective breakpoint for this
line.
Check whether a line or function breakpoint exists and is in
effect. Delete temporary breakpoints based on information
from *note effective(): 4222.
-- Method: break_anywhere (frame)
Return ‘True’ if any breakpoint exists for 'frame'’s filename.
Derived classes should override these methods to gain control over
debugger operation.
-- Method: user_call (frame, argument_list)
Called from *note dispatch_call(): 4219. if a break might stop
inside the called function.
'argument_list' is not used anymore and will always be ‘None’.
The argument is kept for backwards compatibility.
-- Method: user_line (frame)
Called from *note dispatch_line(): 4217. when either *note
stop_here(): 4220. or *note break_here(): 4221. returns
‘True’.
-- Method: user_return (frame, return_value)
Called from *note dispatch_return(): 421b. when *note
stop_here(): 4220. returns ‘True’.
-- Method: user_exception (frame, exc_info)
Called from *note dispatch_exception(): 421d. when *note
stop_here(): 4220. returns ‘True’.
-- Method: do_clear (arg)
Handle how a breakpoint must be removed when it is a temporary
one.
This method must be implemented by derived classes.
Derived classes and clients can call the following methods to
affect the stepping state.
-- Method: set_step ()
Stop after one line of code.
-- Method: set_next (frame)
Stop on the next line in or below the given frame.
-- Method: set_return (frame)
Stop when returning from the given frame.
-- Method: set_until (frame, lineno=None)
Stop when the line with the 'lineno' greater than the current
one is reached or when returning from current frame.
-- Method: set_trace ([frame])
Start debugging from 'frame'. If 'frame' is not specified,
debugging starts from caller’s frame.
Changed in version 3.13: *note set_trace(): 422a. will enter
the debugger immediately, rather than on the next line of code
to be executed.
-- Method: set_continue ()
Stop only at breakpoints or when finished. If there are no
breakpoints, set the system trace function to ‘None’.
-- Method: set_quit ()
Set the ‘quitting’ attribute to ‘True’. This raises *note
BdbQuit: 4202. in the next call to one of the ‘dispatch_*()’
methods.
Derived classes and clients can call the following methods to
manipulate breakpoints. These methods return a string containing
an error message if something went wrong, or ‘None’ if all is well.
-- Method: set_break (filename, lineno, temporary=False,
cond=None, funcname=None)
Set a new breakpoint. If the 'lineno' line doesn’t exist for
the 'filename' passed as argument, return an error message.
The 'filename' should be in canonical form, as described in
the *note canonic(): 4213. method.
-- Method: clear_break (filename, lineno)
Delete the breakpoints in 'filename' and 'lineno'. If none
were set, return an error message.
-- Method: clear_bpbynumber (arg)
Delete the breakpoint which has the index 'arg' in the *note
Breakpoint.bpbynumber: 4204. If 'arg' is not numeric or out
of range, return an error message.
-- Method: clear_all_file_breaks (filename)
Delete all breakpoints in 'filename'. If none were set,
return an error message.
-- Method: clear_all_breaks ()
Delete all existing breakpoints. If none were set, return an
error message.
-- Method: get_bpbynumber (arg)
Return a breakpoint specified by the given number. If 'arg'
is a string, it will be converted to a number. If 'arg' is a
non-numeric string, if the given breakpoint never existed or
has been deleted, a *note ValueError: 204. is raised.
Added in version 3.2.
-- Method: get_break (filename, lineno)
Return ‘True’ if there is a breakpoint for 'lineno' in
'filename'.
-- Method: get_breaks (filename, lineno)
Return all breakpoints for 'lineno' in 'filename', or an empty
list if none are set.
-- Method: get_file_breaks (filename)
Return all breakpoints in 'filename', or an empty list if none
are set.
-- Method: get_all_breaks ()
Return all breakpoints that are set.
Derived classes and clients can call the following methods to get a
data structure representing a stack trace.
-- Method: get_stack (f, t)
Return a list of (frame, lineno) tuples in a stack trace, and
a size.
The most recently called frame is last in the list. The size
is the number of frames below the frame where the debugger was
invoked.
-- Method: format_stack_entry (frame_lineno, lprefix=': ')
Return a string with information about a stack entry, which is
a ‘(frame, lineno)’ tuple. The return string contains:
* The canonical filename which contains the frame.
* The function name or ‘""’.
* The input arguments.
* The return value.
* The line of code (if it exists).
The following two methods can be called by clients to use a
debugger to debug a *note statement: 278b, given as a string.
-- Method: run (cmd, globals=None, locals=None)
Debug a statement executed via the *note exec(): 17f.
function. 'globals' defaults to ‘__main__.__dict__’, 'locals'
defaults to 'globals'.
-- Method: runeval (expr, globals=None, locals=None)
Debug an expression executed via the *note eval(): 180.
function. 'globals' and 'locals' have the same meaning as in
*note run(): 4238.
-- Method: runctx (cmd, globals, locals)
For backwards compatibility. Calls the *note run(): 4238.
method.
-- Method: runcall (func, /, *args, **kwds)
Debug a single function call, and return its result.
Finally, the module defines the following functions:
-- Function: bdb.checkfuncname (b, frame)
Return ‘True’ if we should break here, depending on the way the
*note Breakpoint: 4203. 'b' was set.
If it was set via line number, it checks if *note b.line: 420f. is
the same as the one in 'frame'. If the breakpoint was set via
*note function name: 4207, we have to check we are in the right
'frame' (the right function) and if we are on its first executable
line.
-- Function: bdb.effective (file, line, frame)
Return ‘(active breakpoint, delete temporary flag)’ or ‘(None,
None)’ as the breakpoint to act upon.
The 'active breakpoint' is the first entry in *note bplist: 4205.
for the (*note file: 4206, *note line: 420f.) (which must exist)
that is *note enabled: 4211, for which *note checkfuncname(): 423b.
is true, and that has neither a false *note condition: 4209. nor
positive *note ignore: 4212. count. The 'flag', meaning that a
temporary breakpoint should be deleted, is ‘False’ only when the
*note cond: 4209. cannot be evaluated (in which case, *note ignore:
4212. count is ignored).
If no such entry exists, then ‘(None, None)’ is returned.
-- Function: bdb.set_trace ()
Start debugging with a *note Bdb: 12f5. instance from caller’s
frame.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/bdb.py
File: python.info, Node: faulthandler — Dump the Python traceback, Next: pdb — The Python Debugger, Prev: bdb — Debugger framework, Up: Debugging and Profiling
5.28.3 ‘faulthandler’ — Dump the Python traceback
-------------------------------------------------
Added in version 3.3.
__________________________________________________________________
This module contains functions to dump Python tracebacks explicitly, on
a fault, after a timeout, or on a user signal. Call *note
faulthandler.enable(): c9e. to install fault handlers for the *note
SIGSEGV: 1d49, *note SIGFPE: 1d4a, *note SIGABRT: 1d4b, *note SIGBUS:
1d4c, and *note SIGILL: 1d4d. signals. You can also enable them at
startup by setting the *note PYTHONFAULTHANDLER: 1079. environment
variable or by using the *note -X: 176. ‘faulthandler’ command line
option.
The fault handler is compatible with system fault handlers like Apport
or the Windows fault handler. The module uses an alternative stack for
signal handlers if the ‘sigaltstack()’ function is available. This
allows it to dump the traceback even on a stack overflow.
The fault handler is called on catastrophic cases and therefore can only
use signal-safe functions (e.g. it cannot allocate memory on the heap).
Because of this limitation traceback dumping is minimal compared to
normal Python tracebacks:
* Only ASCII is supported. The ‘backslashreplace’ error handler is
used on encoding.
* Each string is limited to 500 characters.
* Only the filename, the function name and the line number are
displayed. (no source code)
* It is limited to 100 frames and 100 threads.
* The order is reversed: the most recent call is shown first.
By default, the Python traceback is written to *note sys.stderr: 939.
To see tracebacks, applications must be run in the terminal. A log file
can alternatively be passed to *note faulthandler.enable(): c9e.
The module is implemented in C, so tracebacks can be dumped on a crash
or when Python is deadlocked.
The *note Python Development Mode: 1fa. calls *note
faulthandler.enable(): c9e. at Python startup.
See also
........
Module *note pdb: a5.
Interactive source code debugger for Python programs.
Module *note traceback: fe.
Standard interface to extract, format and print stack traces of
Python programs.
* Menu:
* Dumping the traceback::
* Fault handler state::
* Dumping the tracebacks after a timeout::
* Dumping the traceback on a user signal::
* Issue with file descriptors::
* Example: Example<13>.
File: python.info, Node: Dumping the traceback, Next: Fault handler state, Up: faulthandler — Dump the Python traceback
5.28.3.1 Dumping the traceback
..............................
-- Function: faulthandler.dump_traceback (file=sys.stderr,
all_threads=True)
Dump the tracebacks of all threads into 'file'. If 'all_threads'
is ‘False’, dump only the current thread.
See also
........
*note traceback.print_tb(): e80, which can be used to print a
traceback object.
Changed in version 3.5: Added support for passing file descriptor
to this function.
File: python.info, Node: Fault handler state, Next: Dumping the tracebacks after a timeout, Prev: Dumping the traceback, Up: faulthandler — Dump the Python traceback
5.28.3.2 Fault handler state
............................
-- Function: faulthandler.enable (file=sys.stderr, all_threads=True)
Enable the fault handler: install handlers for the *note SIGSEGV:
1d49, *note SIGFPE: 1d4a, *note SIGABRT: 1d4b, *note SIGBUS: 1d4c.
and *note SIGILL: 1d4d. signals to dump the Python traceback. If
'all_threads' is ‘True’, produce tracebacks for every running
thread. Otherwise, dump only the current thread.
The 'file' must be kept open until the fault handler is disabled:
see *note issue with file descriptors: 4240.
Changed in version 3.5: Added support for passing file descriptor
to this function.
Changed in version 3.6: On Windows, a handler for Windows exception
is also installed.
Changed in version 3.10: The dump now mentions if a garbage
collector collection is running if 'all_threads' is true.
-- Function: faulthandler.disable ()
Disable the fault handler: uninstall the signal handlers installed
by *note enable(): c9e.
-- Function: faulthandler.is_enabled ()
Check if the fault handler is enabled.
File: python.info, Node: Dumping the tracebacks after a timeout, Next: Dumping the traceback on a user signal, Prev: Fault handler state, Up: faulthandler — Dump the Python traceback
5.28.3.3 Dumping the tracebacks after a timeout
...............................................
-- Function: faulthandler.dump_traceback_later (timeout, repeat=False,
file=sys.stderr, exit=False)
Dump the tracebacks of all threads, after a timeout of 'timeout'
seconds, or every 'timeout' seconds if 'repeat' is ‘True’. If
'exit' is ‘True’, call ‘_exit()’ with status=1 after dumping the
tracebacks. (Note ‘_exit()’ exits the process immediately, which
means it doesn’t do any cleanup like flushing file buffers.) If
the function is called twice, the new call replaces previous
parameters and resets the timeout. The timer has a sub-second
resolution.
The 'file' must be kept open until the traceback is dumped or *note
cancel_dump_traceback_later(): 4244. is called: see *note issue
with file descriptors: 4240.
This function is implemented using a watchdog thread.
Changed in version 3.5: Added support for passing file descriptor
to this function.
Changed in version 3.7: This function is now always available.
-- Function: faulthandler.cancel_dump_traceback_later ()
Cancel the last call to *note dump_traceback_later(): dfe.
File: python.info, Node: Dumping the traceback on a user signal, Next: Issue with file descriptors, Prev: Dumping the tracebacks after a timeout, Up: faulthandler — Dump the Python traceback
5.28.3.4 Dumping the traceback on a user signal
...............................................
-- Function: faulthandler.register (signum, file=sys.stderr,
all_threads=True, chain=False)
Register a user signal: install a handler for the 'signum' signal
to dump the traceback of all threads, or of the current thread if
'all_threads' is ‘False’, into 'file'. Call the previous handler
if chain is ‘True’.
The 'file' must be kept open until the signal is unregistered by
*note unregister(): 4246.: see *note issue with file descriptors:
4240.
Not available on Windows.
Changed in version 3.5: Added support for passing file descriptor
to this function.
-- Function: faulthandler.unregister (signum)
Unregister a user signal: uninstall the handler of the 'signum'
signal installed by *note register(): dfc. Return ‘True’ if the
signal was registered, ‘False’ otherwise.
Not available on Windows.
File: python.info, Node: Issue with file descriptors, Next: Example<13>, Prev: Dumping the traceback on a user signal, Up: faulthandler — Dump the Python traceback
5.28.3.5 Issue with file descriptors
....................................
*note enable(): c9e, *note dump_traceback_later(): dfe. and *note
register(): dfc. keep the file descriptor of their 'file' argument. If
the file is closed and its file descriptor is reused by a new file, or
if *note os.dup2(): b63. is used to replace the file descriptor, the
traceback will be written into a different file. Call these functions
again each time that the file is replaced.
File: python.info, Node: Example<13>, Prev: Issue with file descriptors, Up: faulthandler — Dump the Python traceback
5.28.3.6 Example
................
Example of a segmentation fault on Linux with and without enabling the
fault handler:
$ python -c "import ctypes; ctypes.string_at(0)"
Segmentation fault
$ python -q -X faulthandler
>>> import ctypes
>>> ctypes.string_at(0)
Fatal Python error: Segmentation fault
Current thread 0x00007fb899f39700 (most recent call first):
File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at
File "", line 1 in
Segmentation fault
File: python.info, Node: pdb — The Python Debugger, Next: The Python Profilers, Prev: faulthandler — Dump the Python traceback, Up: Debugging and Profiling
5.28.4 ‘pdb’ — The Python Debugger
----------------------------------
'Source code:' Lib/pdb.py(1)
__________________________________________________________________
The module *note pdb: a5. defines an interactive source code debugger
for Python programs. It supports setting (conditional) breakpoints and
single stepping at the source line level, inspection of stack frames,
source code listing, and evaluation of arbitrary Python code in the
context of any stack frame. It also supports post-mortem debugging and
can be called under program control.
The debugger is extensible – it is actually defined as the class *note
Pdb: 921. This is currently undocumented but easily understood by
reading the source. The extension interface uses the modules *note bdb:
f. and *note cmd: 19.
See also
........
Module *note faulthandler: 58.
Used to dump Python tracebacks explicitly, on a fault, after a
timeout, or on a user signal.
Module *note traceback: fe.
Standard interface to extract, format and print stack traces of
Python programs.
The typical usage to break into the debugger is to insert:
import pdb; pdb.set_trace()
Or:
breakpoint()
at the location you want to break into the debugger, and then run the
program. You can then step through the code following this statement,
and continue running without the debugger using the *note continue:
424c. command.
Changed in version 3.7: The built-in *note breakpoint(): 236, when
called with defaults, can be used instead of ‘import pdb;
pdb.set_trace()’.
def double(x):
breakpoint()
return x * 2
val = 3
print(f"{val} * 2 is {double(val)}")
The debugger’s prompt is ‘(Pdb)’, which is the indicator that you are in
debug mode:
> ...(2)double()
-> breakpoint()
(Pdb) p x
3
(Pdb) continue
3 * 2 is 6
Changed in version 3.3: Tab-completion via the *note readline: ba.
module is available for commands and command arguments, e.g. the
current global and local names are offered as arguments of the ‘p’
command.
You can also invoke *note pdb: a5. from the command line to debug other
scripts. For example:
python -m pdb [-c command] (-m module | pyfile) [args ...]
When invoked as a module, pdb will automatically enter post-mortem
debugging if the program being debugged exits abnormally. After
post-mortem debugging (or after normal exit of the program), pdb will
restart the program. Automatic restarting preserves pdb’s state (such
as breakpoints) and in most cases is more useful than quitting the
debugger upon program’s exit.
-- Option: -c, --command
To execute commands as if given in a ‘.pdbrc’ file; see *note
Debugger Commands: 424f.
Changed in version 3.2: Added the ‘-c’ option.
-- Option: -m
To execute modules similar to the way ‘python -m’ does. As with a
script, the debugger will pause execution just before the first
line of the module.
Changed in version 3.7: Added the ‘-m’ option.
Typical usage to execute a statement under control of the debugger is:
>>> import pdb
>>> def f(x):
... print(1 / x)
>>> pdb.run("f(2)")
> (1)()
(Pdb) continue
0.5
>>>
The typical usage to inspect a crashed program is:
>>> import pdb
>>> def f(x):
... print(1 / x)
...
>>> f(0)
Traceback (most recent call last):
File "", line 1, in
File "", line 2, in f
ZeroDivisionError: division by zero
>>> pdb.pm()
> (2)f()
(Pdb) p x
0
(Pdb)
Changed in version 3.13: The implementation of PEP 667(2) means that
name assignments made via ‘pdb’ will immediately affect the active
scope, even when running inside an *note optimized scope: 17e.
The module defines the following functions; each enters the debugger in
a slightly different way:
-- Function: pdb.run (statement, globals=None, locals=None)
Execute the 'statement' (given as a string or a code object) under
debugger control. The debugger prompt appears before any code is
executed; you can set breakpoints and type *note continue: 424c, or
you can step through the statement using *note step: 4251. or *note
next: 4252. (all these commands are explained below). The optional
'globals' and 'locals' arguments specify the environment in which
the code is executed; by default the dictionary of the module *note
__main__: 1. is used. (See the explanation of the built-in *note
exec(): 17f. or *note eval(): 180. functions.)
-- Function: pdb.runeval (expression, globals=None, locals=None)
Evaluate the 'expression' (given as a string or a code object)
under debugger control. When *note runeval(): 4253. returns, it
returns the value of the 'expression'. Otherwise this function is
similar to *note run(): 4056.
-- Function: pdb.runcall (function, *args, **kwds)
Call the 'function' (a function or method object, not a string)
with the given arguments. When *note runcall(): 4254. returns, it
returns whatever the function call returned. The debugger prompt
appears as soon as the function is entered.
-- Function: pdb.set_trace (*, header=None)
Enter the debugger at the calling stack frame. This is useful to
hard-code a breakpoint at a given point in a program, even if the
code is not otherwise being debugged (e.g. when an assertion
fails). If given, 'header' is printed to the console just before
debugging begins.
Changed in version 3.7: The keyword-only argument 'header'.
Changed in version 3.13: *note set_trace(): 237. will enter the
debugger immediately, rather than on the next line of code to be
executed.
-- Function: pdb.post_mortem (t=None)
Enter post-mortem debugging of the given exception or *note
traceback object: af4. If no value is given, it uses the exception
that is currently being handled, or raises ‘ValueError’ if there
isn’t one.
Changed in version 3.13: Support for exception objects was added.
-- Function: pdb.pm ()
Enter post-mortem debugging of the exception found in *note
sys.last_exc: 4ba.
The ‘run*’ functions and *note set_trace(): 237. are aliases for
instantiating the *note Pdb: 921. class and calling the method of the
same name. If you want to access further features, you have to do this
yourself:
-- Class: pdb.Pdb (completekey='tab', stdin=None, stdout=None,
skip=None, nosigint=False, readrc=True)
*note Pdb: 921. is the debugger class.
The 'completekey', 'stdin' and 'stdout' arguments are passed to the
underlying *note cmd.Cmd: 16c5. class; see the description there.
The 'skip' argument, if given, must be an iterable of glob-style
module name patterns. The debugger will not step into frames that
originate in a module that matches one of these patterns. (3)
By default, Pdb sets a handler for the SIGINT signal (which is sent
when the user presses ‘Ctrl’-‘C’ on the console) when you give a
*note continue: 424c. command. This allows you to break into the
debugger again by pressing ‘Ctrl’-‘C’. If you want Pdb not to
touch the SIGINT handler, set 'nosigint' to true.
The 'readrc' argument defaults to true and controls whether Pdb
will load .pdbrc files from the filesystem.
Example call to enable tracing with 'skip':
import pdb; pdb.Pdb(skip=['django.*']).set_trace()
Raises an *note auditing event: 18ba. ‘pdb.Pdb’ with no arguments.
Changed in version 3.1: Added the 'skip' parameter.
Changed in version 3.2: Added the 'nosigint' parameter.
Previously, a SIGINT handler was never set by Pdb.
Changed in version 3.6: The 'readrc' argument.
-- Method: run (statement, globals=None, locals=None)
-- Method: runeval (expression, globals=None, locals=None)
-- Method: runcall (function, *args, **kwds)
-- Method: set_trace ()
See the documentation for the functions explained above.
* Menu:
* Debugger Commands::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/pdb.py
(2) https://peps.python.org/pep-0667/
(3) Whether a frame is considered to originate in a certain module is
determined by the ‘__name__’ in the frame globals.
File: python.info, Node: Debugger Commands, Up: pdb — The Python Debugger
5.28.4.1 Debugger Commands
..........................
The commands recognized by the debugger are listed below. Most commands
can be abbreviated to one or two letters as indicated; e.g. ‘h(elp)’
means that either ‘h’ or ‘help’ can be used to enter the help command
(but not ‘he’ or ‘hel’, nor ‘H’ or ‘Help’ or ‘HELP’). Arguments to
commands must be separated by whitespace (spaces or tabs). Optional
arguments are enclosed in square brackets (‘[]’) in the command syntax;
the square brackets must not be typed. Alternatives in the command
syntax are separated by a vertical bar (‘|’).
Entering a blank line repeats the last command entered. Exception: if
the last command was a *note list: 425a. command, the next 11 lines are
listed.
Commands that the debugger doesn’t recognize are assumed to be Python
statements and are executed in the context of the program being
debugged. Python statements can also be prefixed with an exclamation
point (‘!’). This is a powerful way to inspect the program being
debugged; it is even possible to change a variable or call a function.
When an exception occurs in such a statement, the exception name is
printed but the debugger’s state is not changed.
Changed in version 3.13: Expressions/Statements whose prefix is a pdb
command are now correctly identified and executed.
The debugger supports *note aliases: 425b. Aliases can have parameters
which allows one a certain level of adaptability to the context under
examination.
Multiple commands may be entered on a single line, separated by ‘;;’.
(A single ‘;’ is not used as it is the separator for multiple commands
in a line that is passed to the Python parser.) No intelligence is
applied to separating the commands; the input is split at the first ‘;;’
pair, even if it is in the middle of a quoted string. A workaround for
strings with double semicolons is to use implicit string concatenation
‘';'';'’ or ‘";"";"’.
To set a temporary global variable, use a 'convenience variable'. A
'convenience variable' is a variable whose name starts with ‘$’. For
example, ‘$foo = 1’ sets a global variable ‘$foo’ which you can use in
the debugger session. The 'convenience variables' are cleared when the
program resumes execution so it’s less likely to interfere with your
program compared to using normal variables like ‘foo = 1’.
There are three preset 'convenience variables':
* ‘$_frame’: the current frame you are debugging
* ‘$_retval’: the return value if the frame is returning
* ‘$_exception’: the exception if the frame is raising an exception
Added in version 3.12: Added the 'convenience variable' feature.
If a file ‘.pdbrc’ exists in the user’s home directory or in the current
directory, it is read with ‘'utf-8'’ encoding and executed as if it had
been typed at the debugger prompt, with the exception that empty lines
and lines starting with ‘#’ are ignored. This is particularly useful
for aliases. If both files exist, the one in the home directory is read
first and aliases defined there can be overridden by the local file.
Changed in version 3.2: ‘.pdbrc’ can now contain commands that continue
debugging, such as *note continue: 424c. or *note next: 4252.
Previously, these commands had no effect.
Changed in version 3.11: ‘.pdbrc’ is now read with ‘'utf-8'’ encoding.
Previously, it was read with the system locale encoding.
-- Pdbcommand: h(elp) [command]
Without argument, print the list of available commands. With a
'command' as argument, print help about that command. ‘help pdb’
displays the full documentation (the docstring of the *note pdb:
a5. module). Since the 'command' argument must be an identifier,
‘help exec’ must be entered to get help on the ‘!’ command.
-- Pdbcommand: w(here)
Print a stack trace, with the most recent frame at the bottom. An
arrow (‘>’) indicates the current frame, which determines the
context of most commands.
-- Pdbcommand: d(own) [count]
Move the current frame 'count' (default one) levels down in the
stack trace (to a newer frame).
-- Pdbcommand: u(p) [count]
Move the current frame 'count' (default one) levels up in the stack
trace (to an older frame).
-- Pdbcommand: b(reak) [([filename:]lineno | function) [, condition]]
With a 'lineno' argument, set a break at line 'lineno' in the
current file. The line number may be prefixed with a 'filename'
and a colon, to specify a breakpoint in another file (possibly one
that hasn’t been loaded yet). The file is searched on *note
sys.path: 3b0. Accepatable forms of 'filename' are
‘/abspath/to/file.py’, ‘relpath/file.py’, ‘module’ and
‘package.module’.
With a 'function' argument, set a break at the first executable
statement within that function. 'function' can be any expression
that evaluates to a function in the current namespace.
If a second argument is present, it is an expression which must
evaluate to true before the breakpoint is honored.
Without argument, list all breaks, including for each breakpoint,
the number of times that breakpoint has been hit, the current
ignore count, and the associated condition if any.
Each breakpoint is assigned a number to which all the other
breakpoint commands refer.
-- Pdbcommand: tbreak [([filename:]lineno | function) [, condition]]
Temporary breakpoint, which is removed automatically when it is
first hit. The arguments are the same as for *note break: 4260.
-- Pdbcommand: cl(ear) [filename:lineno | bpnumber ...]
With a 'filename:lineno' argument, clear all the breakpoints at
this line. With a space separated list of breakpoint numbers,
clear those breakpoints. Without argument, clear all breaks (but
first ask confirmation).
-- Pdbcommand: disable bpnumber [bpnumber ...]
Disable the breakpoints given as a space separated list of
breakpoint numbers. Disabling a breakpoint means it cannot cause
the program to stop execution, but unlike clearing a breakpoint, it
remains in the list of breakpoints and can be (re-)enabled.
-- Pdbcommand: enable bpnumber [bpnumber ...]
Enable the breakpoints specified.
-- Pdbcommand: ignore bpnumber [count]
Set the ignore count for the given breakpoint number. If 'count'
is omitted, the ignore count is set to 0. A breakpoint becomes
active when the ignore count is zero. When non-zero, the 'count'
is decremented each time the breakpoint is reached and the
breakpoint is not disabled and any associated condition evaluates
to true.
-- Pdbcommand: condition bpnumber [condition]
Set a new 'condition' for the breakpoint, an expression which must
evaluate to true before the breakpoint is honored. If 'condition'
is absent, any existing condition is removed; i.e., the breakpoint
is made unconditional.
-- Pdbcommand: commands [bpnumber]
Specify a list of commands for breakpoint number 'bpnumber'. The
commands themselves appear on the following lines. Type a line
containing just ‘end’ to terminate the commands. An example:
(Pdb) commands 1
(com) p some_variable
(com) end
(Pdb)
To remove all commands from a breakpoint, type ‘commands’ and
follow it immediately with ‘end’; that is, give no commands.
With no 'bpnumber' argument, ‘commands’ refers to the last
breakpoint set.
You can use breakpoint commands to start your program up again.
Simply use the *note continue: 424c. command, or *note step: 4251,
or any other command that resumes execution.
Specifying any command resuming execution (currently *note
continue: 424c, *note step: 4251, *note next: 4252, *note return:
4268, *note jump: 4269, *note quit: 426a. and their abbreviations)
terminates the command list (as if that command was immediately
followed by end). This is because any time you resume execution
(even with a simple next or step), you may encounter another
breakpoint—which could have its own command list, leading to
ambiguities about which list to execute.
If you use the ‘silent’ command in the command list, the usual
message about stopping at a breakpoint is not printed. This may be
desirable for breakpoints that are to print a specific message and
then continue. If none of the other commands print anything, you
see no sign that the breakpoint was reached.
-- Pdbcommand: s(tep)
Execute the current line, stop at the first possible occasion
(either in a function that is called or on the next line in the
current function).
-- Pdbcommand: n(ext)
Continue execution until the next line in the current function is
reached or it returns. (The difference between *note next: 4252.
and *note step: 4251. is that *note step: 4251. stops inside a
called function, while *note next: 4252. executes called functions
at (nearly) full speed, only stopping at the next line in the
current function.)
-- Pdbcommand: unt(il) [lineno]
Without argument, continue execution until the line with a number
greater than the current one is reached.
With 'lineno', continue execution until a line with a number
greater or equal to 'lineno' is reached. In both cases, also stop
when the current frame returns.
Changed in version 3.2: Allow giving an explicit line number.
-- Pdbcommand: r(eturn)
Continue execution until the current function returns.
-- Pdbcommand: c(ont(inue))
Continue execution, only stop when a breakpoint is encountered.
-- Pdbcommand: j(ump) lineno
Set the next line that will be executed. Only available in the
bottom-most frame. This lets you jump back and execute code again,
or jump forward to skip code that you don’t want to run.
It should be noted that not all jumps are allowed – for instance it
is not possible to jump into the middle of a *note for: 2ec. loop
or out of a *note finally: 9ca. clause.
-- Pdbcommand: l(ist) [first[, last]]
List source code for the current file. Without arguments, list 11
lines around the current line or continue the previous listing.
With ‘.’ as argument, list 11 lines around the current line. With
one argument, list 11 lines around at that line. With two
arguments, list the given range; if the second argument is less
than the first, it is interpreted as a count.
The current line in the current frame is indicated by ‘->’. If an
exception is being debugged, the line where the exception was
originally raised or propagated is indicated by ‘>>’, if it differs
from the current line.
Changed in version 3.2: Added the ‘>>’ marker.
-- Pdbcommand: ll | longlist
List all source code for the current function or frame.
Interesting lines are marked as for *note list: 425a.
Added in version 3.2.
-- Pdbcommand: a(rgs)
Print the arguments of the current function and their current
values.
-- Pdbcommand: p expression
Evaluate 'expression' in the current context and print its value.
Note: ‘print()’ can also be used, but is not a debugger
command — this executes the Python *note print(): f70.
function.
-- Pdbcommand: pp expression
Like the *note p: f71. command, except the value of 'expression' is
pretty-printed using the *note pprint: ae. module.
-- Pdbcommand: whatis expression
Print the type of 'expression'.
-- Pdbcommand: source expression
Try to get source code of 'expression' and display it.
Added in version 3.2.
-- Pdbcommand: display [expression]
Display the value of 'expression' if it changed, each time
execution stops in the current frame.
Without 'expression', list all display expressions for the current
frame.
Note: Display evaluates 'expression' and compares to the
result of the previous evaluation of 'expression', so when the
result is mutable, display may not be able to pick up the
changes.
Example:
lst = []
breakpoint()
pass
lst.append(1)
print(lst)
Display won’t realize ‘lst’ has been changed because the result of
evaluation is modified in place by ‘lst.append(1)’ before being
compared:
> example.py(3)()
-> pass
(Pdb) display lst
display lst: []
(Pdb) n
> example.py(4)()
-> lst.append(1)
(Pdb) n
> example.py(5)()
-> print(lst)
(Pdb)
You can do some tricks with copy mechanism to make it work:
> example.py(3)()
-> pass
(Pdb) display lst[:]
display lst[:]: []
(Pdb) n
> example.py(4)()
-> lst.append(1)
(Pdb) n
> example.py(5)()
-> print(lst)
display lst[:]: [1] [old: []]
(Pdb)
Added in version 3.2.
-- Pdbcommand: undisplay [expression]
Do not display 'expression' anymore in the current frame. Without
'expression', clear all display expressions for the current frame.
Added in version 3.2.
-- Pdbcommand: interact
Start an interactive interpreter (using the *note code: 1a. module)
in a new global namespace initialised from the local and global
namespaces for the current scope. Use ‘exit()’ or ‘quit()’ to exit
the interpreter and return to the debugger.
Note: As ‘interact’ creates a new dedicated namespace for code
execution, assignments to variables will not affect the
original namespaces. However, modifications to any referenced
mutable objects will be reflected in the original namespaces
as usual.
Added in version 3.2.
Changed in version 3.13: ‘exit()’ and ‘quit()’ can be used to exit
the *note interact: 4273. command.
Changed in version 3.13: *note interact: 4273. directs its output
to the debugger’s output channel rather than *note sys.stderr: 939.
-- Pdbcommand: alias [name [command]]
Create an alias called 'name' that executes 'command'. The
'command' must 'not' be enclosed in quotes. Replaceable parameters
can be indicated by ‘%1’, ‘%2’, … and ‘%9’, while ‘%*’ is replaced
by all the parameters. If 'command' is omitted, the current alias
for 'name' is shown. If no arguments are given, all aliases are
listed.
Aliases may be nested and can contain anything that can be legally
typed at the pdb prompt. Note that internal pdb commands 'can' be
overridden by aliases. Such a command is then hidden until the
alias is removed. Aliasing is recursively applied to the first
word of the command line; all other words in the line are left
alone.
As an example, here are two useful aliases (especially when placed
in the ‘.pdbrc’ file):
# Print instance variables (usage "pi classInst")
alias pi for k in %1.__dict__.keys(): print(f"%1.{k} = {%1.__dict__[k]}")
# Print instance variables in self
alias ps pi self
-- Pdbcommand: unalias name
Delete the specified alias 'name'.
-- Pdbcommand: ! statement
Execute the (one-line) 'statement' in the context of the current
stack frame. The exclamation point can be omitted unless the first
word of the statement resembles a debugger command, e.g.:
(Pdb) ! n=42
(Pdb)
To set a global variable, you can prefix the assignment command
with a *note global: 18a. statement on the same line, e.g.:
(Pdb) global list_options; list_options = ['-l']
(Pdb)
-- Pdbcommand: run [args ...]
-- Pdbcommand: restart [args ...]
Restart the debugged Python program. If 'args' is supplied, it is
split with *note shlex: c4. and the result is used as the new *note
sys.argv: 1258. History, breakpoints, actions and debugger options
are preserved. *note restart: 4278. is an alias for *note run:
4277.
-- Pdbcommand: q(uit)
Quit from the debugger. The program being executed is aborted.
-- Pdbcommand: debug code
Enter a recursive debugger that steps through 'code' (which is an
arbitrary expression or statement to be executed in the current
environment).
-- Pdbcommand: retval
Print the return value for the last return of the current function.
-- Pdbcommand: exceptions [excnumber]
List or jump between chained exceptions.
When using ‘pdb.pm()’ or ‘Pdb.post_mortem(...)’ with a chained
exception instead of a traceback, it allows the user to move
between the chained exceptions using ‘exceptions’ command to list
exceptions, and ‘exceptions ’ to switch to that exception.
Example:
def out():
try:
middle()
except Exception as e:
raise ValueError("reraise middle() error") from e
def middle():
try:
return inner(0)
except Exception as e:
raise ValueError("Middle fail")
def inner(x):
1 / x
out()
calling ‘pdb.pm()’ will allow to move between exceptions:
> example.py(5)out()
-> raise ValueError("reraise middle() error") from e
(Pdb) exceptions
0 ZeroDivisionError('division by zero')
1 ValueError('Middle fail')
> 2 ValueError('reraise middle() error')
(Pdb) exceptions 0
> example.py(16)inner()
-> 1 / x
(Pdb) up
> example.py(10)middle()
-> return inner(0)
Added in version 3.13.
File: python.info, Node: The Python Profilers, Next: timeit — Measure execution time of small code snippets, Prev: pdb — The Python Debugger, Up: Debugging and Profiling
5.28.5 The Python Profilers
---------------------------
'Source code:' Lib/profile.py(1) and Lib/pstats.py(2)
__________________________________________________________________
* Menu:
* Introduction to the profilers::
* Instant User’s Manual::
* profile and cProfile Module Reference::
* The Stats Class::
* What Is Deterministic Profiling?::
* Limitations::
* Calibration::
* Using a custom timer::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/profile.py
(2) https://github.com/python/cpython/tree/3.13/Lib/pstats.py
File: python.info, Node: Introduction to the profilers, Next: Instant User’s Manual, Up: The Python Profilers
5.28.5.1 Introduction to the profilers
......................................
*note cProfile: 27. and *note profile: af. provide 'deterministic
profiling' of Python programs. A 'profile' is a set of statistics that
describes how often and for how long various parts of the program
executed. These statistics can be formatted into reports via the *note
pstats: b0. module.
The Python standard library provides two different implementations of
the same profiling interface:
1. *note cProfile: 27. is recommended for most users; it’s a C
extension with reasonable overhead that makes it suitable for
profiling long-running programs. Based on ‘lsprof’, contributed by
Brett Rosen and Ted Czotter.
2. *note profile: af, a pure Python module whose interface is imitated
by *note cProfile: 27, but which adds significant overhead to
profiled programs. If you’re trying to extend the profiler in some
way, the task might be easier with this module. Originally
designed and written by Jim Roskind.
Note: The profiler modules are designed to provide an execution
profile for a given program, not for benchmarking purposes (for
that, there is *note timeit: ef. for reasonably accurate results).
This particularly applies to benchmarking Python code against C
code: the profilers introduce overhead for Python code, but not for
C-level functions, and so the C code would seem faster than any
Python one.
File: python.info, Node: Instant User’s Manual, Next: profile and cProfile Module Reference, Prev: Introduction to the profilers, Up: The Python Profilers
5.28.5.2 Instant User’s Manual
..............................
This section is provided for users that “don’t want to read the manual.”
It provides a very brief overview, and allows a user to rapidly perform
profiling on an existing application.
To profile a function that takes a single argument, you can do:
import cProfile
import re
cProfile.run('re.compile("foo|bar")')
(Use *note profile: af. instead of *note cProfile: 27. if the latter is
not available on your system.)
The above action would run *note re.compile(): bd3. and print profile
results like the following:
214 function calls (207 primitive calls) in 0.002 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.002 0.002 {built-in method builtins.exec}
1 0.000 0.000 0.001 0.001 :1()
1 0.000 0.000 0.001 0.001 __init__.py:250(compile)
1 0.000 0.000 0.001 0.001 __init__.py:289(_compile)
1 0.000 0.000 0.000 0.000 _compiler.py:759(compile)
1 0.000 0.000 0.000 0.000 _parser.py:937(parse)
1 0.000 0.000 0.000 0.000 _compiler.py:598(_code)
1 0.000 0.000 0.000 0.000 _parser.py:435(_parse_sub)
The first line indicates that 214 calls were monitored. Of those calls,
207 were 'primitive', meaning that the call was not induced via
recursion. The next line: ‘Ordered by: cumulative time’ indicates the
output is sorted by the ‘cumtime’ values. The column headings include:
ncalls
for the number of calls.
tottime
for the total time spent in the given function (and excluding time
made in calls to sub-functions)
percall
is the quotient of ‘tottime’ divided by ‘ncalls’
cumtime
is the cumulative time spent in this and all subfunctions (from
invocation till exit). This figure is accurate 'even' for
recursive functions.
percall
is the quotient of ‘cumtime’ divided by primitive calls
filename:lineno(function)
provides the respective data of each function
When there are two numbers in the first column (for example ‘3/1’), it
means that the function recursed. The second value is the number of
primitive calls and the former is the total number of calls. Note that
when the function does not recurse, these two values are the same, and
only the single figure is printed.
Instead of printing the output at the end of the profile run, you can
save the results to a file by specifying a filename to the ‘run()’
function:
import cProfile
import re
cProfile.run('re.compile("foo|bar")', 'restats')
The *note pstats.Stats: 4282. class reads profile results from a file
and formats them in various ways. The files *note cProfile: 27. and
*note profile: af. can also be invoked as a script to profile another
script. For example:
python -m cProfile [-o output_file] [-s sort_order] (-m module | myscript.py)
-- Option: -o
Writes the profile results to a file instead of to stdout.
-- Option: -s
Specifies one of the *note sort_stats(): 4286. sort values to sort
the output by. This only applies when *note -o: 4284. is not
supplied.
-- Option: -m
Specifies that a module is being profiled instead of a script.
Added in version 3.7: Added the ‘-m’ option to *note cProfile: 27.
Added in version 3.8: Added the ‘-m’ option to *note profile: af.
The *note pstats: b0. module’s *note Stats: 4282. class has a variety of
methods for manipulating and printing the data saved into a profile
results file:
import pstats
from pstats import SortKey
p = pstats.Stats('restats')
p.strip_dirs().sort_stats(-1).print_stats()
The *note strip_dirs(): 4288. method removed the extraneous path from
all the module names. The *note sort_stats(): 4286. method sorted all
the entries according to the standard module/line/name string that is
printed. The *note print_stats(): 4289. method printed out all the
statistics. You might try the following sort calls:
p.sort_stats(SortKey.NAME)
p.print_stats()
The first call will actually sort the list by function name, and the
second call will print out the statistics. The following are some
interesting calls to experiment with:
p.sort_stats(SortKey.CUMULATIVE).print_stats(10)
This sorts the profile by cumulative time in a function, and then only
prints the ten most significant lines. If you want to understand what
algorithms are taking time, the above line is what you would use.
If you were looking to see what functions were looping a lot, and taking
a lot of time, you would do:
p.sort_stats(SortKey.TIME).print_stats(10)
to sort according to time spent within each function, and then print the
statistics for the top ten functions.
You might also try:
p.sort_stats(SortKey.FILENAME).print_stats('__init__')
This will sort all the statistics by file name, and then print out
statistics for only the class init methods (since they are spelled with
‘__init__’ in them). As one final example, you could try:
p.sort_stats(SortKey.TIME, SortKey.CUMULATIVE).print_stats(.5, 'init')
This line sorts statistics with a primary key of time, and a secondary
key of cumulative time, and then prints out some of the statistics. To
be specific, the list is first culled down to 50% (re: ‘.5’) of its
original size, then only lines containing ‘init’ are maintained, and
that sub-sub-list is printed.
If you wondered what functions called the above functions, you could now
(‘p’ is still sorted according to the last criteria) do:
p.print_callers(.5, 'init')
and you would get a list of callers for each of the listed functions.
If you want more functionality, you’re going to have to read the manual,
or guess what the following functions do:
p.print_callees()
p.add('restats')
Invoked as a script, the *note pstats: b0. module is a statistics
browser for reading and examining profile dumps. It has a simple
line-oriented interface (implemented using *note cmd: 19.) and
interactive help.
File: python.info, Node: profile and cProfile Module Reference, Next: The Stats Class, Prev: Instant User’s Manual, Up: The Python Profilers
5.28.5.3 ‘profile’ and ‘cProfile’ Module Reference
..................................................
Both the *note profile: af. and *note cProfile: 27. modules provide the
following functions:
-- Function: profile.run (command, filename=None, sort=-1)
This function takes a single argument that can be passed to the
*note exec(): 17f. function, and an optional file name. In all
cases this routine executes:
exec(command, __main__.__dict__, __main__.__dict__)
and gathers profiling statistics from the execution. If no file
name is present, then this function automatically creates a *note
Stats: 4282. instance and prints a simple profiling report. If the
sort value is specified, it is passed to this *note Stats: 4282.
instance to control how the results are sorted.
-- Function: profile.runctx (command, globals, locals, filename=None,
sort=-1)
This function is similar to *note run(): 428b, with added arguments
to supply the globals and locals mappings for the 'command' string.
This routine executes:
exec(command, globals, locals)
and gathers profiling statistics as in the *note run(): 428b.
function above.
-- Class: profile.Profile (timer=None, timeunit=0.0, subcalls=True,
builtins=True)
This class is normally only used if more precise control over
profiling is needed than what the ‘cProfile.run()’ function
provides.
A custom timer can be supplied for measuring how long code takes to
run via the 'timer' argument. This must be a function that returns
a single number representing the current time. If the number is an
integer, the 'timeunit' specifies a multiplier that specifies the
duration of each unit of time. For example, if the timer returns
times measured in thousands of seconds, the time unit would be
‘.001’.
Directly using the *note Profile: 9e2. class allows formatting
profile results without writing the profile data to a file:
import cProfile, pstats, io
from pstats import SortKey
pr = cProfile.Profile()
pr.enable()
# ... do something ...
pr.disable()
s = io.StringIO()
sortby = SortKey.CUMULATIVE
ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
ps.print_stats()
print(s.getvalue())
The *note Profile: 9e2. class can also be used as a context manager
(supported only in *note cProfile: 27. module. see *note Context
Manager Types: 1fbf.):
import cProfile
with cProfile.Profile() as pr:
# ... do something ...
pr.print_stats()
Changed in version 3.8: Added context manager support.
-- Method: enable ()
Start collecting profiling data. Only in *note cProfile: 27.
-- Method: disable ()
Stop collecting profiling data. Only in *note cProfile: 27.
-- Method: create_stats ()
Stop collecting profiling data and record the results
internally as the current profile.
-- Method: print_stats (sort=-1)
Create a *note Stats: 4282. object based on the current
profile and print the results to stdout.
The 'sort' parameter specifies the sorting order of the
displayed statistics. It accepts a single key or a tuple of
keys to enable multi-level sorting, as in *note
Stats.sort_stats: 4286.
Added in version 3.13: *note print_stats(): 4290. now accepts
a tuple of keys.
-- Method: dump_stats (filename)
Write the results of the current profile to 'filename'.
-- Method: run (cmd)
Profile the cmd via *note exec(): 17f.
-- Method: runctx (cmd, globals, locals)
Profile the cmd via *note exec(): 17f. with the specified
global and local environment.
-- Method: runcall (func, /, *args, **kwargs)
Profile ‘func(*args, **kwargs)’
Note that profiling will only work if the called command/function
actually returns. If the interpreter is terminated (e.g. via a *note
sys.exit(): 133c. call during the called command/function execution) no
profiling results will be printed.
File: python.info, Node: The Stats Class, Next: What Is Deterministic Profiling?, Prev: profile and cProfile Module Reference, Up: The Python Profilers
5.28.5.4 The ‘Stats’ Class
..........................
Analysis of the profiler data is done using the *note Stats: 4282.
class.
-- Class: pstats.Stats (*filenames or profile, stream=sys.stdout)
This class constructor creates an instance of a “statistics object”
from a 'filename' (or list of filenames) or from a ‘Profile’
instance. Output will be printed to the stream specified by
'stream'.
The file selected by the above constructor must have been created
by the corresponding version of *note profile: af. or *note
cProfile: 27. To be specific, there is 'no' file compatibility
guaranteed with future versions of this profiler, and there is no
compatibility with files produced by other profilers, or the same
profiler run on a different operating system. If several files are
provided, all the statistics for identical functions will be
coalesced, so that an overall view of several processes can be
considered in a single report. If additional files need to be
combined with data in an existing *note Stats: 4282. object, the
*note add(): 4296. method can be used.
Instead of reading the profile data from a file, a
‘cProfile.Profile’ or *note profile.Profile: 9e2. object can be
used as the profile data source.
*note Stats: 4282. objects have the following methods:
-- Method: strip_dirs ()
This method for the *note Stats: 4282. class removes all
leading path information from file names. It is very useful
in reducing the size of the printout to fit within (close to)
80 columns. This method modifies the object, and the stripped
information is lost. After performing a strip operation, the
object is considered to have its entries in a “random” order,
as it was just after object initialization and loading. If
*note strip_dirs(): 4288. causes two function names to be
indistinguishable (they are on the same line of the same
filename, and have the same function name), then the
statistics for these two entries are accumulated into a single
entry.
-- Method: add (*filenames)
This method of the *note Stats: 4282. class accumulates
additional profiling information into the current profiling
object. Its arguments should refer to filenames created by
the corresponding version of *note profile.run(): 428b. or
‘cProfile.run()’. Statistics for identically named (re: file,
line, name) functions are automatically accumulated into
single function statistics.
-- Method: dump_stats (filename)
Save the data loaded into the *note Stats: 4282. object to a
file named 'filename'. The file is created if it does not
exist, and is overwritten if it already exists. This is
equivalent to the method of the same name on the *note
profile.Profile: 9e2. and ‘cProfile.Profile’ classes.
-- Method: sort_stats (*keys)
This method modifies the *note Stats: 4282. object by sorting
it according to the supplied criteria. The argument can be
either a string or a SortKey enum identifying the basis of a
sort (example: ‘'time'’, ‘'name'’, ‘SortKey.TIME’ or
‘SortKey.NAME’). The SortKey enums argument have advantage
over the string argument in that it is more robust and less
error prone.
When more than one key is provided, then additional keys are
used as secondary criteria when there is equality in all keys
selected before them. For example, ‘sort_stats(SortKey.NAME,
SortKey.FILE)’ will sort all the entries according to their
function name, and resolve all ties (identical function names)
by sorting by file name.
For the string argument, abbreviations can be used for any key
names, as long as the abbreviation is unambiguous.
The following are the valid string and SortKey:
Valid String Arg Valid enum Arg Meaning
----------------------------------------------------------------------------
‘'calls'’ SortKey.CALLS call count
‘'cumulative'’ SortKey.CUMULATIVE cumulative time
‘'cumtime'’ N/A cumulative time
‘'file'’ N/A file name
‘'filename'’ SortKey.FILENAME file name
‘'module'’ N/A file name
‘'ncalls'’ N/A call count
‘'pcalls'’ SortKey.PCALLS primitive call count
‘'line'’ SortKey.LINE line number
‘'name'’ SortKey.NAME function name
‘'nfl'’ SortKey.NFL name/file/line
‘'stdname'’ SortKey.STDNAME standard name
‘'time'’ SortKey.TIME internal time
‘'tottime'’ N/A internal time
Note that all sorts on statistics are in descending order
(placing most time consuming items first), where as name,
file, and line number searches are in ascending order
(alphabetical). The subtle distinction between ‘SortKey.NFL’
and ‘SortKey.STDNAME’ is that the standard name is a sort of
the name as printed, which means that the embedded line
numbers get compared in an odd way. For example, lines 3, 20,
and 40 would (if the file names were the same) appear in the
string order 20, 3 and 40. In contrast, ‘SortKey.NFL’ does a
numeric compare of the line numbers. In fact,
‘sort_stats(SortKey.NFL)’ is the same as
‘sort_stats(SortKey.NAME, SortKey.FILENAME, SortKey.LINE)’.
For backward-compatibility reasons, the numeric arguments
‘-1’, ‘0’, ‘1’, and ‘2’ are permitted. They are interpreted
as ‘'stdname'’, ‘'calls'’, ‘'time'’, and ‘'cumulative'’
respectively. If this old style format (numeric) is used,
only one sort key (the numeric key) will be used, and
additional arguments will be silently ignored.
Added in version 3.7: Added the SortKey enum.
-- Method: reverse_order ()
This method for the *note Stats: 4282. class reverses the
ordering of the basic list within the object. Note that by
default ascending vs descending order is properly selected
based on the sort key of choice.
-- Method: print_stats (*restrictions)
This method for the *note Stats: 4282. class prints out a
report as described in the *note profile.run(): 428b.
definition.
The order of the printing is based on the last *note
sort_stats(): 4286. operation done on the object (subject to
caveats in *note add(): 4296. and *note strip_dirs(): 4288.).
The arguments provided (if any) can be used to limit the list
down to the significant entries. Initially, the list is taken
to be the complete set of profiled functions. Each
restriction is either an integer (to select a count of lines),
or a decimal fraction between 0.0 and 1.0 inclusive (to select
a percentage of lines), or a string that will interpreted as a
regular expression (to pattern match the standard name that is
printed). If several restrictions are provided, then they are
applied sequentially. For example:
print_stats(.1, 'foo:')
would first limit the printing to first 10% of list, and then
only print functions that were part of filename ‘.*foo:’. In
contrast, the command:
print_stats('foo:', .1)
would limit the list to all functions having file names
‘.*foo:’, and then proceed to only print the first 10% of
them.
-- Method: print_callers (*restrictions)
This method for the *note Stats: 4282. class prints a list of
all functions that called each function in the profiled
database. The ordering is identical to that provided by *note
print_stats(): 4289, and the definition of the restricting
argument is also identical. Each caller is reported on its
own line. The format differs slightly depending on the
profiler that produced the stats:
* With *note profile: af, a number is shown in parentheses
after each caller to show how many times this specific
call was made. For convenience, a second
non-parenthesized number repeats the cumulative time
spent in the function at the right.
* With *note cProfile: 27, each caller is preceded by three
numbers: the number of times this specific call was made,
and the total and cumulative times spent in the current
function while it was invoked by this specific caller.
-- Method: print_callees (*restrictions)
This method for the *note Stats: 4282. class prints a list of
all function that were called by the indicated function.
Aside from this reversal of direction of calls (re: called vs
was called by), the arguments and ordering are identical to
the *note print_callers(): 4299. method.
-- Method: get_stats_profile ()
This method returns an instance of StatsProfile, which
contains a mapping of function names to instances of
FunctionProfile. Each FunctionProfile instance holds
information related to the function’s profile such as how long
the function took to run, how many times it was called, etc…
Added in version 3.9: Added the following dataclasses:
StatsProfile, FunctionProfile. Added the following function:
get_stats_profile.
File: python.info, Node: What Is Deterministic Profiling?, Next: Limitations, Prev: The Stats Class, Up: The Python Profilers
5.28.5.5 What Is Deterministic Profiling?
.........................................
'Deterministic profiling' is meant to reflect the fact that all
'function call', 'function return', and 'exception' events are
monitored, and precise timings are made for the intervals between these
events (during which time the user’s code is executing). In contrast,
'statistical profiling' (which is not done by this module) randomly
samples the effective instruction pointer, and deduces where time is
being spent. The latter technique traditionally involves less overhead
(as the code does not need to be instrumented), but provides only
relative indications of where time is being spent.
In Python, since there is an interpreter active during execution, the
presence of instrumented code is not required in order to do
deterministic profiling. Python automatically provides a 'hook'
(optional callback) for each event. In addition, the interpreted nature
of Python tends to add so much overhead to execution, that deterministic
profiling tends to only add small processing overhead in typical
applications. The result is that deterministic profiling is not that
expensive, yet provides extensive run time statistics about the
execution of a Python program.
Call count statistics can be used to identify bugs in code (surprising
counts), and to identify possible inline-expansion points (high call
counts). Internal time statistics can be used to identify “hot loops”
that should be carefully optimized. Cumulative time statistics should
be used to identify high level errors in the selection of algorithms.
Note that the unusual handling of cumulative times in this profiler
allows statistics for recursive implementations of algorithms to be
directly compared to iterative implementations.
File: python.info, Node: Limitations, Next: Calibration, Prev: What Is Deterministic Profiling?, Up: The Python Profilers
5.28.5.6 Limitations
....................
One limitation has to do with accuracy of timing information. There is
a fundamental problem with deterministic profilers involving accuracy.
The most obvious restriction is that the underlying “clock” is only
ticking at a rate (typically) of about .001 seconds. Hence no
measurements will be more accurate than the underlying clock. If enough
measurements are taken, then the “error” will tend to average out.
Unfortunately, removing this first error induces a second source of
error.
The second problem is that it “takes a while” from when an event is
dispatched until the profiler’s call to get the time actually 'gets' the
state of the clock. Similarly, there is a certain lag when exiting the
profiler event handler from the time that the clock’s value was obtained
(and then squirreled away), until the user’s code is once again
executing. As a result, functions that are called many times, or call
many functions, will typically accumulate this error. The error that
accumulates in this fashion is typically less than the accuracy of the
clock (less than one clock tick), but it 'can' accumulate and become
very significant.
The problem is more important with *note profile: af. than with the
lower-overhead *note cProfile: 27. For this reason, *note profile: af.
provides a means of calibrating itself for a given platform so that this
error can be probabilistically (on the average) removed. After the
profiler is calibrated, it will be more accurate (in a least square
sense), but it will sometimes produce negative numbers (when call counts
are exceptionally low, and the gods of probability work against you :-).
) Do 'not' be alarmed by negative numbers in the profile. They should
'only' appear if you have calibrated your profiler, and the results are
actually better than without calibration.
File: python.info, Node: Calibration, Next: Using a custom timer, Prev: Limitations, Up: The Python Profilers
5.28.5.7 Calibration
....................
The profiler of the *note profile: af. module subtracts a constant from
each event handling time to compensate for the overhead of calling the
time function, and socking away the results. By default, the constant
is 0. The following procedure can be used to obtain a better constant
for a given platform (see *note Limitations: 429f.).
import profile
pr = profile.Profile()
for i in range(5):
print(pr.calibrate(10000))
The method executes the number of Python calls given by the argument,
directly and again under the profiler, measuring the time for both. It
then computes the hidden overhead per profiler event, and returns that
as a float. For example, on a 1.8Ghz Intel Core i5 running macOS, and
using Python’s time.process_time() as the timer, the magical number is
about 4.04e-6.
The object of this exercise is to get a fairly consistent result. If
your computer is 'very' fast, or your timer function has poor
resolution, you might have to pass 100000, or even 1000000, to get
consistent results.
When you have a consistent answer, there are three ways you can use it:
import profile
# 1. Apply computed bias to all Profile instances created hereafter.
profile.Profile.bias = your_computed_bias
# 2. Apply computed bias to a specific Profile instance.
pr = profile.Profile()
pr.bias = your_computed_bias
# 3. Specify computed bias in instance constructor.
pr = profile.Profile(bias=your_computed_bias)
If you have a choice, you are better off choosing a smaller constant,
and then your results will “less often” show up as negative in profile
statistics.
File: python.info, Node: Using a custom timer, Prev: Calibration, Up: The Python Profilers
5.28.5.8 Using a custom timer
.............................
If you want to change how current time is determined (for example, to
force use of wall-clock time or elapsed process time), pass the timing
function you want to the ‘Profile’ class constructor:
pr = profile.Profile(your_time_func)
The resulting profiler will then call ‘your_time_func’. Depending on
whether you are using *note profile.Profile: 9e2. or ‘cProfile.Profile’,
‘your_time_func’’s return value will be interpreted differently:
*note profile.Profile: 9e2.
‘your_time_func’ should return a single number, or a list of
numbers whose sum is the current time (like what *note os.times():
1109. returns). If the function returns a single time number, or
the list of returned numbers has length 2, then you will get an
especially fast version of the dispatch routine.
Be warned that you should calibrate the profiler class for the
timer function that you choose (see *note Calibration: 42a1.). For
most machines, a timer that returns a lone integer value will
provide the best results in terms of low overhead during profiling.
(*note os.times(): 1109. is 'pretty' bad, as it returns a tuple of
floating-point values). If you want to substitute a better timer
in the cleanest fashion, derive a class and hardwire a replacement
dispatch method that best handles your timer call, along with the
appropriate calibration constant.
‘cProfile.Profile’
‘your_time_func’ should return a single number. If it returns
integers, you can also invoke the class constructor with a second
argument specifying the real duration of one unit of time. For
example, if ‘your_integer_time_func’ returns times measured in
thousands of seconds, you would construct the ‘Profile’ instance as
follows:
pr = cProfile.Profile(your_integer_time_func, 0.001)
As the ‘cProfile.Profile’ class cannot be calibrated, custom timer
functions should be used with care and should be as fast as
possible. For the best results with a custom timer, it might be
necessary to hard-code it in the C source of the internal ‘_lsprof’
module.
Python 3.3 adds several new functions in *note time: ee. that can be
used to make precise measurements of process or wall-clock time. For
example, see *note time.perf_counter(): a88.
File: python.info, Node: timeit — Measure execution time of small code snippets, Next: trace — Trace or track Python statement execution, Prev: The Python Profilers, Up: Debugging and Profiling
5.28.6 ‘timeit’ — Measure execution time of small code snippets
---------------------------------------------------------------
'Source code:' Lib/timeit.py(1)
__________________________________________________________________
This module provides a simple way to time small bits of Python code. It
has both a *note Command-Line Interface: 42a6. as well as a *note
callable: 42a7. one. It avoids a number of common traps for measuring
execution times. See also Tim Peters’ introduction to the “Algorithms”
chapter in the second edition of 'Python Cookbook', published by
O’Reilly.
* Menu:
* Basic Examples: Basic Examples<2>.
* Python Interface::
* Command-Line Interface: Command-Line Interface<4>.
* Examples: Examples<30>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/timeit.py
File: python.info, Node: Basic Examples<2>, Next: Python Interface, Up: timeit — Measure execution time of small code snippets
5.28.6.1 Basic Examples
.......................
The following example shows how the *note Command-Line Interface: 42a6.
can be used to compare three different expressions:
$ python -m timeit "'-'.join(str(n) for n in range(100))"
10000 loops, best of 5: 30.2 usec per loop
$ python -m timeit "'-'.join([str(n) for n in range(100)])"
10000 loops, best of 5: 27.5 usec per loop
$ python -m timeit "'-'.join(map(str, range(100)))"
10000 loops, best of 5: 23.2 usec per loop
This can be achieved from the *note Python Interface: 42a7. with:
>>> import timeit
>>> timeit.timeit('"-".join(str(n) for n in range(100))', number=10000)
0.3018611848820001
>>> timeit.timeit('"-".join([str(n) for n in range(100)])', number=10000)
0.2727368790656328
>>> timeit.timeit('"-".join(map(str, range(100)))', number=10000)
0.23702679807320237
A callable can also be passed from the *note Python Interface: 42a7.:
>>> timeit.timeit(lambda: "-".join(map(str, range(100))), number=10000)
0.19665591977536678
Note however that *note timeit(): e79. will automatically determine the
number of repetitions only when the command-line interface is used. In
the *note Examples: 42a9. section you can find more advanced examples.
File: python.info, Node: Python Interface, Next: Command-Line Interface<4>, Prev: Basic Examples<2>, Up: timeit — Measure execution time of small code snippets
5.28.6.2 Python Interface
.........................
The module defines three convenience functions and a public class:
-- Function: timeit.timeit (stmt='pass', setup='pass', timer=, number=1000000, globals=None)
Create a *note Timer: 1491. instance with the given statement,
'setup' code and 'timer' function and run its *note timeit(): ced.
method with 'number' executions. The optional 'globals' argument
specifies a namespace in which to execute the code.
Changed in version 3.5: The optional 'globals' parameter was added.
-- Function: timeit.repeat (stmt='pass', setup='pass', timer=, repeat=5, number=1000000, globals=None)
Create a *note Timer: 1491. instance with the given statement,
'setup' code and 'timer' function and run its *note repeat(): 42ac.
method with the given 'repeat' count and 'number' executions. The
optional 'globals' argument specifies a namespace in which to
execute the code.
Changed in version 3.5: The optional 'globals' parameter was added.
Changed in version 3.7: Default value of 'repeat' changed from 3 to
5.
-- Function: timeit.default_timer ()
The default timer, which is always time.perf_counter(), returns
float seconds. An alternative, time.perf_counter_ns, returns
integer nanoseconds.
Changed in version 3.3: *note time.perf_counter(): a88. is now the
default timer.
-- Class: timeit.Timer (stmt='pass', setup='pass', timer=, globals=None)
Class for timing execution speed of small code snippets.
The constructor takes a statement to be timed, an additional
statement used for setup, and a timer function. Both statements
default to ‘'pass'’; the timer function is platform-dependent (see
the module doc string). 'stmt' and 'setup' may also contain
multiple statements separated by ‘;’ or newlines, as long as they
don’t contain multi-line string literals. The statement will by
default be executed within timeit’s namespace; this behavior can be
controlled by passing a namespace to 'globals'.
To measure the execution time of the first statement, use the *note
timeit(): ced. method. The *note repeat(): 42ac. and *note
autorange(): cec. methods are convenience methods to call *note
timeit(): ced. multiple times.
The execution time of 'setup' is excluded from the overall timed
execution run.
The 'stmt' and 'setup' parameters can also take objects that are
callable without arguments. This will embed calls to them in a
timer function that will then be executed by *note timeit(): ced.
Note that the timing overhead is a little larger in this case
because of the extra function calls.
Changed in version 3.5: The optional 'globals' parameter was added.
-- Method: timeit (number=1000000)
Time 'number' executions of the main statement. This executes
the setup statement once, and then returns the time it takes
to execute the main statement a number of times. The default
timer returns seconds as a float. The argument is the number
of times through the loop, defaulting to one million. The
main statement, the setup statement and the timer function to
be used are passed to the constructor.
Note: By default, *note timeit(): ced. temporarily turns
off *note garbage collection: 1919. during the timing.
The advantage of this approach is that it makes
independent timings more comparable. The disadvantage is
that GC may be an important component of the performance
of the function being measured. If so, GC can be
re-enabled as the first statement in the 'setup' string.
For example:
timeit.Timer('for i in range(10): oct(i)', 'gc.enable()').timeit()
-- Method: autorange (callback=None)
Automatically determine how many times to call *note timeit():
ced.
This is a convenience function that calls *note timeit(): ced.
repeatedly so that the total time >= 0.2 second, returning the
eventual (number of loops, time taken for that number of
loops). It calls *note timeit(): ced. with increasing numbers
from the sequence 1, 2, 5, 10, 20, 50, … until the time taken
is at least 0.2 seconds.
If 'callback' is given and is not ‘None’, it will be called
after each trial with two arguments: ‘callback(number,
time_taken)’.
Added in version 3.6.
-- Method: repeat (repeat=5, number=1000000)
Call *note timeit(): ced. a few times.
This is a convenience function that calls the *note timeit():
ced. repeatedly, returning a list of results. The first
argument specifies how many times to call *note timeit(): ced.
The second argument specifies the 'number' argument for *note
timeit(): ced.
Note: It’s tempting to calculate mean and standard
deviation from the result vector and report these.
However, this is not very useful. In a typical case, the
lowest value gives a lower bound for how fast your
machine can run the given code snippet; higher values in
the result vector are typically not caused by variability
in Python’s speed, but by other processes interfering
with your timing accuracy. So the *note min(): f03. of
the result is probably the only number you should be
interested in. After that, you should look at the entire
vector and apply common sense rather than statistics.
Changed in version 3.7: Default value of 'repeat' changed from
3 to 5.
-- Method: print_exc (file=None)
Helper to print a traceback from the timed code.
Typical use:
t = Timer(...) # outside the try/except
try:
t.timeit(...) # or t.repeat(...)
except Exception:
t.print_exc()
The advantage over the standard traceback is that source lines
in the compiled template will be displayed. The optional
'file' argument directs where the traceback is sent; it
defaults to *note sys.stderr: 939.
File: python.info, Node: Command-Line Interface<4>, Next: Examples<30>, Prev: Python Interface, Up: timeit — Measure execution time of small code snippets
5.28.6.3 Command-Line Interface
...............................
When called as a program from the command line, the following form is
used:
python -m timeit [-n N] [-r N] [-u U] [-s S] [-p] [-v] [-h] [statement ...]
Where the following options are understood:
-- Option: -n N, --number=N
how many times to execute ‘statement’
-- Option: -r N, --repeat=N
how many times to repeat the timer (default 5)
-- Option: -s S, --setup=S
statement to be executed once initially (default ‘pass’)
-- Option: -p, --process
measure process time, not wallclock time, using *note
time.process_time(): a89. instead of *note time.perf_counter():
a88, which is the default
Added in version 3.3.
-- Option: -u, --unit=U
specify a time unit for timer output; can select ‘nsec’, ‘usec’,
‘msec’, or ‘sec’
Added in version 3.5.
-- Option: -v, --verbose
print raw timing results; repeat for more digits precision
-- Option: -h, --help
print a short usage message and exit
A multi-line statement may be given by specifying each line as a
separate statement argument; indented lines are possible by enclosing an
argument in quotes and using leading spaces. Multiple *note -s: 42b4.
options are treated similarly.
If *note -n: 42b0. is not given, a suitable number of loops is
calculated by trying increasing numbers from the sequence 1, 2, 5, 10,
20, 50, … until the total time is at least 0.2 seconds.
*note default_timer(): 42ad. measurements can be affected by other
programs running on the same machine, so the best thing to do when
accurate timing is necessary is to repeat the timing a few times and use
the best time. The *note -r: 42b2. option is good for this; the default
of 5 repetitions is probably enough in most cases. You can use *note
time.process_time(): a89. to measure CPU time.
Note: There is a certain baseline overhead associated with
executing a pass statement. The code here doesn’t try to hide it,
but you should be aware of it. The baseline overhead can be
measured by invoking the program without arguments, and it might
differ between Python versions.
File: python.info, Node: Examples<30>, Prev: Command-Line Interface<4>, Up: timeit — Measure execution time of small code snippets
5.28.6.4 Examples
.................
It is possible to provide a setup statement that is executed only once
at the beginning:
$ python -m timeit -s "text = 'sample string'; char = 'g'" "char in text"
5000000 loops, best of 5: 0.0877 usec per loop
$ python -m timeit -s "text = 'sample string'; char = 'g'" "text.find(char)"
1000000 loops, best of 5: 0.342 usec per loop
In the output, there are three fields. The loop count, which tells you
how many times the statement body was run per timing loop repetition.
The repetition count (‘best of 5’) which tells you how many times the
timing loop was repeated, and finally the time the statement body took
on average within the best repetition of the timing loop. That is, the
time the fastest repetition took divided by the loop count.
>>> import timeit
>>> timeit.timeit('char in text', setup='text = "sample string"; char = "g"')
0.41440500499993504
>>> timeit.timeit('text.find(char)', setup='text = "sample string"; char = "g"')
1.7246671520006203
The same can be done using the *note Timer: 1491. class and its methods:
>>> import timeit
>>> t = timeit.Timer('char in text', setup='text = "sample string"; char = "g"')
>>> t.timeit()
0.3955516149999312
>>> t.repeat()
[0.40183617287970225, 0.37027556854118704, 0.38344867356679524, 0.3712595970846668, 0.37866875250654886]
The following examples show how to time expressions that contain
multiple lines. Here we compare the cost of using *note hasattr(): 4ce.
vs. *note try: 6e4./*note except: 18b. to test for missing and present
object attributes:
$ python -m timeit "try:" " str.__bool__" "except AttributeError:" " pass"
20000 loops, best of 5: 15.7 usec per loop
$ python -m timeit "if hasattr(str, '__bool__'): pass"
50000 loops, best of 5: 4.26 usec per loop
$ python -m timeit "try:" " int.__bool__" "except AttributeError:" " pass"
200000 loops, best of 5: 1.43 usec per loop
$ python -m timeit "if hasattr(int, '__bool__'): pass"
100000 loops, best of 5: 2.23 usec per loop
>>> import timeit
>>> # attribute is missing
>>> s = """\
... try:
... str.__bool__
... except AttributeError:
... pass
... """
>>> timeit.timeit(stmt=s, number=100000)
0.9138244460009446
>>> s = "if hasattr(str, '__bool__'): pass"
>>> timeit.timeit(stmt=s, number=100000)
0.5829014980008651
>>>
>>> # attribute is present
>>> s = """\
... try:
... int.__bool__
... except AttributeError:
... pass
... """
>>> timeit.timeit(stmt=s, number=100000)
0.04215312199994514
>>> s = "if hasattr(int, '__bool__'): pass"
>>> timeit.timeit(stmt=s, number=100000)
0.08588060699912603
To give the *note timeit: ef. module access to functions you define, you
can pass a 'setup' parameter which contains an import statement:
def test():
"""Stupid test function"""
L = [i for i in range(100)]
if __name__ == '__main__':
import timeit
print(timeit.timeit("test()", setup="from __main__ import test"))
Another option is to pass *note globals(): 1867. to the 'globals'
parameter, which will cause the code to be executed within your current
global namespace. This can be more convenient than individually
specifying imports:
def f(x):
return x**2
def g(x):
return x**4
def h(x):
return x**8
import timeit
print(timeit.timeit('[func(42) for func in (f,g,h)]', globals=globals()))
File: python.info, Node: trace — Trace or track Python statement execution, Next: tracemalloc — Trace memory allocations, Prev: timeit — Measure execution time of small code snippets, Up: Debugging and Profiling
5.28.7 ‘trace’ — Trace or track Python statement execution
----------------------------------------------------------
'Source code:' Lib/trace.py(1)
__________________________________________________________________
The *note trace: fd. module allows you to trace program execution,
generate annotated statement coverage listings, print caller/callee
relationships and list functions executed during a program run. It can
be used in another program or from the command line.
See also
........
Coverage.py(2)
A popular third-party coverage tool that provides HTML output along
with advanced features such as branch coverage.
* Menu:
* Command-Line Usage: Command-Line Usage<3>.
* Programmatic Interface::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/trace.py
(2) https://coverage.readthedocs.io/
File: python.info, Node: Command-Line Usage<3>, Next: Programmatic Interface, Up: trace — Trace or track Python statement execution
5.28.7.1 Command-Line Usage
...........................
The *note trace: fd. module can be invoked from the command line. It
can be as simple as
python -m trace --count -C . somefile.py ...
The above will execute ‘somefile.py’ and generate annotated listings of
all Python modules imported during the execution into the current
directory.
-- Option: --help
Display usage and exit.
-- Option: --version
Display the version of the module and exit.
Added in version 3.8: Added ‘--module’ option that allows to run an
executable module.
* Menu:
* Main options::
* Modifiers::
* Filters::
File: python.info, Node: Main options, Next: Modifiers, Up: Command-Line Usage<3>
5.28.7.2 Main options
.....................
At least one of the following options must be specified when invoking
*note trace: fd. The *note -listfuncs: 42c6. option is mutually
exclusive with the *note -trace: 42c7. and *note -count: 42c8. options.
When *note -listfuncs: 42c6. is provided, neither *note -count: 42c8.
nor *note -trace: 42c7. are accepted, and vice versa.
-- Option: -c, --count
Produce a set of annotated listing files upon program completion
that shows how many times each statement was executed. See also
*note -coverdir: 42ca, *note -file: 42cb. and *note -no-report:
42cc. below.
-- Option: -t, --trace
Display lines as they are executed.
-- Option: -l, --listfuncs
Display the functions executed by running the program.
-- Option: -r, --report
Produce an annotated list from an earlier program run that used the
*note -count: 42c8. and *note -file: 42cb. option. This does not
execute any code.
-- Option: -T, --trackcalls
Display the calling relationships exposed by running the program.
File: python.info, Node: Modifiers, Next: Filters, Prev: Main options, Up: Command-Line Usage<3>
5.28.7.3 Modifiers
..................
-- Option: -f, --file=
Name of a file to accumulate counts over several tracing runs.
Should be used with the *note -count: 42c8. option.
-- Option: -C, --coverdir=
Directory where the report files go. The coverage report for
‘package.module’ is written to file
‘`dir'/`package'/`module'.cover’.
-- Option: -m, --missing
When generating annotated listings, mark lines which were not
executed with ‘>>>>>>’.
-- Option: -s, --summary
When using *note -count: 42c8. or *note -report: 42cf, write a
brief summary to stdout for each file processed.
-- Option: -R, --no-report
Do not generate annotated listings. This is useful if you intend
to make several runs with *note -count: 42c8, and then produce a
single set of annotated listings at the end.
-- Option: -g, --timing
Prefix each line with the time since the program started. Only
used while tracing.
File: python.info, Node: Filters, Prev: Modifiers, Up: Command-Line Usage<3>
5.28.7.4 Filters
................
These options may be repeated multiple times.
-- Option: --ignore-module=
Ignore each of the given module names and its submodules (if it is
a package). The argument can be a list of names separated by a
comma.
-- Option: --ignore-dir=
Ignore all modules and packages in the named directory and
subdirectories. The argument can be a list of directories
separated by *note os.pathsep: 1d44.
File: python.info, Node: Programmatic Interface, Prev: Command-Line Usage<3>, Up: trace — Trace or track Python statement execution
5.28.7.5 Programmatic Interface
...............................
-- Class: trace.Trace (count=1, trace=1, countfuncs=0, countcallers=0,
ignoremods=(), ignoredirs=(), infile=None, outfile=None,
timing=False)
Create an object to trace execution of a single statement or
expression. All parameters are optional. 'count' enables counting
of line numbers. 'trace' enables line execution tracing.
'countfuncs' enables listing of the functions called during the
run. 'countcallers' enables call relationship tracking.
'ignoremods' is a list of modules or packages to ignore.
'ignoredirs' is a list of directories whose modules or packages
should be ignored. 'infile' is the name of the file from which to
read stored count information. 'outfile' is the name of the file
in which to write updated count information. 'timing' enables a
timestamp relative to when tracing was started to be displayed.
-- Method: run (cmd)
Execute the command and gather statistics from the execution
with the current tracing parameters. 'cmd' must be a string
or code object, suitable for passing into *note exec(): 17f.
-- Method: runctx (cmd, globals=None, locals=None)
Execute the command and gather statistics from the execution
with the current tracing parameters, in the defined global and
local environments. If not defined, 'globals' and 'locals'
default to empty dictionaries.
-- Method: runfunc (func, /, *args, **kwds)
Call 'func' with the given arguments under control of the
*note Trace: 42e2. object with the current tracing parameters.
-- Method: results ()
Return a *note CoverageResults: 42e6. object that contains the
cumulative results of all previous calls to ‘run’, ‘runctx’
and ‘runfunc’ for the given *note Trace: 42e2. instance. Does
not reset the accumulated trace results.
-- Class: trace.CoverageResults
A container for coverage results, created by *note Trace.results():
42e5. Should not be created directly by the user.
-- Method: update (other)
Merge in data from another *note CoverageResults: 42e6.
object.
-- Method: write_results (show_missing=True, summary=False,
coverdir=None, *, ignore_missing_files=False)
Write coverage results. Set 'show_missing' to show lines that
had no hits. Set 'summary' to include in the output the
coverage summary per module. 'coverdir' specifies the
directory into which the coverage result files will be output.
If ‘None’, the results for each source file are placed in its
directory.
If 'ignore_missing_files' is ‘True’, coverage counts for files
that no longer exist are silently ignored. Otherwise, a
missing file will raise a *note FileNotFoundError: 427.
Changed in version 3.13: Added 'ignore_missing_files'
parameter.
A simple example demonstrating the use of the programmatic interface:
import sys
import trace
# create a Trace object, telling it what to ignore, and whether to
# do tracing or line-counting or both.
tracer = trace.Trace(
ignoredirs=[sys.prefix, sys.exec_prefix],
trace=0,
count=1)
# run the new command using the given tracer
tracer.run('main()')
# make a report, placing output in the current directory
r = tracer.results()
r.write_results(show_missing=True, coverdir=".")
File: python.info, Node: tracemalloc — Trace memory allocations, Prev: trace — Trace or track Python statement execution, Up: Debugging and Profiling
5.28.8 ‘tracemalloc’ — Trace memory allocations
-----------------------------------------------
Added in version 3.4.
'Source code:' Lib/tracemalloc.py(1)
__________________________________________________________________
The tracemalloc module is a debug tool to trace memory blocks allocated
by Python. It provides the following information:
* Traceback where an object was allocated
* Statistics on allocated memory blocks per filename and per line
number: total size, number and average size of allocated memory
blocks
* Compute the differences between two snapshots to detect memory
leaks
To trace most memory blocks allocated by Python, the module should be
started as early as possible by setting the *note PYTHONTRACEMALLOC:
1d3c. environment variable to ‘1’, or by using *note -X: 176.
‘tracemalloc’ command line option. The *note tracemalloc.start(): 1d3b.
function can be called at runtime to start tracing Python memory
allocations.
By default, a trace of an allocated memory block only stores the most
recent frame (1 frame). To store 25 frames at startup: set the *note
PYTHONTRACEMALLOC: 1d3c. environment variable to ‘25’, or use the *note
-X: 176. ‘tracemalloc=25’ command line option.
* Menu:
* Examples: Examples<31>.
* API::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/tracemalloc.py
File: python.info, Node: Examples<31>, Next: API, Up: tracemalloc — Trace memory allocations
5.28.8.1 Examples
.................
* Menu:
* Display the top 10::
* Compute differences::
* Get the traceback of a memory block::
* Pretty top::
File: python.info, Node: Display the top 10, Next: Compute differences, Up: Examples<31>
5.28.8.2 Display the top 10
...........................
Display the 10 files allocating the most memory:
import tracemalloc
tracemalloc.start()
# ... run your application ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)
Example of output of the Python test suite:
[ Top 10 ]
:716: size=4855 KiB, count=39328, average=126 B
:284: size=521 KiB, count=3199, average=167 B
/usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B
/usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B
/usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B
/usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B
:1446: size=70.4 KiB, count=911, average=79 B
:1454: size=52.0 KiB, count=25, average=2131 B
:5: size=49.7 KiB, count=148, average=344 B
/usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB
We can see that Python loaded ‘4855 KiB’ data (bytecode and constants)
from modules and that the *note collections: 1d. module allocated ‘244
KiB’ to build *note namedtuple: 1ca. types.
See *note Snapshot.statistics(): 42ed. for more options.
File: python.info, Node: Compute differences, Next: Get the traceback of a memory block, Prev: Display the top 10, Up: Examples<31>
5.28.8.3 Compute differences
............................
Take two snapshots and display the differences:
import tracemalloc
tracemalloc.start()
# ... start your application ...
snapshot1 = tracemalloc.take_snapshot()
# ... call the function leaking memory ...
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("[ Top 10 differences ]")
for stat in top_stats[:10]:
print(stat)
Example of output before/after running some tests of the Python test
suite:
[ Top 10 differences ]
:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B
/usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B
/usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B
:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B
/usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B
/usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB
/usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B
/usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B
/usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B
/usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B
We can see that Python has loaded ‘8173 KiB’ of module data (bytecode
and constants), and that this is ‘4428 KiB’ more than had been loaded
before the tests, when the previous snapshot was taken. Similarly, the
*note linecache: 85. module has cached ‘940 KiB’ of Python source code
to format tracebacks, all of it since the previous snapshot.
If the system has little free memory, snapshots can be written on disk
using the *note Snapshot.dump(): 42ef. method to analyze the snapshot
offline. Then use the *note Snapshot.load(): 42f0. method reload the
snapshot.
File: python.info, Node: Get the traceback of a memory block, Next: Pretty top, Prev: Compute differences, Up: Examples<31>
5.28.8.4 Get the traceback of a memory block
............................................
Code to display the traceback of the biggest memory block:
import tracemalloc
# Store 25 frames
tracemalloc.start(25)
# ... run your application ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('traceback')
# pick the biggest memory block
stat = top_stats[0]
print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
for line in stat.traceback.format():
print(line)
Example of output of the Python test suite (traceback limited to 25
frames):
903 memory blocks: 870.1 KiB
File "", line 716
File "", line 1036
File "", line 934
File "", line 1068
File "", line 619
File "", line 1581
File "", line 1614
File "/usr/lib/python3.4/doctest.py", line 101
import pdb
File "", line 284
File "", line 938
File "", line 1068
File "", line 619
File "", line 1581
File "", line 1614
File "/usr/lib/python3.4/test/support/__init__.py", line 1728
import doctest
File "/usr/lib/python3.4/test/test_pickletools.py", line 21
support.run_doctest(pickletools)
File "/usr/lib/python3.4/test/regrtest.py", line 1276
test_runner()
File "/usr/lib/python3.4/test/regrtest.py", line 976
display_failure=not verbose)
File "/usr/lib/python3.4/test/regrtest.py", line 761
match_tests=ns.match_tests)
File "/usr/lib/python3.4/test/regrtest.py", line 1563
main()
File "/usr/lib/python3.4/test/__main__.py", line 3
regrtest.main_in_temp_cwd()
File "/usr/lib/python3.4/runpy.py", line 73
exec(code, run_globals)
File "/usr/lib/python3.4/runpy.py", line 160
"__main__", fname, loader, pkg_name)
We can see that the most memory was allocated in the *note importlib:
77. module to load data (bytecode and constants) from modules: ‘870.1
KiB’. The traceback is where the *note importlib: 77. loaded data most
recently: on the ‘import pdb’ line of the *note doctest: 3a. module.
The traceback may change if a new module is loaded.
File: python.info, Node: Pretty top, Prev: Get the traceback of a memory block, Up: Examples<31>
5.28.8.5 Pretty top
...................
Code to display the 10 lines allocating the most memory with a pretty
output, ignoring ‘’ and ‘’ files:
import linecache
import os
import tracemalloc
def display_top(snapshot, key_type='lineno', limit=10):
snapshot = snapshot.filter_traces((
tracemalloc.Filter(False, ""),
tracemalloc.Filter(False, ""),
))
top_stats = snapshot.statistics(key_type)
print("Top %s lines" % limit)
for index, stat in enumerate(top_stats[:limit], 1):
frame = stat.traceback[0]
print("#%s: %s:%s: %.1f KiB"
% (index, frame.filename, frame.lineno, stat.size / 1024))
line = linecache.getline(frame.filename, frame.lineno).strip()
if line:
print(' %s' % line)
other = top_stats[limit:]
if other:
size = sum(stat.size for stat in other)
print("%s other: %.1f KiB" % (len(other), size / 1024))
total = sum(stat.size for stat in top_stats)
print("Total allocated size: %.1f KiB" % (total / 1024))
tracemalloc.start()
# ... run your application ...
snapshot = tracemalloc.take_snapshot()
display_top(snapshot)
Example of output of the Python test suite:
Top 10 lines
#1: Lib/base64.py:414: 419.8 KiB
_b85chars2 = [(a + b) for a in _b85chars for b in _b85chars]
#2: Lib/base64.py:306: 419.8 KiB
_a85chars2 = [(a + b) for a in _a85chars for b in _a85chars]
#3: collections/__init__.py:368: 293.6 KiB
exec(class_definition, namespace)
#4: Lib/abc.py:133: 115.2 KiB
cls = super().__new__(mcls, name, bases, namespace)
#5: unittest/case.py:574: 103.1 KiB
testMethod()
#6: Lib/linecache.py:127: 95.4 KiB
lines = fp.readlines()
#7: urllib/parse.py:476: 71.8 KiB
for a in _hexdig for b in _hexdig}
#8: :5: 62.0 KiB
#9: Lib/_weakrefset.py:37: 60.0 KiB
self.data = set()
#10: Lib/base64.py:142: 59.8 KiB
_b32tab2 = [a + b for a in _b32tab for b in _b32tab]
6220 other: 3602.8 KiB
Total allocated size: 5303.1 KiB
See *note Snapshot.statistics(): 42ed. for more options.
* Menu:
* Record the current and peak size of all traced memory blocks::
File: python.info, Node: Record the current and peak size of all traced memory blocks, Up: Pretty top
5.28.8.6 Record the current and peak size of all traced memory blocks
.....................................................................
The following code computes two sums like ‘0 + 1 + 2 + ...’
inefficiently, by creating a list of those numbers. This list consumes
a lot of memory temporarily. We can use *note get_traced_memory():
42f4. and *note reset_peak(): 93b. to observe the small memory usage
after the sum is computed as well as the peak memory usage during the
computations:
import tracemalloc
tracemalloc.start()
# Example code: compute a sum with a large temporary list
large_sum = sum(list(range(100000)))
first_size, first_peak = tracemalloc.get_traced_memory()
tracemalloc.reset_peak()
# Example code: compute a sum with a small temporary list
small_sum = sum(list(range(1000)))
second_size, second_peak = tracemalloc.get_traced_memory()
print(f"{first_size=}, {first_peak=}")
print(f"{second_size=}, {second_peak=}")
Output:
first_size=664, first_peak=3592984
second_size=804, second_peak=29704
Using *note reset_peak(): 93b. ensured we could accurately record the
peak during the computation of ‘small_sum’, even though it is much
smaller than the overall peak size of memory blocks since the *note
start(): 1d3b. call. Without the call to *note reset_peak(): 93b,
‘second_peak’ would still be the peak from the computation ‘large_sum’
(that is, equal to ‘first_peak’). In this case, both peaks are much
higher than the final memory usage, and which suggests we could optimise
(by removing the unnecessary call to *note list: 60d, and writing
‘sum(range(...))’).
File: python.info, Node: API, Prev: Examples<31>, Up: tracemalloc — Trace memory allocations
5.28.8.7 API
............
* Menu:
* Functions: Functions<11>.
* DomainFilter::
* Filter::
* Frame::
* Snapshot::
* Statistic::
* StatisticDiff::
* Trace::
* Traceback::
File: python.info, Node: Functions<11>, Next: DomainFilter, Up: API
5.28.8.8 Functions
..................
-- Function: tracemalloc.clear_traces ()
Clear traces of memory blocks allocated by Python.
See also *note stop(): 1582.
-- Function: tracemalloc.get_object_traceback (obj)
Get the traceback where the Python object 'obj' was allocated.
Return a *note Traceback: b96. instance, or ‘None’ if the *note
tracemalloc: ff. module is not tracing memory allocations or did
not trace the allocation of the object.
See also *note gc.get_referrers(): 800. and *note sys.getsizeof():
176a. functions.
-- Function: tracemalloc.get_traceback_limit ()
Get the maximum number of frames stored in the traceback of a
trace.
The *note tracemalloc: ff. module must be tracing memory
allocations to get the limit, otherwise an exception is raised.
The limit is set by the *note start(): 1d3b. function.
-- Function: tracemalloc.get_traced_memory ()
Get the current size and peak size of memory blocks traced by the
*note tracemalloc: ff. module as a tuple: ‘(current: int, peak:
int)’.
-- Function: tracemalloc.reset_peak ()
Set the peak size of memory blocks traced by the *note tracemalloc:
ff. module to the current size.
Do nothing if the *note tracemalloc: ff. module is not tracing
memory allocations.
This function only modifies the recorded peak size, and does not
modify or clear any traces, unlike *note clear_traces(): 42f7.
Snapshots taken with *note take_snapshot(): 42f9. before a call to
*note reset_peak(): 93b. can be meaningfully compared to snapshots
taken after the call.
See also *note get_traced_memory(): 42f4.
Added in version 3.9.
-- Function: tracemalloc.get_tracemalloc_memory ()
Get the memory usage in bytes of the *note tracemalloc: ff. module
used to store traces of memory blocks. Return an *note int: 259.
-- Function: tracemalloc.is_tracing ()
‘True’ if the *note tracemalloc: ff. module is tracing Python
memory allocations, ‘False’ otherwise.
See also *note start(): 1d3b. and *note stop(): 1582. functions.
-- Function: tracemalloc.start (nframe: int = 1)
Start tracing Python memory allocations: install hooks on Python
memory allocators. Collected tracebacks of traces will be limited
to 'nframe' frames. By default, a trace of a memory block only
stores the most recent frame: the limit is ‘1’. 'nframe' must be
greater or equal to ‘1’.
You can still read the original number of total frames that
composed the traceback by looking at the *note
Traceback.total_nframe: 42fc. attribute.
Storing more than ‘1’ frame is only useful to compute statistics
grouped by ‘'traceback'’ or to compute cumulative statistics: see
the *note Snapshot.compare_to(): 42fd. and *note
Snapshot.statistics(): 42ed. methods.
Storing more frames increases the memory and CPU overhead of the
*note tracemalloc: ff. module. Use the *note
get_tracemalloc_memory(): 42fa. function to measure how much memory
is used by the *note tracemalloc: ff. module.
The *note PYTHONTRACEMALLOC: 1d3c. environment variable
(‘PYTHONTRACEMALLOC=NFRAME’) and the *note -X: 176.
‘tracemalloc=NFRAME’ command line option can be used to start
tracing at startup.
See also *note stop(): 1582, *note is_tracing(): 42fb. and *note
get_traceback_limit(): 42f8. functions.
-- Function: tracemalloc.stop ()
Stop tracing Python memory allocations: uninstall hooks on Python
memory allocators. Also clears all previously collected traces of
memory blocks allocated by Python.
Call *note take_snapshot(): 42f9. function to take a snapshot of
traces before clearing them.
See also *note start(): 1d3b, *note is_tracing(): 42fb. and *note
clear_traces(): 42f7. functions.
-- Function: tracemalloc.take_snapshot ()
Take a snapshot of traces of memory blocks allocated by Python.
Return a new *note Snapshot: 42fe. instance.
The snapshot does not include memory blocks allocated before the
*note tracemalloc: ff. module started to trace memory allocations.
Tracebacks of traces are limited to *note get_traceback_limit():
42f8. frames. Use the 'nframe' parameter of the *note start():
1d3b. function to store more frames.
The *note tracemalloc: ff. module must be tracing memory
allocations to take a snapshot, see the *note start(): 1d3b.
function.
See also the *note get_object_traceback(): 1b57. function.
File: python.info, Node: DomainFilter, Next: Filter, Prev: Functions<11>, Up: API
5.28.8.9 DomainFilter
.....................
-- Class: tracemalloc.DomainFilter (inclusive: bool, domain: int)
Filter traces of memory blocks by their address space (domain).
Added in version 3.6.
-- Attribute: inclusive
If 'inclusive' is ‘True’ (include), match memory blocks
allocated in the address space *note domain: 4301.
If 'inclusive' is ‘False’ (exclude), match memory blocks not
allocated in the address space *note domain: 4301.
-- Attribute: domain
Address space of a memory block (‘int’). Read-only property.
File: python.info, Node: Filter, Next: Frame, Prev: DomainFilter, Up: API
5.28.8.10 Filter
................
-- Class: tracemalloc.Filter (inclusive: bool, filename_pattern: str,
lineno: int = None, all_frames: bool = False, domain: int =
None)
Filter on traces of memory blocks.
See the *note fnmatch.fnmatch(): 189f. function for the syntax of
'filename_pattern'. The ‘'.pyc'’ file extension is replaced with
‘'.py'’.
Examples:
* ‘Filter(True, subprocess.__file__)’ only includes traces of
the *note subprocess: d6. module
* ‘Filter(False, tracemalloc.__file__)’ excludes traces of the
*note tracemalloc: ff. module
* ‘Filter(False, "")’ excludes empty tracebacks
Changed in version 3.5: The ‘'.pyo'’ file extension is no longer
replaced with ‘'.py'’.
Changed in version 3.6: Added the *note domain: 4304. attribute.
-- Attribute: domain
Address space of a memory block (‘int’ or ‘None’).
tracemalloc uses the domain ‘0’ to trace memory allocations
made by Python. C extensions can use other domains to trace
other resources.
-- Attribute: inclusive
If 'inclusive' is ‘True’ (include), only match memory blocks
allocated in a file with a name matching *note
filename_pattern: 4306. at line number *note lineno: 4307.
If 'inclusive' is ‘False’ (exclude), ignore memory blocks
allocated in a file with a name matching *note
filename_pattern: 4306. at line number *note lineno: 4307.
-- Attribute: lineno
Line number (‘int’) of the filter. If 'lineno' is ‘None’, the
filter matches any line number.
-- Attribute: filename_pattern
Filename pattern of the filter (‘str’). Read-only property.
-- Attribute: all_frames
If 'all_frames' is ‘True’, all frames of the traceback are
checked. If 'all_frames' is ‘False’, only the most recent
frame is checked.
This attribute has no effect if the traceback limit is ‘1’.
See the *note get_traceback_limit(): 42f8. function and *note
Snapshot.traceback_limit: 4309. attribute.
File: python.info, Node: Frame, Next: Snapshot, Prev: Filter, Up: API
5.28.8.11 Frame
...............
-- Class: tracemalloc.Frame
Frame of a traceback.
The *note Traceback: b96. class is a sequence of *note Frame: 430b.
instances.
-- Attribute: filename
Filename (‘str’).
-- Attribute: lineno
Line number (‘int’).
File: python.info, Node: Snapshot, Next: Statistic, Prev: Frame, Up: API
5.28.8.12 Snapshot
..................
-- Class: tracemalloc.Snapshot
Snapshot of traces of memory blocks allocated by Python.
The *note take_snapshot(): 42f9. function creates a snapshot
instance.
-- Method: compare_to (old_snapshot: Snapshot, key_type: str,
cumulative: bool = False)
Compute the differences with an old snapshot. Get statistics
as a sorted list of *note StatisticDiff: 430f. instances
grouped by 'key_type'.
See the *note Snapshot.statistics(): 42ed. method for
'key_type' and 'cumulative' parameters.
The result is sorted from the biggest to the smallest by:
absolute value of *note StatisticDiff.size_diff: 4310, *note
StatisticDiff.size: 4311, absolute value of *note
StatisticDiff.count_diff: 4312, *note Statistic.count: 4313.
and then by *note StatisticDiff.traceback: 4314.
-- Method: dump (filename)
Write the snapshot into a file.
Use *note load(): 42f0. to reload the snapshot.
-- Method: filter_traces (filters)
Create a new *note Snapshot: 42fe. instance with a filtered
*note traces: 4316. sequence, 'filters' is a list of *note
DomainFilter: cf1. and *note Filter: 4303. instances. If
'filters' is an empty list, return a new *note Snapshot: 42fe.
instance with a copy of the traces.
All inclusive filters are applied at once, a trace is ignored
if no inclusive filters match it. A trace is ignored if at
least one exclusive filter matches it.
Changed in version 3.6: *note DomainFilter: cf1. instances are
now also accepted in 'filters'.
-- Method: classmethod load (filename)
Load a snapshot from a file.
See also *note dump(): 42ef.
-- Method: statistics (key_type: str, cumulative: bool = False)
Get statistics as a sorted list of *note Statistic: 4317.
instances grouped by 'key_type':
key_type description
-------------------------------------------------------
‘'filename'’ filename
‘'lineno'’ filename and line number
‘'traceback'’ traceback
If 'cumulative' is ‘True’, cumulate size and count of memory
blocks of all frames of the traceback of a trace, not only the
most recent frame. The cumulative mode can only be used with
'key_type' equals to ‘'filename'’ and ‘'lineno'’.
The result is sorted from the biggest to the smallest by:
*note Statistic.size: 4318, *note Statistic.count: 4313. and
then by *note Statistic.traceback: 4319.
-- Attribute: traceback_limit
Maximum number of frames stored in the traceback of *note
traces: 4316.: result of the *note get_traceback_limit():
42f8. when the snapshot was taken.
-- Attribute: traces
Traces of all memory blocks allocated by Python: sequence of
*note Trace: 431a. instances.
The sequence has an undefined order. Use the *note
Snapshot.statistics(): 42ed. method to get a sorted list of
statistics.
File: python.info, Node: Statistic, Next: StatisticDiff, Prev: Snapshot, Up: API
5.28.8.13 Statistic
...................
-- Class: tracemalloc.Statistic
Statistic on memory allocations.
*note Snapshot.statistics(): 42ed. returns a list of *note
Statistic: 4317. instances.
See also the *note StatisticDiff: 430f. class.
-- Attribute: count
Number of memory blocks (‘int’).
-- Attribute: size
Total size of memory blocks in bytes (‘int’).
-- Attribute: traceback
Traceback where the memory block was allocated, *note
Traceback: b96. instance.
File: python.info, Node: StatisticDiff, Next: Trace, Prev: Statistic, Up: API
5.28.8.14 StatisticDiff
.......................
-- Class: tracemalloc.StatisticDiff
Statistic difference on memory allocations between an old and a new
*note Snapshot: 42fe. instance.
*note Snapshot.compare_to(): 42fd. returns a list of *note
StatisticDiff: 430f. instances. See also the *note Statistic:
4317. class.
-- Attribute: count
Number of memory blocks in the new snapshot (‘int’): ‘0’ if
the memory blocks have been released in the new snapshot.
-- Attribute: count_diff
Difference of number of memory blocks between the old and the
new snapshots (‘int’): ‘0’ if the memory blocks have been
allocated in the new snapshot.
-- Attribute: size
Total size of memory blocks in bytes in the new snapshot
(‘int’): ‘0’ if the memory blocks have been released in the
new snapshot.
-- Attribute: size_diff
Difference of total size of memory blocks in bytes between the
old and the new snapshots (‘int’): ‘0’ if the memory blocks
have been allocated in the new snapshot.
-- Attribute: traceback
Traceback where the memory blocks were allocated, *note
Traceback: b96. instance.
File: python.info, Node: Trace, Next: Traceback, Prev: StatisticDiff, Up: API
5.28.8.15 Trace
...............
-- Class: tracemalloc.Trace
Trace of a memory block.
The *note Snapshot.traces: 4316. attribute is a sequence of *note
Trace: 431a. instances.
Changed in version 3.6: Added the *note domain: 431f. attribute.
-- Attribute: domain
Address space of a memory block (‘int’). Read-only property.
tracemalloc uses the domain ‘0’ to trace memory allocations
made by Python. C extensions can use other domains to trace
other resources.
-- Attribute: size
Size of the memory block in bytes (‘int’).
-- Attribute: traceback
Traceback where the memory block was allocated, *note
Traceback: b96. instance.
File: python.info, Node: Traceback, Prev: Trace, Up: API
5.28.8.16 Traceback
...................
-- Class: tracemalloc.Traceback
Sequence of *note Frame: 430b. instances sorted from the oldest
frame to the most recent frame.
A traceback contains at least ‘1’ frame. If the ‘tracemalloc’
module failed to get a frame, the filename ‘""’ at line
number ‘0’ is used.
When a snapshot is taken, tracebacks of traces are limited to *note
get_traceback_limit(): 42f8. frames. See the *note
take_snapshot(): 42f9. function. The original number of frames of
the traceback is stored in the *note Traceback.total_nframe: 42fc.
attribute. That allows to know if a traceback has been truncated
by the traceback limit.
The *note Trace.traceback: 4321. attribute is an instance of *note
Traceback: b96. instance.
Changed in version 3.7: Frames are now sorted from the oldest to
the most recent, instead of most recent to oldest.
-- Attribute: total_nframe
Total number of frames that composed the traceback before
truncation. This attribute can be set to ‘None’ if the
information is not available.
Changed in version 3.9: The *note Traceback.total_nframe: 42fc.
attribute was added.
-- Method: format (limit=None, most_recent_first=False)
Format the traceback as a list of lines. Use the *note
linecache: 85. module to retrieve lines from the source code.
If 'limit' is set, format the 'limit' most recent frames if
'limit' is positive. Otherwise, format the ‘abs(limit)’
oldest frames. If 'most_recent_first' is ‘True’, the order of
the formatted frames is reversed, returning the most recent
frame first instead of last.
Similar to the *note traceback.format_tb(): 4323. function,
except that *note format(): b97. does not include newlines.
Example:
print("Traceback (most recent call first):")
for line in traceback:
print(line)
Output:
Traceback (most recent call first):
File "test.py", line 9
obj = Object()
File "test.py", line 12
tb = tracemalloc.get_object_traceback(f())
File: python.info, Node: Software Packaging and Distribution, Next: Python Runtime Services, Prev: Debugging and Profiling, Up: The Python Standard Library
5.29 Software Packaging and Distribution
========================================
These libraries help you with publishing and installing Python software.
While these modules are designed to work in conjunction with the Python
Package Index(1), they can also be used with a local index server, or
without any index server at all.
* Menu:
* ensurepip — Bootstrapping the pip installer::
* venv — Creation of virtual environments::
* zipapp — Manage executable Python zip archives::
---------- Footnotes ----------
(1) https://pypi.org
File: python.info, Node: ensurepip — Bootstrapping the pip installer, Next: venv — Creation of virtual environments, Up: Software Packaging and Distribution
5.29.1 ‘ensurepip’ — Bootstrapping the ‘pip’ installer
------------------------------------------------------
Added in version 3.4.
'Source code:' Lib/ensurepip(1)
__________________________________________________________________
The *note ensurepip: 55. package provides support for bootstrapping the
‘pip’ installer into an existing Python installation or virtual
environment. This bootstrapping approach reflects the fact that ‘pip’
is an independent project with its own release cycle, and the latest
available stable version is bundled with maintenance and feature
releases of the CPython reference interpreter.
In most cases, end users of Python shouldn’t need to invoke this module
directly (as ‘pip’ should be bootstrapped by default), but it may be
needed if installing ‘pip’ was skipped when installing Python (or when
creating a virtual environment) or after explicitly uninstalling ‘pip’.
Note: This module 'does not' access the internet. All of the
components needed to bootstrap ‘pip’ are included as internal parts
of the package.
See also
........
*note Installing Python Modules: ef0.
The end user guide for installing Python packages
PEP 453(2): Explicit bootstrapping of pip in Python installations
The original rationale and specification for this module.
*note Availability: 1d54.: not Android, not iOS, not WASI.
This module is not supported on *note mobile platforms: 1e55. or *note
WebAssembly platforms: 17e0.
* Menu:
* Command line interface::
* Module API::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/ensurepip
(2) https://peps.python.org/pep-0453/
File: python.info, Node: Command line interface, Next: Module API, Up: ensurepip — Bootstrapping the pip installer
5.29.1.1 Command line interface
...............................
The command line interface is invoked using the interpreter’s ‘-m’
switch.
The simplest possible invocation is:
python -m ensurepip
This invocation will install ‘pip’ if it is not already installed, but
otherwise does nothing. To ensure the installed version of ‘pip’ is at
least as recent as the one available in ‘ensurepip’, pass the
‘--upgrade’ option:
python -m ensurepip --upgrade
By default, ‘pip’ is installed into the current virtual environment (if
one is active) or into the system site packages (if there is no active
virtual environment). The installation location can be controlled
through two additional command line options:
-- Option: --root
Installs ‘pip’ relative to the given root directory rather than the
root of the currently active virtual environment (if any) or the
default root for the current Python installation.
-- Option: --user
Installs ‘pip’ into the user site packages directory rather than
globally for the current Python installation (this option is not
permitted inside an active virtual environment).
By default, the scripts ‘pipX’ and ‘pipX.Y’ will be installed (where X.Y
stands for the version of Python used to invoke ‘ensurepip’). The
scripts installed can be controlled through two additional command line
options:
-- Option: --altinstall
If an alternate installation is requested, the ‘pipX’ script will
'not' be installed.
-- Option: --default-pip
If a “default pip” installation is requested, the ‘pip’ script will
be installed in addition to the two regular scripts.
Providing both of the script selection options will trigger an
exception.
File: python.info, Node: Module API, Prev: Command line interface, Up: ensurepip — Bootstrapping the pip installer
5.29.1.2 Module API
...................
*note ensurepip: 55. exposes two functions for programmatic use:
-- Function: ensurepip.version ()
Returns a string specifying the available version of pip that will
be installed when bootstrapping an environment.
-- Function: ensurepip.bootstrap (root=None, upgrade=False, user=False,
altinstall=False, default_pip=False, verbosity=0)
Bootstraps ‘pip’ into the current or designated environment.
'root' specifies an alternative root directory to install relative
to. If 'root' is ‘None’, then installation uses the default
install location for the current environment.
'upgrade' indicates whether or not to upgrade an existing
installation of an earlier version of ‘pip’ to the available
version.
'user' indicates whether to use the user scheme rather than
installing globally.
By default, the scripts ‘pipX’ and ‘pipX.Y’ will be installed
(where X.Y stands for the current version of Python).
If 'altinstall' is set, then ‘pipX’ will 'not' be installed.
If 'default_pip' is set, then ‘pip’ will be installed in addition
to the two regular scripts.
Setting both 'altinstall' and 'default_pip' will trigger *note
ValueError: 204.
'verbosity' controls the level of output to *note sys.stdout: ad6.
from the bootstrapping operation.
Raises an *note auditing event: 18ba. ‘ensurepip.bootstrap’ with
argument ‘root’.
Note: The bootstrapping process has side effects on both
‘sys.path’ and ‘os.environ’. Invoking the command line
interface in a subprocess instead allows these side effects to
be avoided.
Note: The bootstrapping process may install additional modules
required by ‘pip’, but other software should not assume those
dependencies will always be present by default (as the
dependencies may be removed in a future version of ‘pip’).
File: python.info, Node: venv — Creation of virtual environments, Next: zipapp — Manage executable Python zip archives, Prev: ensurepip — Bootstrapping the pip installer, Up: Software Packaging and Distribution
5.29.2 ‘venv’ — Creation of virtual environments
------------------------------------------------
Added in version 3.3.
'Source code:' Lib/venv/(1)
__________________________________________________________________
The ‘venv’ module supports creating lightweight “virtual environments”,
each with their own independent set of Python packages installed in
their *note site: c7. directories. A virtual environment is created on
top of an existing Python installation, known as the virtual
environment’s “base” Python, and by default is isolated from the
packages in the base environment, so that only those explicitly
installed in the virtual environment are available.
When used from within a virtual environment, common installation tools
such as pip(2) will install Python packages into a virtual environment
without needing to be told to do so explicitly.
A virtual environment is (amongst other things):
* Used to contain a specific Python interpreter and software
libraries and binaries which are needed to support a project
(library or application). These are by default isolated from
software in other virtual environments and Python interpreters and
libraries installed in the operating system.
* Contained in a directory, conventionally named ‘.venv’ or ‘venv’ in
the project directory, or under a container directory for lots of
virtual environments, such as ‘~/.virtualenvs’.
* Not checked into source control systems such as Git.
* Considered as disposable – it should be simple to delete and
recreate it from scratch. You don’t place any project code in the
environment.
* Not considered as movable or copyable – you just recreate the same
environment in the target location.
See PEP 405(3) for more background on Python virtual environments.
See also
........
Python Packaging User Guide: Creating and using virtual environments(4)
*note Availability: 1d54.: not Android, not iOS, not WASI.
This module is not supported on *note mobile platforms: 1e55. or *note
WebAssembly platforms: 17e0.
* Menu:
* Creating virtual environments::
* How venvs work::
* API: API<2>.
* An example of extending EnvBuilder::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/venv/
(2) https://pypi.org/project/pip/
(3) https://peps.python.org/pep-0405/
(4)
https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments
File: python.info, Node: Creating virtual environments, Next: How venvs work, Up: venv — Creation of virtual environments
5.29.2.1 Creating virtual environments
......................................
*note Virtual environments: 4331. are created by executing the ‘venv’
module:
python -m venv /path/to/new/virtual/environment
This creates the target directory (including parent directories as
needed) and places a ‘pyvenv.cfg’ file in it with a ‘home’ key pointing
to the Python installation from which the command was run. It also
creates a ‘bin’ (or ‘Scripts’ on Windows) subdirectory containing a copy
or symlink of the Python executable (as appropriate for the platform or
arguments used at environment creation time). It also creates a
‘lib/pythonX.Y/site-packages’ subdirectory (on Windows, this is
‘Libsite-packages’). If an existing directory is specified, it will be
re-used.
Changed in version 3.5: The use of ‘venv’ is now recommended for
creating virtual environments.
Deprecated since version 3.6, removed in version 3.8: ‘pyvenv’ was the
recommended tool for creating virtual environments for Python 3.3 and
3.4, and replaced in 3.5 by executing ‘venv’ directly.
On Windows, invoke the ‘venv’ command as follows:
PS> python -m venv C:\path\to\new\virtual\environment
The command, if run with ‘-h’, will show the available options:
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
[--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps]
[--without-scm-ignore-files]
ENV_DIR [ENV_DIR ...]
Creates virtual Python environments in one or more target directories.
Once an environment has been created, you may wish to activate it, e.g. by
sourcing an activate script in its bin directory.
-- Option: ENV_DIR
A required argument specifying the directory to create the
environment in.
-- Option: --system-site-packages
Give the virtual environment access to the system site-packages
directory.
-- Option: --symlinks
Try to use symlinks rather than copies, when symlinks are not the
default for the platform.
-- Option: --copies
Try to use copies rather than symlinks, even when symlinks are the
default for the platform.
-- Option: --clear
Delete the contents of the environment directory if it already
exists, before environment creation.
-- Option: --upgrade
Upgrade the environment directory to use this version of Python,
assuming Python has been upgraded in-place.
-- Option: --without-pip
Skips installing or upgrading pip in the virtual environment (pip
is bootstrapped by default).
-- Option: --prompt
Provides an alternative prompt prefix for this environment.
-- Option: --upgrade-deps
Upgrade core dependencies (pip) to the latest version in PyPI.
-- Option: --without-scm-ignore-files
Skips adding SCM ignore files to the environment directory (Git is
supported by default).
Changed in version 3.4: Installs pip by default, added the
‘--without-pip’ and ‘--copies’ options.
Changed in version 3.4: In earlier versions, if the target directory
already existed, an error was raised, unless the ‘--clear’ or
‘--upgrade’ option was provided.
Changed in version 3.9: Add ‘--upgrade-deps’ option to upgrade pip +
setuptools to the latest on PyPI.
Changed in version 3.12: ‘setuptools’ is no longer a core venv
dependency.
Changed in version 3.13: Added the ‘--without-scm-ignore-files’ option.
Changed in version 3.13: ‘venv’ now creates a ‘.gitignore’ file for Git
by default.
Note: While symlinks are supported on Windows, they are not
recommended. Of particular note is that double-clicking
‘python.exe’ in File Explorer will resolve the symlink eagerly and
ignore the virtual environment.
Note: On Microsoft Windows, it may be required to enable the
‘Activate.ps1’ script by setting the execution policy for the user.
You can do this by issuing the following PowerShell command:
PS C:\> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
See About Execution Policies(1) for more information.
The created ‘pyvenv.cfg’ file also includes the
‘include-system-site-packages’ key, set to ‘true’ if ‘venv’ is run with
the ‘--system-site-packages’ option, ‘false’ otherwise.
Unless the ‘--without-pip’ option is given, *note ensurepip: 55. will be
invoked to bootstrap ‘pip’ into the virtual environment.
Multiple paths can be given to ‘venv’, in which case an identical
virtual environment will be created, according to the given options, at
each provided path.
---------- Footnotes ----------
(1) https://go.microsoft.com/fwlink/?LinkID=135170
File: python.info, Node: How venvs work, Next: API<2>, Prev: Creating virtual environments, Up: venv — Creation of virtual environments
5.29.2.2 How venvs work
.......................
When a Python interpreter is running from a virtual environment, *note
sys.prefix: 3b2. and *note sys.exec_prefix: 3ae. point to the
directories of the virtual environment, whereas *note sys.base_prefix:
3eb. and *note sys.base_exec_prefix: 3ea. point to those of the base
Python used to create the environment. It is sufficient to check
‘sys.prefix != sys.base_prefix’ to determine if the current interpreter
is running from a virtual environment.
A virtual environment may be “activated” using a script in its binary
directory (‘bin’ on POSIX; ‘Scripts’ on Windows). This will prepend
that directory to your ‘PATH’, so that running ‘python’ will invoke the
environment’s Python interpreter and you can run installed scripts
without having to use their full path. The invocation of the activation
script is platform-specific (‘’ must be replaced by the path to
the directory containing the virtual environment):
Platform Shell Command to activate virtual environment
------------------------------------------------------------------------------------------
POSIX bash/zsh ‘$ source /bin/activate’
fish ‘$ source
/bin/activate.fish’
csh/tcsh ‘$ source
/bin/activate.csh’
pwsh ‘$
/bin/Activate.ps1’
Windows cmd.exe ‘C:\> \Scripts\activate.bat’
PowerShell ‘PS C:\>
\Scripts\Activate.ps1’
Added in version 3.4: ‘fish’ and ‘csh’ activation scripts.
Added in version 3.8: PowerShell activation scripts installed under
POSIX for PowerShell Core support.
You don’t specifically 'need' to activate a virtual environment, as you
can just specify the full path to that environment’s Python interpreter
when invoking Python. Furthermore, all scripts installed in the
environment should be runnable without activating it.
In order to achieve this, scripts installed into virtual environments
have a “shebang” line which points to the environment’s Python
interpreter, ‘#!//bin/python’. This means that the script
will run with that interpreter regardless of the value of ‘PATH’. On
Windows, “shebang” line processing is supported if you have the *note
Python Launcher for Windows: 5ba. installed. Thus, double-clicking an
installed script in a Windows Explorer window should run it with the
correct interpreter without the environment needing to be activated or
on the ‘PATH’.
When a virtual environment has been activated, the ‘VIRTUAL_ENV’
environment variable is set to the path of the environment. Since
explicitly activating a virtual environment is not required to use it,
‘VIRTUAL_ENV’ cannot be relied upon to determine whether a virtual
environment is being used.
Warning: Because scripts installed in environments should not
expect the environment to be activated, their shebang lines contain
the absolute paths to their environment’s interpreters. Because of
this, environments are inherently non-portable, in the general
case. You should always have a simple means of recreating an
environment (for example, if you have a requirements file
‘requirements.txt’, you can invoke ‘pip install -r
requirements.txt’ using the environment’s ‘pip’ to install all of
the packages needed by the environment). If for any reason you
need to move the environment to a new location, you should recreate
it at the desired location and delete the one at the old location.
If you move an environment because you moved a parent directory of
it, you should recreate the environment in its new location.
Otherwise, software installed into the environment may not work as
expected.
You can deactivate a virtual environment by typing ‘deactivate’ in your
shell. The exact mechanism is platform-specific and is an internal
implementation detail (typically, a script or shell function will be
used).
File: python.info, Node: API<2>, Next: An example of extending EnvBuilder, Prev: How venvs work, Up: venv — Creation of virtual environments
5.29.2.3 API
............
The high-level method described above makes use of a simple API which
provides mechanisms for third-party virtual environment creators to
customize environment creation according to their needs, the *note
EnvBuilder: 26c. class.
-- Class: venv.EnvBuilder (system_site_packages=False, clear=False,
symlinks=False, upgrade=False, with_pip=False, prompt=None,
upgrade_deps=False, *, scm_ignore_files=frozenset())
The *note EnvBuilder: 26c. class accepts the following keyword
arguments on instantiation:
* 'system_site_packages' – a boolean value indicating that the
system Python site-packages should be available to the
environment (defaults to ‘False’).
* 'clear' – a boolean value which, if true, will delete the
contents of any existing target directory, before creating the
environment.
* 'symlinks' – a boolean value indicating whether to attempt to
symlink the Python binary rather than copying.
* 'upgrade' – a boolean value which, if true, will upgrade an
existing environment with the running Python - for use when
that Python has been upgraded in-place (defaults to ‘False’).
* 'with_pip' – a boolean value which, if true, ensures pip is
installed in the virtual environment. This uses *note
ensurepip: 55. with the ‘--default-pip’ option.
* 'prompt' – a string to be used after virtual environment is
activated (defaults to ‘None’ which means directory name of
the environment would be used). If the special string ‘"."’
is provided, the basename of the current directory is used as
the prompt.
* 'upgrade_deps' – Update the base venv modules to the latest on
PyPI
* 'scm_ignore_files' – Create ignore files based for the
specified source control managers (SCM) in the iterable.
Support is defined by having a method named
‘create_{scm}_ignore_file’. The only value supported by
default is ‘"git"’ via *note create_git_ignore_file(): 4341.
Changed in version 3.4: Added the ‘with_pip’ parameter
Changed in version 3.6: Added the ‘prompt’ parameter
Changed in version 3.9: Added the ‘upgrade_deps’ parameter
Changed in version 3.13: Added the ‘scm_ignore_files’ parameter
*note EnvBuilder: 26c. may be used as a base class.
-- Method: create (env_dir)
Create a virtual environment by specifying the target
directory (absolute or relative to the current directory)
which is to contain the virtual environment. The ‘create’
method will either create the environment in the specified
directory, or raise an appropriate exception.
The ‘create’ method of the *note EnvBuilder: 26c. class
illustrates the hooks available for subclass customization:
def create(self, env_dir):
"""
Create a virtualized Python environment in a directory.
env_dir is the target directory to create an environment in.
"""
env_dir = os.path.abspath(env_dir)
context = self.ensure_directories(env_dir)
self.create_configuration(context)
self.setup_python(context)
self.setup_scripts(context)
self.post_setup(context)
Each of the methods *note ensure_directories(): 4343, *note
create_configuration(): 4344, *note setup_python(): 4345,
*note setup_scripts(): 4346. and *note post_setup(): 4347. can
be overridden.
-- Method: ensure_directories (env_dir)
Creates the environment directory and all necessary
subdirectories that don’t already exist, and returns a context
object. This context object is just a holder for attributes
(such as paths) for use by the other methods. If the *note
EnvBuilder: 26c. is created with the arg ‘clear=True’,
contents of the environment directory will be cleared and then
all necessary subdirectories will be recreated.
The returned context object is a *note types.SimpleNamespace:
1d1. with the following attributes:
* ‘env_dir’ - The location of the virtual environment.
Used for ‘__VENV_DIR__’ in activation scripts (see *note
install_scripts(): 4348.).
* ‘env_name’ - The name of the virtual environment. Used
for ‘__VENV_NAME__’ in activation scripts (see *note
install_scripts(): 4348.).
* ‘prompt’ - The prompt to be used by the activation
scripts. Used for ‘__VENV_PROMPT__’ in activation
scripts (see *note install_scripts(): 4348.).
* ‘executable’ - The underlying Python executable used by
the virtual environment. This takes into account the
case where a virtual environment is created from another
virtual environment.
* ‘inc_path’ - The include path for the virtual
environment.
* ‘lib_path’ - The purelib path for the virtual
environment.
* ‘bin_path’ - The script path for the virtual environment.
* ‘bin_name’ - The name of the script path relative to the
virtual environment location. Used for
‘__VENV_BIN_NAME__’ in activation scripts (see *note
install_scripts(): 4348.).
* ‘env_exe’ - The name of the Python interpreter in the
virtual environment. Used for ‘__VENV_PYTHON__’ in
activation scripts (see *note install_scripts(): 4348.).
* ‘env_exec_cmd’ - The name of the Python interpreter,
taking into account filesystem redirections. This can be
used to run Python in the virtual environment.
Changed in version 3.11: The 'venv' *note sysconfig
installation scheme: 68b. is used to construct the paths of
the created directories.
Changed in version 3.12: The attribute ‘lib_path’ was added to
the context, and the context object was documented.
-- Method: create_configuration (context)
Creates the ‘pyvenv.cfg’ configuration file in the
environment.
-- Method: setup_python (context)
Creates a copy or symlink to the Python executable in the
environment. On POSIX systems, if a specific executable
‘python3.x’ was used, symlinks to ‘python’ and ‘python3’ will
be created pointing to that executable, unless files with
those names already exist.
-- Method: setup_scripts (context)
Installs activation scripts appropriate to the platform into
the virtual environment.
-- Method: upgrade_dependencies (context)
Upgrades the core venv dependency packages (currently pip(1))
in the environment. This is done by shelling out to the ‘pip’
executable in the environment.
Added in version 3.9.
Changed in version 3.12: setuptools(2) is no longer a core
venv dependency.
-- Method: post_setup (context)
A placeholder method which can be overridden in third party
implementations to pre-install packages in the virtual
environment or perform other post-creation steps.
-- Method: install_scripts (context, path)
This method can be called from *note setup_scripts(): 4346. or
*note post_setup(): 4347. in subclasses to assist in
installing custom scripts into the virtual environment.
'path' is the path to a directory that should contain
subdirectories ‘common’, ‘posix’, ‘nt’; each containing
scripts destined for the ‘bin’ directory in the environment.
The contents of ‘common’ and the directory corresponding to
*note os.name: 2b0e. are copied after some text replacement of
placeholders:
* ‘__VENV_DIR__’ is replaced with the absolute path of the
environment directory.
* ‘__VENV_NAME__’ is replaced with the environment name
(final path segment of environment directory).
* ‘__VENV_PROMPT__’ is replaced with the prompt (the
environment name surrounded by parentheses and with a
following space)
* ‘__VENV_BIN_NAME__’ is replaced with the name of the bin
directory (either ‘bin’ or ‘Scripts’).
* ‘__VENV_PYTHON__’ is replaced with the absolute path of
the environment’s executable.
The directories are allowed to exist (for when an existing
environment is being upgraded).
-- Method: create_git_ignore_file (context)
Creates a ‘.gitignore’ file within the virtual environment
that causes the entire directory to be ignored by the Git
source control manager.
Added in version 3.13.
Changed in version 3.7.2: Windows now uses redirector scripts for
‘python[w].exe’ instead of copying the actual binaries. In 3.7.2
only *note setup_python(): 4345. does nothing unless running from a
build in the source tree.
Changed in version 3.7.3: Windows copies the redirector scripts as
part of *note setup_python(): 4345. instead of *note
setup_scripts(): 4346. This was not the case in 3.7.2. When using
symlinks, the original executables will be linked.
There is also a module-level convenience function:
-- Function: venv.create (env_dir, system_site_packages=False,
clear=False, symlinks=False, with_pip=False, prompt=None,
upgrade_deps=False, *, scm_ignore_files=frozenset())
Create an *note EnvBuilder: 26c. with the given keyword arguments,
and call its *note create(): 4342. method with the 'env_dir'
argument.
Added in version 3.3.
Changed in version 3.4: Added the 'with_pip' parameter
Changed in version 3.6: Added the 'prompt' parameter
Changed in version 3.9: Added the 'upgrade_deps' parameter
Changed in version 3.13: Added the 'scm_ignore_files' parameter
---------- Footnotes ----------
(1) https://pypi.org/project/pip/
(2) https://pypi.org/project/setuptools/
File: python.info, Node: An example of extending EnvBuilder, Prev: API<2>, Up: venv — Creation of virtual environments
5.29.2.4 An example of extending ‘EnvBuilder’
.............................................
The following script shows how to extend *note EnvBuilder: 26c. by
implementing a subclass which installs setuptools and pip into a created
virtual environment:
import os
import os.path
from subprocess import Popen, PIPE
import sys
from threading import Thread
from urllib.parse import urlparse
from urllib.request import urlretrieve
import venv
class ExtendedEnvBuilder(venv.EnvBuilder):
"""
This builder installs setuptools and pip so that you can pip or
easy_install other packages into the created virtual environment.
:param nodist: If true, setuptools and pip are not installed into the
created virtual environment.
:param nopip: If true, pip is not installed into the created
virtual environment.
:param progress: If setuptools or pip are installed, the progress of the
installation can be monitored by passing a progress
callable. If specified, it is called with two
arguments: a string indicating some progress, and a
context indicating where the string is coming from.
The context argument can have one of three values:
'main', indicating that it is called from virtualize()
itself, and 'stdout' and 'stderr', which are obtained
by reading lines from the output streams of a subprocess
which is used to install the app.
If a callable is not specified, default progress
information is output to sys.stderr.
"""
def __init__(self, *args, **kwargs):
self.nodist = kwargs.pop('nodist', False)
self.nopip = kwargs.pop('nopip', False)
self.progress = kwargs.pop('progress', None)
self.verbose = kwargs.pop('verbose', False)
super().__init__(*args, **kwargs)
def post_setup(self, context):
"""
Set up any packages which need to be pre-installed into the
virtual environment being created.
:param context: The information for the virtual environment
creation request being processed.
"""
os.environ['VIRTUAL_ENV'] = context.env_dir
if not self.nodist:
self.install_setuptools(context)
# Can't install pip without setuptools
if not self.nopip and not self.nodist:
self.install_pip(context)
def reader(self, stream, context):
"""
Read lines from a subprocess' output stream and either pass to a progress
callable (if specified) or write progress information to sys.stderr.
"""
progress = self.progress
while True:
s = stream.readline()
if not s:
break
if progress is not None:
progress(s, context)
else:
if not self.verbose:
sys.stderr.write('.')
else:
sys.stderr.write(s.decode('utf-8'))
sys.stderr.flush()
stream.close()
def install_script(self, context, name, url):
_, _, path, _, _, _ = urlparse(url)
fn = os.path.split(path)[-1]
binpath = context.bin_path
distpath = os.path.join(binpath, fn)
# Download script into the virtual environment's binaries folder
urlretrieve(url, distpath)
progress = self.progress
if self.verbose:
term = '\n'
else:
term = ''
if progress is not None:
progress('Installing %s ...%s' % (name, term), 'main')
else:
sys.stderr.write('Installing %s ...%s' % (name, term))
sys.stderr.flush()
# Install in the virtual environment
args = [context.env_exe, fn]
p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath)
t1 = Thread(target=self.reader, args=(p.stdout, 'stdout'))
t1.start()
t2 = Thread(target=self.reader, args=(p.stderr, 'stderr'))
t2.start()
p.wait()
t1.join()
t2.join()
if progress is not None:
progress('done.', 'main')
else:
sys.stderr.write('done.\n')
# Clean up - no longer needed
os.unlink(distpath)
def install_setuptools(self, context):
"""
Install setuptools in the virtual environment.
:param context: The information for the virtual environment
creation request being processed.
"""
url = "https://bootstrap.pypa.io/ez_setup.py"
self.install_script(context, 'setuptools', url)
# clear up the setuptools archive which gets downloaded
pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz')
files = filter(pred, os.listdir(context.bin_path))
for f in files:
f = os.path.join(context.bin_path, f)
os.unlink(f)
def install_pip(self, context):
"""
Install pip in the virtual environment.
:param context: The information for the virtual environment
creation request being processed.
"""
url = 'https://bootstrap.pypa.io/get-pip.py'
self.install_script(context, 'pip', url)
def main(args=None):
import argparse
parser = argparse.ArgumentParser(prog=__name__,
description='Creates virtual Python '
'environments in one or '
'more target '
'directories.')
parser.add_argument('dirs', metavar='ENV_DIR', nargs='+',
help='A directory in which to create the '
'virtual environment.')
parser.add_argument('--no-setuptools', default=False,
action='store_true', dest='nodist',
help="Don't install setuptools or pip in the "
"virtual environment.")
parser.add_argument('--no-pip', default=False,
action='store_true', dest='nopip',
help="Don't install pip in the virtual "
"environment.")
parser.add_argument('--system-site-packages', default=False,
action='store_true', dest='system_site',
help='Give the virtual environment access to the '
'system site-packages dir.')
if os.name == 'nt':
use_symlinks = False
else:
use_symlinks = True
parser.add_argument('--symlinks', default=use_symlinks,
action='store_true', dest='symlinks',
help='Try to use symlinks rather than copies, '
'when symlinks are not the default for '
'the platform.')
parser.add_argument('--clear', default=False, action='store_true',
dest='clear', help='Delete the contents of the '
'virtual environment '
'directory if it already '
'exists, before virtual '
'environment creation.')
parser.add_argument('--upgrade', default=False, action='store_true',
dest='upgrade', help='Upgrade the virtual '
'environment directory to '
'use this version of '
'Python, assuming Python '
'has been upgraded '
'in-place.')
parser.add_argument('--verbose', default=False, action='store_true',
dest='verbose', help='Display the output '
'from the scripts which '
'install setuptools and pip.')
options = parser.parse_args(args)
if options.upgrade and options.clear:
raise ValueError('you cannot supply --upgrade and --clear together.')
builder = ExtendedEnvBuilder(system_site_packages=options.system_site,
clear=options.clear,
symlinks=options.symlinks,
upgrade=options.upgrade,
nodist=options.nodist,
nopip=options.nopip,
verbose=options.verbose)
for d in options.dirs:
builder.create(d)
if __name__ == '__main__':
rc = 1
try:
main()
rc = 0
except Exception as e:
print('Error: %s' % e, file=sys.stderr)
sys.exit(rc)
This script is also available for download online(1).
---------- Footnotes ----------
(1) https://gist.github.com/vsajip/4673395
File: python.info, Node: zipapp — Manage executable Python zip archives, Prev: venv — Creation of virtual environments, Up: Software Packaging and Distribution
5.29.3 ‘zipapp’ — Manage executable Python zip archives
-------------------------------------------------------
Added in version 3.5.
'Source code:' Lib/zipapp.py(1)
__________________________________________________________________
This module provides tools to manage the creation of zip files
containing Python code, which can be *note executed directly by the
Python interpreter: 1d29. The module provides both a *note Command-Line
Interface: 434d. and a *note Python API: 434e.
* Menu:
* Basic Example::
* Command-Line Interface: Command-Line Interface<5>.
* Python API::
* Examples: Examples<32>.
* Specifying the Interpreter::
* Creating Standalone Applications with zipapp::
* The Python Zip Application Archive Format::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/zipapp.py
File: python.info, Node: Basic Example, Next: Command-Line Interface<5>, Up: zipapp — Manage executable Python zip archives
5.29.3.1 Basic Example
......................
The following example shows how the *note Command-Line Interface: 434d.
can be used to create an executable archive from a directory containing
Python code. When run, the archive will execute the ‘main’ function
from the module ‘myapp’ in the archive.
$ python -m zipapp myapp -m "myapp:main"
$ python myapp.pyz
File: python.info, Node: Command-Line Interface<5>, Next: Python API, Prev: Basic Example, Up: zipapp — Manage executable Python zip archives
5.29.3.2 Command-Line Interface
...............................
When called as a program from the command line, the following form is
used:
$ python -m zipapp source [options]
If 'source' is a directory, this will create an archive from the
contents of 'source'. If 'source' is a file, it should be an archive,
and it will be copied to the target archive (or the contents of its
shebang line will be displayed if the –info option is specified).
The following options are understood:
-- Option: -o , --output=
Write the output to a file named 'output'. If this option is not
specified, the output filename will be the same as the input
'source', with the extension ‘.pyz’ added. If an explicit filename
is given, it is used as is (so a ‘.pyz’ extension should be
included if required).
An output filename must be specified if the 'source' is an archive
(and in that case, 'output' must not be the same as 'source').
-- Option: -p , --python=
Add a ‘#!’ line to the archive specifying 'interpreter' as the
command to run. Also, on POSIX, make the archive executable. The
default is to write no ‘#!’ line, and not make the file executable.
-- Option: -m , --main=
Write a ‘__main__.py’ file to the archive that executes 'mainfn'.
The 'mainfn' argument should have the form “pkg.mod:fn”, where
“pkg.mod” is a package/module in the archive, and “fn” is a
callable in the given module. The ‘__main__.py’ file will execute
that callable.
*note -main: 4355. cannot be specified when copying an archive.
-- Option: -c, --compress
Compress files with the deflate method, reducing the size of the
output file. By default, files are stored uncompressed in the
archive.
*note -compress: 4357. has no effect when copying an archive.
Added in version 3.7.
-- Option: --info
Display the interpreter embedded in the archive, for diagnostic
purposes. In this case, any other options are ignored and SOURCE
must be an archive, not a directory.
-- Option: -h, --help
Print a short usage message and exit.
File: python.info, Node: Python API, Next: Examples<32>, Prev: Command-Line Interface<5>, Up: zipapp — Manage executable Python zip archives
5.29.3.3 Python API
...................
The module defines two convenience functions:
-- Function: zipapp.create_archive (source, target=None,
interpreter=None, main=None, filter=None, compressed=False)
Create an application archive from 'source'. The source can be any
of the following:
* The name of a directory, or a *note path-like object: 2a2.
referring to a directory, in which case a new application
archive will be created from the content of that directory.
* The name of an existing application archive file, or a *note
path-like object: 2a2. referring to such a file, in which case
the file is copied to the target (modifying it to reflect the
value given for the 'interpreter' argument). The file name
should include the ‘.pyz’ extension, if required.
* A file object open for reading in bytes mode. The content of
the file should be an application archive, and the file object
is assumed to be positioned at the start of the archive.
The 'target' argument determines where the resulting archive will
be written:
* If it is the name of a file, or a *note path-like object: 2a2,
the archive will be written to that file.
* If it is an open file object, the archive will be written to
that file object, which must be open for writing in bytes
mode.
* If the target is omitted (or ‘None’), the source must be a
directory and the target will be a file with the same name as
the source, with a ‘.pyz’ extension added.
The 'interpreter' argument specifies the name of the Python
interpreter with which the archive will be executed. It is written
as a “shebang” line at the start of the archive. On POSIX, this
will be interpreted by the OS, and on Windows it will be handled by
the Python launcher. Omitting the 'interpreter' results in no
shebang line being written. If an interpreter is specified, and
the target is a filename, the executable bit of the target file
will be set.
The 'main' argument specifies the name of a callable which will be
used as the main program for the archive. It can only be specified
if the source is a directory, and the source does not already
contain a ‘__main__.py’ file. The 'main' argument should take the
form “pkg.module:callable” and the archive will be run by importing
“pkg.module” and executing the given callable with no arguments.
It is an error to omit 'main' if the source is a directory and does
not contain a ‘__main__.py’ file, as otherwise the resulting
archive would not be executable.
The optional 'filter' argument specifies a callback function that
is passed a Path object representing the path to the file being
added (relative to the source directory). It should return ‘True’
if the file is to be added.
The optional 'compressed' argument determines whether files are
compressed. If set to ‘True’, files in the archive are compressed
with the deflate method; otherwise, files are stored uncompressed.
This argument has no effect when copying an existing archive.
If a file object is specified for 'source' or 'target', it is the
caller’s responsibility to close it after calling create_archive.
When copying an existing archive, file objects supplied only need
‘read’ and ‘readline’, or ‘write’ methods. When creating an
archive from a directory, if the target is a file object it will be
passed to the ‘zipfile.ZipFile’ class, and must supply the methods
needed by that class.
Changed in version 3.7: Added the 'filter' and 'compressed'
parameters.
-- Function: zipapp.get_interpreter (archive)
Return the interpreter specified in the ‘#!’ line at the start of
the archive. If there is no ‘#!’ line, return *note None: 671.
The 'archive' argument can be a filename or a file-like object open
for reading in bytes mode. It is assumed to be at the start of the
archive.
File: python.info, Node: Examples<32>, Next: Specifying the Interpreter, Prev: Python API, Up: zipapp — Manage executable Python zip archives
5.29.3.4 Examples
.................
Pack up a directory into an archive, and run it.
$ python -m zipapp myapp
$ python myapp.pyz
The same can be done using the *note create_archive(): bb0. function:
>>> import zipapp
>>> zipapp.create_archive('myapp', 'myapp.pyz')
To make the application directly executable on POSIX, specify an
interpreter to use.
$ python -m zipapp myapp -p "/usr/bin/env python"
$ ./myapp.pyz
To replace the shebang line on an existing archive, create a modified
archive using the *note create_archive(): bb0. function:
>>> import zipapp
>>> zipapp.create_archive('old_archive.pyz', 'new_archive.pyz', '/usr/bin/python3')
To update the file in place, do the replacement in memory using a *note
BytesIO: e9d. object, and then overwrite the source afterwards. Note
that there is a risk when overwriting a file in place that an error will
result in the loss of the original file. This code does not protect
against such errors, but production code should do so. Also, this
method will only work if the archive fits in memory:
>>> import zipapp
>>> import io
>>> temp = io.BytesIO()
>>> zipapp.create_archive('myapp.pyz', temp, '/usr/bin/python2')
>>> with open('myapp.pyz', 'wb') as f:
>>> f.write(temp.getvalue())
File: python.info, Node: Specifying the Interpreter, Next: Creating Standalone Applications with zipapp, Prev: Examples<32>, Up: zipapp — Manage executable Python zip archives
5.29.3.5 Specifying the Interpreter
...................................
Note that if you specify an interpreter and then distribute your
application archive, you need to ensure that the interpreter used is
portable. The Python launcher for Windows supports most common forms of
POSIX ‘#!’ line, but there are other issues to consider:
* If you use “/usr/bin/env python” (or other forms of the “python”
command, such as “/usr/bin/python”), you need to consider that your
users may have either Python 2 or Python 3 as their default, and
write your code to work under both versions.
* If you use an explicit version, for example “/usr/bin/env python3”
your application will not work for users who do not have that
version. (This may be what you want if you have not made your code
Python 2 compatible).
* There is no way to say “python X.Y or later”, so be careful of
using an exact version like “/usr/bin/env python3.4” as you will
need to change your shebang line for users of Python 3.5, for
example.
Typically, you should use an “/usr/bin/env python2” or “/usr/bin/env
python3”, depending on whether your code is written for Python 2 or 3.
File: python.info, Node: Creating Standalone Applications with zipapp, Next: The Python Zip Application Archive Format, Prev: Specifying the Interpreter, Up: zipapp — Manage executable Python zip archives
5.29.3.6 Creating Standalone Applications with zipapp
.....................................................
Using the *note zipapp: 130. module, it is possible to create
self-contained Python programs, which can be distributed to end users
who only need to have a suitable version of Python installed on their
system. The key to doing this is to bundle all of the application’s
dependencies into the archive, along with the application code.
The steps to create a standalone archive are as follows:
1. Create your application in a directory as normal, so you have a
‘myapp’ directory containing a ‘__main__.py’ file, and any
supporting application code.
2. Install all of your application’s dependencies into the ‘myapp’
directory, using pip:
$ python -m pip install -r requirements.txt --target myapp
(this assumes you have your project requirements in a
‘requirements.txt’ file - if not, you can just list the
dependencies manually on the pip command line).
3. Package the application using:
$ python -m zipapp -p "interpreter" myapp
This will produce a standalone executable, which can be run on any
machine with the appropriate interpreter available. See *note
Specifying the Interpreter: 4361. for details. It can be shipped to
users as a single file.
On Unix, the ‘myapp.pyz’ file is executable as it stands. You can
rename the file to remove the ‘.pyz’ extension if you prefer a “plain”
command name. On Windows, the ‘myapp.pyz[w]’ file is executable by
virtue of the fact that the Python interpreter registers the ‘.pyz’ and
‘.pyzw’ file extensions when installed.
* Menu:
* Caveats::
File: python.info, Node: Caveats, Up: Creating Standalone Applications with zipapp
5.29.3.7 Caveats
................
If your application depends on a package that includes a C extension,
that package cannot be run from a zip file (this is an OS limitation, as
executable code must be present in the filesystem for the OS loader to
load it). In this case, you can exclude that dependency from the
zipfile, and either require your users to have it installed, or ship it
alongside your zipfile and add code to your ‘__main__.py’ to include the
directory containing the unzipped module in ‘sys.path’. In this case,
you will need to make sure to ship appropriate binaries for your target
architecture(s) (and potentially pick the correct version to add to
‘sys.path’ at runtime, based on the user’s machine).
File: python.info, Node: The Python Zip Application Archive Format, Prev: Creating Standalone Applications with zipapp, Up: zipapp — Manage executable Python zip archives
5.29.3.8 The Python Zip Application Archive Format
..................................................
Python has been able to execute zip files which contain a ‘__main__.py’
file since version 2.6. In order to be executed by Python, an
application archive simply has to be a standard zip file containing a
‘__main__.py’ file which will be run as the entry point for the
application. As usual for any Python script, the parent of the script
(in this case the zip file) will be placed on *note sys.path: 3b0. and
thus further modules can be imported from the zip file.
The zip file format allows arbitrary data to be prepended to a zip file.
The zip application format uses this ability to prepend a standard POSIX
“shebang” line to the file (‘#!/path/to/interpreter’).
Formally, the Python zip application format is therefore:
1. An optional shebang line, containing the characters ‘b'#!'’
followed by an interpreter name, and then a newline (‘b'\n'’)
character. The interpreter name can be anything acceptable to the
OS “shebang” processing, or the Python launcher on Windows. The
interpreter should be encoded in UTF-8 on Windows, and in *note
sys.getfilesystemencoding(): c59. on POSIX.
2. Standard zipfile data, as generated by the *note zipfile: 131.
module. The zipfile content 'must' include a file called
‘__main__.py’ (which must be in the “root” of the zipfile - i.e.,
it cannot be in a subdirectory). The zipfile data can be
compressed or uncompressed.
If an application archive has a shebang line, it may have the executable
bit set on POSIX systems, to allow it to be executed directly.
There is no requirement that the tools in this module are used to create
application archives - the module is a convenience, but archives in the
above format created by any means are acceptable to Python.
File: python.info, Node: Python Runtime Services, Next: Custom Python Interpreters, Prev: Software Packaging and Distribution, Up: The Python Standard Library
5.30 Python Runtime Services
============================
The modules described in this chapter provide a wide range of services
related to the Python interpreter and its interaction with its
environment. Here’s an overview:
* Menu:
* sys — System-specific parameters and functions::
* sys.monitoring — Execution event monitoring: sys monitoring — Execution event monitoring.
* sysconfig — Provide access to Python’s configuration information::
* builtins — Built-in objects::
* __main__ — Top-level code environment::
* warnings — Warning control::
* dataclasses — Data Classes::
* contextlib — Utilities for with-statement contexts::
* abc — Abstract Base Classes::
* atexit — Exit handlers::
* traceback — Print or retrieve a stack traceback::
* __future__ — Future statement definitions::
* gc — Garbage Collector interface::
* inspect — Inspect live objects::
* site — Site-specific configuration hook::
File: python.info, Node: sys — System-specific parameters and functions, Next: sys monitoring — Execution event monitoring, Up: Python Runtime Services
5.30.1 ‘sys’ — System-specific parameters and functions
-------------------------------------------------------
__________________________________________________________________
This module provides access to some variables used or maintained by the
interpreter and to functions that interact strongly with the
interpreter. It is always available. Unless explicitly noted
otherwise, all variables are read-only.
-- Data: sys.abiflags
On POSIX systems where Python was built with the standard
‘configure’ script, this contains the ABI flags as specified by PEP
3149(1).
Added in version 3.2.
Changed in version 3.8: Default flags became an empty string (‘m’
flag for pymalloc has been removed).
*note Availability: 1d54.: Unix.
-- Function: sys.addaudithook (hook)
Append the callable 'hook' to the list of active auditing hooks for
the current (sub)interpreter.
When an auditing event is raised through the *note sys.audit():
15b9. function, each hook will be called in the order it was added
with the event name and the tuple of arguments. Native hooks added
by *note PySys_AddAuditHook(): 41e2. are called first, followed by
hooks added in the current (sub)interpreter. Hooks can then log
the event, raise an exception to abort the operation, or terminate
the process entirely.
Note that audit hooks are primarily for collecting information
about internal or otherwise unobservable actions, whether by Python
or libraries written in Python. They are not suitable for
implementing a “sandbox”. In particular, malicious code can
trivially disable or bypass hooks added using this function. At a
minimum, any security-sensitive hooks must be added using the C API
*note PySys_AddAuditHook(): 41e2. before initialising the runtime,
and any modules allowing arbitrary memory modification (such as
*note ctypes: 2a.) should be completely removed or closely
monitored.
Calling *note sys.addaudithook(): 41e1. will itself raise an
auditing event named ‘sys.addaudithook’ with no arguments. If any
existing hooks raise an exception derived from *note RuntimeError:
195, the new hook will not be added and the exception suppressed.
As a result, callers cannot assume that their hook has been added
unless they control all existing hooks.
See the *note audit events table: 1870. for all events raised by
CPython, and PEP 578(2) for the original design discussion.
Added in version 3.8.
Changed in version 3.8.1: Exceptions derived from *note Exception:
9d9. but not *note RuntimeError: 195. are no longer suppressed.
'CPython implementation detail:' When tracing is enabled (see *note
settrace(): 14ca.), Python hooks are only traced if the callable
has a ‘__cantrace__’ member that is set to a true value.
Otherwise, trace functions will skip the hook.
-- Data: sys.argv
The list of command line arguments passed to a Python script.
‘argv[0]’ is the script name (it is operating system dependent
whether this is a full pathname or not). If the command was
executed using the *note -c: 5dc. command line option to the
interpreter, ‘argv[0]’ is set to the string ‘'-c'’. If no script
name was passed to the Python interpreter, ‘argv[0]’ is the empty
string.
To loop over the standard input, or the list of files given on the
command line, see the *note fileinput: 5b. module.
See also *note sys.orig_argv: 83c.
Note: On Unix, command line arguments are passed by bytes from
OS. Python decodes them with filesystem encoding and
“surrogateescape” error handler. When you need original
bytes, you can get it by ‘[os.fsencode(arg) for arg in
sys.argv]’.
-- Function: sys.audit (event, *args)
Raise an auditing event and trigger any active auditing hooks.
'event' is a string identifying the event, and 'args' may contain
optional arguments with more information about the event. The
number and types of arguments for a given event are considered a
public and stable API and should not be modified between releases.
For example, one auditing event is named ‘os.chdir’. This event
has one argument called 'path' that will contain the requested new
working directory.
*note sys.audit(): 15b9. will call the existing auditing hooks,
passing the event name and arguments, and will re-raise the first
exception from any hook. In general, if an exception is raised, it
should not be handled and the process should be terminated as
quickly as possible. This allows hook implementations to decide
how to respond to particular events: they can merely log the event
or abort the operation by raising an exception.
Hooks are added using the *note sys.addaudithook(): 41e1. or *note
PySys_AddAuditHook(): 41e2. functions.
The native equivalent of this function is *note PySys_Audit(): 36a.
Using the native function is preferred when possible.
See the *note audit events table: 1870. for all events raised by
CPython.
Added in version 3.8.
-- Data: sys.base_exec_prefix
Set during Python startup, before ‘site.py’ is run, to the same
value as *note exec_prefix: 3ae. If not running in a *note virtual
environment: 4331, the values will stay the same; if ‘site.py’
finds that a virtual environment is in use, the values of *note
prefix: 3b2. and *note exec_prefix: 3ae. will be changed to point
to the virtual environment, whereas *note base_prefix: 3eb. and
*note base_exec_prefix: 3ea. will remain pointing to the base
Python installation (the one which the virtual environment was
created from).
Added in version 3.3.
-- Data: sys.base_prefix
Set during Python startup, before ‘site.py’ is run, to the same
value as *note prefix: 3b2. If not running in a *note virtual
environment: 4331, the values will stay the same; if ‘site.py’
finds that a virtual environment is in use, the values of *note
prefix: 3b2. and *note exec_prefix: 3ae. will be changed to point
to the virtual environment, whereas *note base_prefix: 3eb. and
*note base_exec_prefix: 3ea. will remain pointing to the base
Python installation (the one which the virtual environment was
created from).
Added in version 3.3.
-- Data: sys.byteorder
An indicator of the native byte order. This will have the value
‘'big'’ on big-endian (most-significant byte first) platforms, and
‘'little'’ on little-endian (least-significant byte first)
platforms.
-- Data: sys.builtin_module_names
A tuple of strings containing the names of all modules that are
compiled into this Python interpreter. (This information is not
available in any other way — ‘modules.keys()’ only lists the
imported modules.)
See also the *note sys.stdlib_module_names: 83d. list.
-- Function: sys.call_tracing (func, args)
Call ‘func(*args)’, while tracing is enabled. The tracing state is
saved, and restored afterwards. This is intended to be called from
a debugger from a checkpoint, to recursively debug or profile some
other code.
Tracing is suspended while calling a tracing function set by *note
settrace(): 14ca. or *note setprofile(): 14c9. to avoid infinite
recursion. ‘call_tracing()’ enables explicit recursion of the
tracing function.
-- Data: sys.copyright
A string containing the copyright pertaining to the Python
interpreter.
-- Function: sys._clear_type_cache ()
Clear the internal type cache. The type cache is used to speed up
attribute and method lookups. Use the function 'only' to drop
unnecessary references during reference leak debugging.
This function should be used for internal and specialized purposes
only.
Deprecated since version 3.13: Use the more general *note
_clear_internal_caches(): 1670. function instead.
-- Function: sys._clear_internal_caches ()
Clear all internal performance-related caches. Use this function
'only' to release unnecessary references and memory blocks when
hunting for leaks.
Added in version 3.13.
-- Function: sys._current_frames ()
Return a dictionary mapping each thread’s identifier to the topmost
stack frame currently active in that thread at the time the
function is called. Note that functions in the *note traceback:
fe. module can build the call stack given such a frame.
This is most useful for debugging deadlock: this function does not
require the deadlocked threads’ cooperation, and such threads’ call
stacks are frozen for as long as they remain deadlocked. The frame
returned for a non-deadlocked thread may bear no relationship to
that thread’s current activity by the time calling code examines
the frame.
This function should be used for internal and specialized purposes
only.
Raises an *note auditing event: 18ba. ‘sys._current_frames’ with no
arguments.
-- Function: sys._current_exceptions ()
Return a dictionary mapping each thread’s identifier to the topmost
exception currently active in that thread at the time the function
is called. If a thread is not currently handling an exception, it
is not included in the result dictionary.
This is most useful for statistical profiling.
This function should be used for internal and specialized purposes
only.
Raises an *note auditing event: 18ba. ‘sys._current_exceptions’
with no arguments.
Changed in version 3.12: Each value in the dictionary is now a
single exception instance, rather than a 3-tuple as returned from
‘sys.exc_info()’.
-- Function: sys.breakpointhook ()
This hook function is called by built-in *note breakpoint(): 236.
By default, it drops you into the *note pdb: a5. debugger, but it
can be set to any other function so that you can choose which
debugger gets used.
The signature of this function is dependent on what it calls. For
example, the default binding (e.g. ‘pdb.set_trace()’) expects no
arguments, but you might bind it to a function that expects
additional arguments (positional and/or keyword). The built-in
‘breakpoint()’ function passes its ‘*args’ and ‘**kws’ straight
through. Whatever ‘breakpointhooks()’ returns is returned from
‘breakpoint()’.
The default implementation first consults the environment variable
*note PYTHONBREAKPOINT: adb. If that is set to ‘"0"’ then this
function returns immediately; i.e. it is a no-op. If the
environment variable is not set, or is set to the empty string,
‘pdb.set_trace()’ is called. Otherwise this variable should name a
function to run, using Python’s dotted-import nomenclature, e.g.
‘package.subpackage.module.function’. In this case,
‘package.subpackage.module’ would be imported and the resulting
module must have a callable named ‘function()’. This is run,
passing in ‘*args’ and ‘**kws’, and whatever ‘function()’ returns,
‘sys.breakpointhook()’ returns to the built-in *note breakpoint():
236. function.
Note that if anything goes wrong while importing the callable named
by *note PYTHONBREAKPOINT: adb, a *note RuntimeWarning: 1bd. is
reported and the breakpoint is ignored.
Also note that if ‘sys.breakpointhook()’ is overridden
programmatically, *note PYTHONBREAKPOINT: adb. is 'not' consulted.
Added in version 3.7.
-- Function: sys._debugmallocstats ()
Print low-level information to stderr about the state of CPython’s
memory allocator.
If Python is *note built in debug mode: 1fb. (*note configure
-with-pydebug option: 420.), it also performs some expensive
internal consistency checks.
Added in version 3.3.
'CPython implementation detail:' This function is specific to
CPython. The exact output format is not defined here, and may
change.
-- Data: sys.dllhandle
Integer specifying the handle of the Python DLL.
*note Availability: 1d54.: Windows.
-- Function: sys.displayhook (value)
If 'value' is not ‘None’, this function prints ‘repr(value)’ to
‘sys.stdout’, and saves 'value' in ‘builtins._’. If ‘repr(value)’
is not encodable to ‘sys.stdout.encoding’ with ‘sys.stdout.errors’
error handler (which is probably ‘'strict'’), encode it to
‘sys.stdout.encoding’ with ‘'backslashreplace'’ error handler.
‘sys.displayhook’ is called on the result of evaluating an *note
expression: 1f85. entered in an interactive Python session. The
display of these values can be customized by assigning another
one-argument function to ‘sys.displayhook’.
Pseudo-code:
def displayhook(value):
if value is None:
return
# Set '_' to None to avoid recursion
builtins._ = None
text = repr(value)
try:
sys.stdout.write(text)
except UnicodeEncodeError:
bytes = text.encode(sys.stdout.encoding, 'backslashreplace')
if hasattr(sys.stdout, 'buffer'):
sys.stdout.buffer.write(bytes)
else:
text = bytes.decode(sys.stdout.encoding, 'strict')
sys.stdout.write(text)
sys.stdout.write("\n")
builtins._ = value
Changed in version 3.2: Use ‘'backslashreplace'’ error handler on
*note UnicodeEncodeError: 673.
-- Data: sys.dont_write_bytecode
If this is true, Python won’t try to write ‘.pyc’ files on the
import of source modules. This value is initially set to ‘True’ or
‘False’ depending on the *note -B: 1399. command line option and
the *note PYTHONDONTWRITEBYTECODE: 139a. environment variable, but
you can set it yourself to control bytecode file generation.
-- Data: sys._emscripten_info
A *note named tuple: 64a. holding information about the environment
on the 'wasm32-emscripten' platform. The named tuple is
provisional and may change in the future.
-- Attribute: _emscripten_info.emscripten_version
Emscripten version as tuple of ints (major, minor, micro),
e.g. ‘(3, 1, 8)’.
-- Attribute: _emscripten_info.runtime
Runtime string, e.g. browser user agent, ‘'Node.js
v14.18.2'’, or ‘'UNKNOWN'’.
-- Attribute: _emscripten_info.pthreads
‘True’ if Python is compiled with Emscripten pthreads support.
-- Attribute: _emscripten_info.shared_memory
‘True’ if Python is compiled with shared memory support.
*note Availability: 1d54.: Emscripten.
Added in version 3.11.
-- Data: sys.pycache_prefix
If this is set (not ‘None’), Python will write bytecode-cache
‘.pyc’ files to (and read them from) a parallel directory tree
rooted at this directory, rather than from ‘__pycache__’
directories in the source code tree. Any ‘__pycache__’ directories
in the source code tree will be ignored and new ‘.pyc’ files
written within the pycache prefix. Thus if you use *note
compileall: 20. as a pre-build step, you must ensure you run it
with the same pycache prefix (if any) that you will use at runtime.
A relative path is interpreted relative to the current working
directory.
This value is initially set based on the value of the *note -X:
176. ‘pycache_prefix=PATH’ command-line option or the *note
PYTHONPYCACHEPREFIX: 9a3. environment variable (command-line takes
precedence). If neither are set, it is ‘None’.
Added in version 3.8.
-- Function: sys.excepthook (type, value, traceback)
This function prints out a given traceback and exception to
‘sys.stderr’.
When an exception other than *note SystemExit: d40. is raised and
uncaught, the interpreter calls ‘sys.excepthook’ with three
arguments, the exception class, exception instance, and a traceback
object. In an interactive session this happens just before control
is returned to the prompt; in a Python program this happens just
before the program exits. The handling of such top-level
exceptions can be customized by assigning another three-argument
function to ‘sys.excepthook’.
Raise an auditing event ‘sys.excepthook’ with arguments ‘hook’,
‘type’, ‘value’, ‘traceback’ when an uncaught exception occurs. If
no hook has been set, ‘hook’ may be ‘None’. If any hook raises an
exception derived from *note RuntimeError: 195. the call to the
hook will be suppressed. Otherwise, the audit hook exception will
be reported as unraisable and ‘sys.excepthook’ will be called.
See also
........
The *note sys.unraisablehook(): 1f9. function handles unraisable
exceptions and the *note threading.excepthook(): 847. function
handles exception raised by *note threading.Thread.run(): a3c.
-- Data: sys.__breakpointhook__
-- Data: sys.__displayhook__
-- Data: sys.__excepthook__
-- Data: sys.__unraisablehook__
These objects contain the original values of ‘breakpointhook’,
‘displayhook’, ‘excepthook’, and ‘unraisablehook’ at the start of
the program. They are saved so that ‘breakpointhook’,
‘displayhook’ and ‘excepthook’, ‘unraisablehook’ can be restored in
case they happen to get replaced with broken or alternative
objects.
Added in version 3.7: __breakpointhook__
Added in version 3.8: __unraisablehook__
-- Function: sys.exception ()
This function, when called while an exception handler is executing
(such as an ‘except’ or ‘except*’ clause), returns the exception
instance that was caught by this handler. When exception handlers
are nested within one another, only the exception handled by the
innermost handler is accessible.
If no exception handler is executing, this function returns ‘None’.
Added in version 3.11.
-- Function: sys.exc_info ()
This function returns the old-style representation of the handled
exception. If an exception ‘e’ is currently handled (so *note
exception(): 687. would return ‘e’), *note exc_info(): 686. returns
the tuple ‘(type(e), e, e.__traceback__)’. That is, a tuple
containing the type of the exception (a subclass of *note
BaseException: 5b7.), the exception itself, and a *note traceback
object: af4. which typically encapsulates the call stack at the
point where the exception last occurred.
If no exception is being handled anywhere on the stack, this
function return a tuple containing three ‘None’ values.
Changed in version 3.11: The ‘type’ and ‘traceback’ fields are now
derived from the ‘value’ (the exception instance), so when an
exception is modified while it is being handled, the changes are
reflected in the results of subsequent calls to *note exc_info():
686.
-- Data: sys.exec_prefix
A string giving the site-specific directory prefix where the
platform-dependent Python files are installed; by default, this is
also ‘'/usr/local'’. This can be set at build time with the
‘--exec-prefix’ argument to the ‘configure’ script. Specifically,
all configuration files (e.g. the ‘pyconfig.h’ header file) are
installed in the directory ‘`exec_prefix'/lib/python`X.Y'/config’,
and shared library modules are installed in
‘`exec_prefix'/lib/python`X.Y'/lib-dynload’, where 'X.Y' is the
version number of Python, for example ‘3.2’.
Note: If a *note virtual environment: 4331. is in effect, this
value will be changed in ‘site.py’ to point to the virtual
environment. The value for the Python installation will still
be available, via *note base_exec_prefix: 3ea.
-- Data: sys.executable
A string giving the absolute path of the executable binary for the
Python interpreter, on systems where this makes sense. If Python
is unable to retrieve the real path to its executable, *note
sys.executable: 3b4. will be an empty string or ‘None’.
-- Function: sys.exit ([arg])
Raise a *note SystemExit: d40. exception, signaling an intention to
exit the interpreter.
The optional argument 'arg' can be an integer giving the exit
status (defaulting to zero), or another type of object. If it is
an integer, zero is considered “successful termination” and any
nonzero value is considered “abnormal termination” by shells and
the like. Most systems require it to be in the range 0–127, and
produce undefined results otherwise. Some systems have a
convention for assigning specific meanings to specific exit codes,
but these are generally underdeveloped; Unix programs generally use
2 for command line syntax errors and 1 for all other kind of
errors. If another type of object is passed, ‘None’ is equivalent
to passing zero, and any other object is printed to *note stderr:
939. and results in an exit code of 1. In particular,
‘sys.exit("some error message")’ is a quick way to exit a program
when an error occurs.
Since *note exit(): 2189. ultimately “only” raises an exception, it
will only exit the process when called from the main thread, and
the exception is not intercepted. Cleanup actions specified by
finally clauses of *note try: 6e4. statements are honored, and it
is possible to intercept the exit attempt at an outer level.
Changed in version 3.6: If an error occurs in the cleanup after the
Python interpreter has caught *note SystemExit: d40. (such as an
error flushing buffered data in the standard streams), the exit
status is changed to 120.
-- Data: sys.flags
The *note named tuple: 64a. 'flags' exposes the status of command
line flags. The attributes are read only.
-- Attribute: flags.debug *note -d: 1d31.
-- Attribute: flags.inspect *note -i: 14a6.
-- Attribute: flags.interactive *note -i: 14a6.
-- Attribute: flags.isolated *note -I: 95e.
-- Attribute: flags.optimize *note -O: db4. or *note -OO: db5.
-- Attribute: flags.dont_write_bytecode *note -B: 1399.
-- Attribute: flags.no_user_site *note -s: 1377.
-- Attribute: flags.no_site *note -S: 119b.
-- Attribute: flags.ignore_environment *note -E: 95d.
-- Attribute: flags.verbose *note -v: 13f0.
-- Attribute: flags.bytes_warning *note -b: 5df.
-- Attribute: flags.quiet *note -q: 1d34.
-- Attribute: flags.hash_randomization *note -R: 1a96.
-- Attribute: flags.dev_mode *note -X dev: 176.
(*note Python Development Mode: 1fa.)
-- Attribute: flags.utf8_mode *note -X utf8: 176.
-- Attribute: flags.safe_path *note -P: 5a0.
-- Attribute: flags.int_max_str_digits *note -X int_max_str_digits: 176.
(*note integer string conversion length limitation: 5f1.)
-- Attribute: flags.warn_default_encoding *note -X warn_default_encoding: 176.
Changed in version 3.2: Added ‘quiet’ attribute for the new *note
-q: 1d34. flag.
Added in version 3.2.3: The ‘hash_randomization’ attribute.
Changed in version 3.3: Removed obsolete ‘division_warning’
attribute.
Changed in version 3.4: Added ‘isolated’ attribute for *note -I:
95e. ‘isolated’ flag.
Changed in version 3.7: Added the ‘dev_mode’ attribute for the new
*note Python Development Mode: 1fa. and the ‘utf8_mode’ attribute
for the new *note -X: 176. ‘utf8’ flag.
Changed in version 3.10: Added ‘warn_default_encoding’ attribute
for *note -X: 176. ‘warn_default_encoding’ flag.
Changed in version 3.11: Added the ‘safe_path’ attribute for *note
-P: 5a0. option.
Changed in version 3.11: Added the ‘int_max_str_digits’ attribute.
-- Data: sys.float_info
A *note named tuple: 64a. holding information about the float type.
It contains low level information about the precision and internal
representation. The values correspond to the various
floating-point constants defined in the standard header file
‘float.h’ for the ‘C’ programming language; see section 5.2.4.2.2
of the 1999 ISO/IEC C standard *note [C99]: 4385, ‘Characteristics
of floating types’, for details. Attributes of the ‘float_info’
named tuple
attribute float.h macro explanation
------------------------------------------------------------------------------------------------------------------
-- Attribute: float_info.epsilon ‘DBL_EPSILON’ difference between 1.0 and the
least value greater than 1.0 that
is representable as a float.
See also *note math.ulp(): 911.
-- Attribute: float_info.dig ‘DBL_DIG’ The maximum number of decimal
digits that can be faithfully
represented in a float; see below.
-- Attribute: float_info.mant_dig ‘DBL_MANT_DIG’ Float precision: the number of
base-‘radix’ digits in the
significand of a float.
-- Attribute: float_info.max ‘DBL_MAX’ The maximum representable positive
finite float.
-- Attribute: float_info.max_exp ‘DBL_MAX_EXP’ The maximum integer 'e' such that
‘radix**(e-1)’ is a representable
finite float.
-- Attribute: ‘DBL_MAX_10_EXP’ The maximum integer 'e' such that
float_info.max_10_exp ‘10**e’ is in the range of
representable finite floats.
-- Attribute: float_info.min ‘DBL_MIN’ The minimum representable positive
'normalized' float.
Use *note math.ulp(0.0): 911. to
get the smallest positive
'denormalized' representable float.
-- Attribute: float_info.min_exp ‘DBL_MIN_EXP’ The minimum integer 'e' such that
‘radix**(e-1)’ is a normalized
float.
-- Attribute: ‘DBL_MIN_10_EXP’ The minimum integer 'e' such that
float_info.min_10_exp ‘10**e’ is a normalized float.
-- Attribute: float_info.radix ‘FLT_RADIX’ The radix of exponent
representation.
-- Attribute: float_info.rounds ‘FLT_ROUNDS’ An integer representing the
rounding mode for floating-point
arithmetic. This reflects the
value of the system ‘FLT_ROUNDS’
macro at interpreter startup time:
* ‘-1’: indeterminable
* ‘0’: toward zero
* ‘1’: to nearest
* ‘2’: toward positive infinity
* ‘3’: toward negative infinity
All other values for ‘FLT_ROUNDS’
characterize implementation-defined
rounding behavior.
The attribute *note sys.float_info.dig: 4387. needs further
explanation. If ‘s’ is any string representing a decimal number
with at most ‘sys.float_info.dig’ significant digits, then
converting ‘s’ to a float and back again will recover a string
representing the same decimal value:
>>> import sys
>>> sys.float_info.dig
15
>>> s = '3.14159265358979' # decimal string with 15 significant digits
>>> format(float(s), '.15g') # convert to float and back -> same value
'3.14159265358979'
But for strings with more than *note sys.float_info.dig: 4387.
significant digits, this isn’t always true:
>>> s = '9876543211234567' # 16 significant digits is too many!
>>> format(float(s), '.16g') # conversion changes value
'9876543211234568'
-- Data: sys.float_repr_style
A string indicating how the *note repr(): 7f9. function behaves for
floats. If the string has value ‘'short'’ then for a finite float
‘x’, ‘repr(x)’ aims to produce a short string with the property
that ‘float(repr(x)) == x’. This is the usual behaviour in Python
3.1 and later. Otherwise, ‘float_repr_style’ has value ‘'legacy'’
and ‘repr(x)’ behaves in the same way as it did in versions of
Python prior to 3.1.
Added in version 3.1.
-- Function: sys.getallocatedblocks ()
Return the number of memory blocks currently allocated by the
interpreter, regardless of their size. This function is mainly
useful for tracking and debugging memory leaks. Because of the
interpreter’s internal caches, the result can vary from call to
call; you may have to call *note _clear_internal_caches(): 1670.
and *note gc.collect(): a39. to get more predictable results.
If a Python build or implementation cannot reasonably compute this
information, *note getallocatedblocks(): fc0. is allowed to return
0 instead.
Added in version 3.4.
-- Function: sys.getunicodeinternedsize ()
Return the number of unicode objects that have been interned.
Added in version 3.12.
-- Function: sys.getandroidapilevel ()
Return the build-time API level of Android as an integer. This
represents the minimum version of Android this build of Python can
run on. For runtime version information, see *note
platform.android_ver(): 164c.
*note Availability: 1d54.: Android.
Added in version 3.7.
-- Function: sys.getdefaultencoding ()
Return ‘'utf-8'’. This is the name of the default string encoding,
used in methods like *note str.encode(): 8d4.
-- Function: sys.getdlopenflags ()
Return the current value of the flags that are used for ‘dlopen()’
calls. Symbolic names for the flag values can be found in the
*note os: a1. module (‘RTLD_XXX’ constants, e.g. *note
os.RTLD_LAZY: 110c.).
*note Availability: 1d54.: Unix.
-- Function: sys.getfilesystemencoding ()
Get the *note filesystem encoding: 537.: the encoding used with the
*note filesystem error handler: 537. to convert between Unicode
filenames and bytes filenames. The filesystem error handler is
returned from *note getfilesystemencodeerrors(): ce6.
For best compatibility, str should be used for filenames in all
cases, although representing filenames as bytes is also supported.
Functions accepting or returning filenames should support either
str or bytes and internally convert to the system’s preferred
representation.
*note os.fsencode(): c55. and *note os.fsdecode(): c54. should be
used to ensure that the correct encoding and errors mode are used.
The *note filesystem encoding and error handler: 537. are
configured at Python startup by the *note PyConfig_Read(): 78b.
function: see *note filesystem_encoding: 3e4. and *note
filesystem_errors: 3e5. members of *note PyConfig: 3a2.
Changed in version 3.2: *note getfilesystemencoding(): c59. result
cannot be ‘None’ anymore.
Changed in version 3.6: Windows is no longer guaranteed to return
‘'mbcs'’. See PEP 529(3) and *note
_enablelegacywindowsfsencoding(): 2b0. for more information.
Changed in version 3.7: Return ‘'utf-8'’ if the *note Python UTF-8
Mode: 652. is enabled.
-- Function: sys.getfilesystemencodeerrors ()
Get the *note filesystem error handler: 537.: the error handler
used with the *note filesystem encoding: 537. to convert between
Unicode filenames and bytes filenames. The filesystem encoding is
returned from *note getfilesystemencoding(): c59.
*note os.fsencode(): c55. and *note os.fsdecode(): c54. should be
used to ensure that the correct encoding and errors mode are used.
The *note filesystem encoding and error handler: 537. are
configured at Python startup by the *note PyConfig_Read(): 78b.
function: see *note filesystem_encoding: 3e4. and *note
filesystem_errors: 3e5. members of *note PyConfig: 3a2.
Added in version 3.6.
-- Function: sys.get_int_max_str_digits ()
Returns the current value for the *note integer string conversion
length limitation: 5f1. See also *note set_int_max_str_digits():
17be.
Added in version 3.11.
-- Function: sys.getrefcount (object)
Return the reference count of the 'object'. The count returned is
generally one higher than you might expect, because it includes the
(temporary) reference as an argument to *note getrefcount(): 4393.
Note that the returned value may not actually reflect how many
references to the object are actually held. For example, some
objects are *note immortal: 4394. and have a very high refcount
that does not reflect the actual number of references.
Consequently, do not rely on the returned value to be accurate,
other than a value of 0 or 1.
Changed in version 3.12: Immortal objects have very large refcounts
that do not match the actual number of references to the object.
-- Function: sys.getrecursionlimit ()
Return the current value of the recursion limit, the maximum depth
of the Python interpreter stack. This limit prevents infinite
recursion from causing an overflow of the C stack and crashing
Python. It can be set by *note setrecursionlimit(): 4bf.
-- Function: sys.getsizeof (object[, default])
Return the size of an object in bytes. The object can be any type
of object. All built-in objects will return correct results, but
this does not have to hold true for third-party extensions as it is
implementation specific.
Only the memory consumption directly attributed to the object is
accounted for, not the memory consumption of objects it refers to.
If given, 'default' will be returned if the object does not provide
means to retrieve the size. Otherwise a *note TypeError: 534. will
be raised.
*note getsizeof(): 176a. calls the object’s ‘__sizeof__’ method and
adds an additional garbage collector overhead if the object is
managed by the garbage collector.
See recursive sizeof recipe(4) for an example of using *note
getsizeof(): 176a. recursively to find the size of containers and
all their contents.
-- Function: sys.getswitchinterval ()
Return the interpreter’s “thread switch interval” in seconds; see
*note setswitchinterval(): 94a.
Added in version 3.2.
-- Function: sys._getframe ([depth])
Return a frame object from the call stack. If optional integer
'depth' is given, return the frame object that many calls below the
top of the stack. If that is deeper than the call stack, *note
ValueError: 204. is raised. The default for 'depth' is zero,
returning the frame at the top of the call stack.
Raises an *note auditing event: 18ba. ‘sys._getframe’ with argument
‘frame’.
'CPython implementation detail:' This function should be used for
internal and specialized purposes only. It is not guaranteed to
exist in all implementations of Python.
-- Function: sys._getframemodulename ([depth])
Return the name of a module from the call stack. If optional
integer 'depth' is given, return the module that many calls below
the top of the stack. If that is deeper than the call stack, or if
the module is unidentifiable, ‘None’ is returned. The default for
'depth' is zero, returning the module at the top of the call stack.
Raises an *note auditing event: 18ba. ‘sys._getframemodulename’
with argument ‘depth’.
'CPython implementation detail:' This function should be used for
internal and specialized purposes only. It is not guaranteed to
exist in all implementations of Python.
Added in version 3.12.
-- Function: sys.getobjects (limit[, type])
This function only exists if CPython was built using the
specialized configure option *note -with-trace-refs: 390. It is
intended only for debugging garbage-collection issues.
Return a list of up to 'limit' dynamically allocated Python
objects. If 'type' is given, only objects of that exact type (not
subtypes) are included.
Objects from the list are not safe to use. Specifically, the
result will include objects from all interpreters that share their
object allocator state (that is, ones created with *note
PyInterpreterConfig.use_main_obmalloc: 4395. set to 1 or using
*note Py_NewInterpreter(): 17b6, and the *note main interpreter:
584.). Mixing objects from different interpreters may lead to
crashes or other unexpected behavior.
'CPython implementation detail:' This function should be used for
specialized purposes only. It is not guaranteed to exist in all
implementations of Python.
Changed in version 3.13.1: The result may include objects from
other interpreters.
-- Function: sys.getprofile ()
Get the profiler function as set by *note setprofile(): 14c9.
-- Function: sys.gettrace ()
Get the trace function as set by *note settrace(): 14ca.
'CPython implementation detail:' The *note gettrace(): 13ad.
function is intended only for implementing debuggers, profilers,
coverage tools and the like. Its behavior is part of the
implementation platform, rather than part of the language
definition, and thus may not be available in all Python
implementations.
-- Function: sys.getwindowsversion ()
Return a named tuple describing the Windows version currently
running. The named elements are 'major', 'minor', 'build',
'platform', 'service_pack', 'service_pack_minor',
'service_pack_major', 'suite_mask', 'product_type' and
'platform_version'. 'service_pack' contains a string,
'platform_version' a 3-tuple and all other values are integers.
The components can also be accessed by name, so
‘sys.getwindowsversion()[0]’ is equivalent to
‘sys.getwindowsversion().major’. For compatibility with prior
versions, only the first 5 elements are retrievable by indexing.
'platform' will be ‘2’ (VER_PLATFORM_WIN32_NT).
'product_type' may be one of the following values:
Constant Meaning
----------------------------------------------------------------------------------
‘1’ (VER_NT_WORKSTATION) The system is a workstation.
‘2’ (VER_NT_DOMAIN_CONTROLLER) The system is a domain controller.
‘3’ (VER_NT_SERVER) The system is a server, but not a
domain controller.
This function wraps the Win32 ‘GetVersionEx()’ function; see the
Microsoft documentation on ‘OSVERSIONINFOEX()’ for more information
about these fields.
'platform_version' returns the major version, minor version and
build number of the current operating system, rather than the
version that is being emulated for the process. It is intended for
use in logging rather than for feature detection.
Note: 'platform_version' derives the version from kernel32.dll
which can be of a different version than the OS version.
Please use *note platform: aa. module for achieving accurate
OS version.
*note Availability: 1d54.: Windows.
Changed in version 3.2: Changed to a named tuple and added
'service_pack_minor', 'service_pack_major', 'suite_mask', and
'product_type'.
Changed in version 3.6: Added 'platform_version'
-- Function: sys.get_asyncgen_hooks ()
Returns an 'asyncgen_hooks' object, which is similar to a *note
namedtuple: 1ca. of the form ‘(firstiter, finalizer)’, where
'firstiter' and 'finalizer' are expected to be either ‘None’ or
functions which take an *note asynchronous generator iterator:
4396. as an argument, and are used to schedule finalization of an
asynchronous generator by an event loop.
Added in version 3.6: See PEP 525(5) for more details.
Note: This function has been added on a provisional basis (see
PEP 411(6) for details.)
-- Function: sys.get_coroutine_origin_tracking_depth ()
Get the current coroutine origin tracking depth, as set by *note
set_coroutine_origin_tracking_depth(): b8c.
Added in version 3.7.
Note: This function has been added on a provisional basis (see
PEP 411(7) for details.) Use it only for debugging purposes.
-- Data: sys.hash_info
A *note named tuple: 64a. giving parameters of the numeric hash
implementation. For more details about hashing of numeric types,
see *note Hashing of numeric types: 21a0.
-- Attribute: hash_info.width
The width in bits used for hash values
-- Attribute: hash_info.modulus
The prime modulus P used for numeric hash scheme
-- Attribute: hash_info.inf
The hash value returned for a positive infinity
-- Attribute: hash_info.nan
(This attribute is no longer used)
-- Attribute: hash_info.imag
The multiplier used for the imaginary part of a complex number
-- Attribute: hash_info.algorithm
The name of the algorithm for hashing of str, bytes, and
memoryview
-- Attribute: hash_info.hash_bits
The internal output size of the hash algorithm
-- Attribute: hash_info.seed_bits
The size of the seed key of the hash algorithm
Added in version 3.2.
Changed in version 3.4: Added 'algorithm', 'hash_bits' and
'seed_bits'
-- Data: sys.hexversion
The version number encoded as a single integer. This is guaranteed
to increase with each version, including proper support for
non-production releases. For example, to test that the Python
interpreter is at least version 1.5.2, use:
if sys.hexversion >= 0x010502F0:
# use some advanced feature
...
else:
# use an alternative implementation or warn the user
...
This is called ‘hexversion’ since it only really looks meaningful
when viewed as the result of passing it to the built-in *note
hex(): 12c2. function. The *note named tuple: 64a. *note
sys.version_info: 69c. may be used for a more human-friendly
encoding of the same information.
More details of ‘hexversion’ can be found at *note API and ABI
Versioning: 439f.
-- Data: sys.implementation
An object containing information about the implementation of the
currently running Python interpreter. The following attributes are
required to exist in all Python implementations.
'name' is the implementation’s identifier, e.g. ‘'cpython'’. The
actual string is defined by the Python implementation, but it is
guaranteed to be lower case.
'version' is a named tuple, in the same format as *note
sys.version_info: 69c. It represents the version of the Python
'implementation'. This has a distinct meaning from the specific
version of the Python 'language' to which the currently running
interpreter conforms, which ‘sys.version_info’ represents. For
example, for PyPy 1.8 ‘sys.implementation.version’ might be
‘sys.version_info(1, 8, 0, 'final', 0)’, whereas ‘sys.version_info’
would be ‘sys.version_info(2, 7, 2, 'final', 0)’. For CPython they
are the same value, since it is the reference implementation.
'hexversion' is the implementation version in hexadecimal format,
like *note sys.hexversion: 439e.
'cache_tag' is the tag used by the import machinery in the
filenames of cached modules. By convention, it would be a
composite of the implementation’s name and version, like
‘'cpython-33'’. However, a Python implementation may use some
other value if appropriate. If ‘cache_tag’ is set to ‘None’, it
indicates that module caching should be disabled.
*note sys.implementation: 512. may contain additional attributes
specific to the Python implementation. These non-standard
attributes must start with an underscore, and are not described
here. Regardless of its contents, *note sys.implementation: 512.
will not change during a run of the interpreter, nor between
implementation versions. (It may change between Python language
versions, however.) See PEP 421(8) for more information.
Added in version 3.3.
Note: The addition of new required attributes must go through
the normal PEP process. See PEP 421(9) for more information.
-- Data: sys.int_info
A *note named tuple: 64a. that holds information about Python’s
internal representation of integers. The attributes are read only.
-- Attribute: int_info.bits_per_digit
The number of bits held in each digit. Python integers are
stored internally in base ‘2**int_info.bits_per_digit’.
-- Attribute: int_info.sizeof_digit
The size in bytes of the C type used to represent a digit.
-- Attribute: int_info.default_max_str_digits
The default value for *note sys.get_int_max_str_digits():
152d. when it is not otherwise explicitly configured.
-- Attribute: int_info.str_digits_check_threshold
The minimum non-zero value for *note
sys.set_int_max_str_digits(): 17be, *note
PYTHONINTMAXSTRDIGITS: 17bd, or *note -X int_max_str_digits:
176.
Added in version 3.1.
Changed in version 3.11: Added *note default_max_str_digits: 227b.
and *note str_digits_check_threshold: 43a2.
-- Data: sys.__interactivehook__
When this attribute exists, its value is automatically called (with
no arguments) when the interpreter is launched in *note interactive
mode: fc1. This is done after the *note PYTHONSTARTUP: fc3. file
is read, so that you can set this hook there. The *note site: c7.
module *note sets this: fc4.
Raises an *note auditing event: 18ba. ‘cpython.run_interactivehook’
with the hook object as the argument when the hook is called on
startup.
Added in version 3.4.
-- Function: sys.intern (string)
Enter 'string' in the table of “interned” strings and return the
interned string – which is 'string' itself or a copy. Interning
strings is useful to gain a little performance on dictionary lookup
– if the keys in a dictionary are interned, and the lookup key is
interned, the key comparisons (after hashing) can be done by a
pointer compare instead of a string compare. Normally, the names
used in Python programs are automatically interned, and the
dictionaries used to hold module, class or instance attributes have
interned keys.
Interned strings are not *note immortal: 4394.; you must keep a
reference to the return value of *note intern(): 12ce. around to
benefit from it.
-- Function: sys._is_gil_enabled ()
Return *note True: c0d. if the *note GIL: 159. is enabled and *note
False: b37. if it is disabled.
Added in version 3.13.
'CPython implementation detail:' It is not guaranteed to exist in
all implementations of Python.
-- Function: sys.is_finalizing ()
Return *note True: c0d. if the main Python interpreter is *note
shutting down: 14e. Return *note False: b37. otherwise.
See also the *note PythonFinalizationError: 14d. exception.
Added in version 3.5.
-- Data: sys.last_exc
This variable is not always defined; it is set to the exception
instance when an exception is not handled and the interpreter
prints an error message and a stack traceback. Its intended use is
to allow an interactive user to import a debugger module and engage
in post-mortem debugging without having to re-execute the command
that caused the error. (Typical use is ‘import pdb; pdb.pm()’ to
enter the post-mortem debugger; see *note pdb: a5. module for more
information.)
Added in version 3.12.
-- Function: sys._is_interned (string)
Return *note True: c0d. if the given string is “interned”, *note
False: b37. otherwise.
Added in version 3.13.
'CPython implementation detail:' It is not guaranteed to exist in
all implementations of Python.
-- Data: sys.last_type
-- Data: sys.last_value
-- Data: sys.last_traceback
These three variables are deprecated; use *note sys.last_exc: 4ba.
instead. They hold the legacy representation of ‘sys.last_exc’, as
returned from *note exc_info(): 686. above.
-- Data: sys.maxsize
An integer giving the maximum value a variable of type *note
Py_ssize_t: a5f. can take. It’s usually ‘2**31 - 1’ on a 32-bit
platform and ‘2**63 - 1’ on a 64-bit platform.
-- Data: sys.maxunicode
An integer giving the value of the largest Unicode code point, i.e.
‘1114111’ (‘0x10FFFF’ in hexadecimal).
Changed in version 3.3: Before PEP 393(10), ‘sys.maxunicode’ used
to be either ‘0xFFFF’ or ‘0x10FFFF’, depending on the configuration
option that specified whether Unicode characters were stored as
UCS-2 or UCS-4.
-- Data: sys.meta_path
A list of *note meta path finder: 1069. objects that have their
*note find_spec(): 865. methods called to see if one of the objects
can find the module to be imported. By default, it holds entries
that implement Python’s default import semantics. The *note
find_spec(): 865. method is called with at least the absolute name
of the module being imported. If the module to be imported is
contained in a package, then the parent package’s *note __path__:
1c6e. attribute is passed in as a second argument. The method
returns a *note module spec: 1f16, or ‘None’ if the module cannot
be found.
See also
........
*note importlib.abc.MetaPathFinder: 86b.
The abstract base class defining the interface of finder
objects on *note meta_path: d2b.
*note importlib.machinery.ModuleSpec: 1e64.
The concrete class which *note find_spec(): 865. should return
instances of.
Changed in version 3.4: *note Module specs: 1f16. were introduced
in Python 3.4, by PEP 451(11).
Changed in version 3.12: Removed the fallback that looked for a
‘find_module()’ method if a *note meta_path: d2b. entry didn’t have
a *note find_spec(): 865. method.
-- Data: sys.modules
This is a dictionary that maps module names to modules which have
already been loaded. This can be manipulated to force reloading of
modules and other tricks. However, replacing the dictionary will
not necessarily work as expected and deleting essential items from
the dictionary may cause Python to fail. If you want to iterate
over this global dictionary always use ‘sys.modules.copy()’ or
‘tuple(sys.modules)’ to avoid exceptions as its size may change
during iteration as a side effect of code or activity in other
threads.
-- Data: sys.orig_argv
The list of the original command line arguments passed to the
Python executable.
The elements of *note sys.orig_argv: 83c. are the arguments to the
Python interpreter, while the elements of *note sys.argv: 1258. are
the arguments to the user’s program. Arguments consumed by the
interpreter itself will be present in *note sys.orig_argv: 83c. and
missing from *note sys.argv: 1258.
Added in version 3.10.
-- Data: sys.path
A list of strings that specifies the search path for modules.
Initialized from the environment variable *note PYTHONPATH: 1016,
plus an installation-dependent default.
By default, as initialized upon program startup, a potentially
unsafe path is prepended to *note sys.path: 3b0. ('before' the
entries inserted as a result of *note PYTHONPATH: 1016.):
* ‘python -m module’ command line: prepend the current working
directory.
* ‘python script.py’ command line: prepend the script’s
directory. If it’s a symbolic link, resolve symbolic links.
* ‘python -c code’ and ‘python’ (REPL) command lines: prepend an
empty string, which means the current working directory.
To not prepend this potentially unsafe path, use the *note -P: 5a0.
command line option or the *note PYTHONSAFEPATH: 5a1. environment
variable.
A program is free to modify this list for its own purposes. Only
strings should be added to *note sys.path: 3b0.; all other data
types are ignored during import.
See also
........
* Module *note site: c7. This describes how to use .pth files to
extend *note sys.path: 3b0.
-- Data: sys.path_hooks
A list of callables that take a path argument to try to create a
*note finder: 1f1e. for the path. If a finder can be created, it
is to be returned by the callable, else raise *note ImportError:
415.
Originally specified in PEP 302(12).
-- Data: sys.path_importer_cache
A dictionary acting as a cache for *note finder: 1f1e. objects.
The keys are paths that have been passed to *note sys.path_hooks:
101d. and the values are the finders that are found. If a path is
a valid file system path but no finder is found on *note
sys.path_hooks: 101d. then ‘None’ is stored.
Originally specified in PEP 302(13).
-- Data: sys.platform
A string containing a platform identifier. Known values are:
System ‘platform’ value
-----------------------------------------------------
AIX ‘'aix'’
Android ‘'android'’
Emscripten ‘'emscripten'’
iOS ‘'ios'’
Linux ‘'linux'’
macOS ‘'darwin'’
Windows ‘'win32'’
Windows/Cygwin ‘'cygwin'’
WASI ‘'wasi'’
On Unix systems not listed in the table, the value is the
lowercased OS name as returned by ‘uname -s’, with the first part
of the version as returned by ‘uname -r’ appended, e.g. ‘'sunos5'’
or ‘'freebsd8'’, 'at the time when Python was built'. Unless you
want to test for a specific system version, it is therefore
recommended to use the following idiom:
if sys.platform.startswith('freebsd'):
# FreeBSD-specific code here...
Changed in version 3.3: On Linux, *note sys.platform: a8d. doesn’t
contain the major version anymore. It is always ‘'linux'’, instead
of ‘'linux2'’ or ‘'linux3'’.
Changed in version 3.8: On AIX, *note sys.platform: a8d. doesn’t
contain the major version anymore. It is always ‘'aix'’, instead
of ‘'aix5'’ or ‘'aix7'’.
Changed in version 3.13: On Android, *note sys.platform: a8d. now
returns ‘'android'’ rather than ‘'linux'’.
See also
........
*note os.name: 2b0e. has a coarser granularity. *note os.uname():
110a. gives system-dependent version information.
The *note platform: aa. module provides detailed checks for the
system’s identity.
-- Data: sys.platlibdir
Name of the platform-specific library directory. It is used to
build the path of standard library and the paths of installed
extension modules.
It is equal to ‘"lib"’ on most platforms. On Fedora and SuSE, it
is equal to ‘"lib64"’ on 64-bit platforms which gives the following
‘sys.path’ paths (where ‘X.Y’ is the Python ‘major.minor’ version):
* ‘/usr/lib64/pythonX.Y/’: Standard library (like ‘os.py’ of the
*note os: a1. module)
* ‘/usr/lib64/pythonX.Y/lib-dynload/’: C extension modules of
the standard library (like the *note errno: 57. module, the
exact filename is platform specific)
* ‘/usr/lib/pythonX.Y/site-packages/’ (always use ‘lib’, not
*note sys.platlibdir: 938.): Third-party modules
* ‘/usr/lib64/pythonX.Y/site-packages/’: C extension modules of
third-party packages
Added in version 3.9.
-- Data: sys.prefix
A string giving the site-specific directory prefix where the
platform independent Python files are installed; on Unix, the
default is ‘/usr/local’. This can be set at build time with the
*note -prefix: 1d62. argument to the ‘configure’ script. See *note
Installation paths: 68b. for derived paths.
Note: If a *note virtual environment: 4331. is in effect, this
value will be changed in ‘site.py’ to point to the virtual
environment. The value for the Python installation will still
be available, via *note base_prefix: 3eb.
-- Data: sys.ps1
-- Data: sys.ps2
Strings specifying the primary and secondary prompt of the
interpreter. These are only defined if the interpreter is in
interactive mode. Their initial values in this case are ‘'>>> '’
and ‘'... '’. If a non-string object is assigned to either
variable, its *note str(): 447. is re-evaluated each time the
interpreter prepares to read a new interactive command; this can be
used to implement a dynamic prompt.
-- Function: sys.setdlopenflags (n)
Set the flags used by the interpreter for ‘dlopen()’ calls, such as
when the interpreter loads extension modules. Among other things,
this will enable a lazy resolving of symbols when importing a
module, if called as ‘sys.setdlopenflags(0)’. To share symbols
across extension modules, call as
‘sys.setdlopenflags(os.RTLD_GLOBAL)’. Symbolic names for the flag
values can be found in the *note os: a1. module (‘RTLD_XXX’
constants, e.g. *note os.RTLD_LAZY: 110c.).
*note Availability: 1d54.: Unix.
-- Function: sys.set_int_max_str_digits (maxdigits)
Set the *note integer string conversion length limitation: 5f1.
used by this interpreter. See also *note get_int_max_str_digits():
152d.
Added in version 3.11.
-- Function: sys.setprofile (profilefunc)
Set the system’s profile function, which allows you to implement a
Python source code profiler in Python. See chapter *note The
Python Profilers: 427c. for more information on the Python
profiler. The system’s profile function is called similarly to the
system’s trace function (see *note settrace(): 14ca.), but it is
called with different events, for example it isn’t called for each
executed line of code (only on call and return, but the return
event is reported even when an exception has been set). The
function is thread-specific, but there is no way for the profiler
to know about context switches between threads, so it does not make
sense to use this in the presence of multiple threads. Also, its
return value is not used, so it can simply return ‘None’. Error in
the profile function will cause itself unset.
Note: The same tracing mechanism is used for ‘setprofile()’ as
*note settrace(): 14ca. To trace calls with ‘setprofile()’
inside a tracing function (e.g. in a debugger breakpoint),
see *note call_tracing(): 1832.
Profile functions should have three arguments: 'frame', 'event',
and 'arg'. 'frame' is the current stack frame. 'event' is a
string: ‘'call'’, ‘'return'’, ‘'c_call'’, ‘'c_return'’, or
‘'c_exception'’. 'arg' depends on the event type.
The events have the following meaning:
‘'call'’
A function is called (or some other code block entered). The
profile function is called; 'arg' is ‘None’.
‘'return'’
A function (or other code block) is about to return. The
profile function is called; 'arg' is the value that will be
returned, or ‘None’ if the event is caused by an exception
being raised.
‘'c_call'’
A C function is about to be called. This may be an extension
function or a built-in. 'arg' is the C function object.
‘'c_return'’
A C function has returned. 'arg' is the C function object.
‘'c_exception'’
A C function has raised an exception. 'arg' is the C function
object.
Raises an *note auditing event: 18ba. ‘sys.setprofile’ with no
arguments.
-- Function: sys.setrecursionlimit (limit)
Set the maximum depth of the Python interpreter stack to 'limit'.
This limit prevents infinite recursion from causing an overflow of
the C stack and crashing Python.
The highest possible limit is platform-dependent. A user may need
to set the limit higher when they have a program that requires deep
recursion and a platform that supports a higher limit. This should
be done with care, because a too-high limit can lead to a crash.
If the new limit is too low at the current recursion depth, a *note
RecursionError: d6b. exception is raised.
Changed in version 3.5.1: A *note RecursionError: d6b. exception is
now raised if the new limit is too low at the current recursion
depth.
-- Function: sys.setswitchinterval (interval)
Set the interpreter’s thread switch interval (in seconds). This
floating-point value determines the ideal duration of the
“timeslices” allocated to concurrently running Python threads.
Please note that the actual value can be higher, especially if
long-running internal functions or methods are used. Also, which
thread becomes scheduled at the end of the interval is the
operating system’s decision. The interpreter doesn’t have its own
scheduler.
Added in version 3.2.
-- Function: sys.settrace (tracefunc)
Set the system’s trace function, which allows you to implement a
Python source code debugger in Python. The function is
thread-specific; for a debugger to support multiple threads, it
must register a trace function using *note settrace(): 14ca. for
each thread being debugged or use *note threading.settrace(): 844.
Trace functions should have three arguments: 'frame', 'event', and
'arg'. 'frame' is the current stack frame. 'event' is a string:
‘'call'’, ‘'line'’, ‘'return'’, ‘'exception'’ or ‘'opcode'’. 'arg'
depends on the event type.
The trace function is invoked (with 'event' set to ‘'call'’)
whenever a new local scope is entered; it should return a reference
to a local trace function to be used for the new scope, or ‘None’
if the scope shouldn’t be traced.
The local trace function should return a reference to itself, or to
another function which would then be used as the local trace
function for the scope.
If there is any error occurred in the trace function, it will be
unset, just like ‘settrace(None)’ is called.
Note: Tracing is disabled while calling the trace function
(e.g. a function set by ‘settrace()’). For recursive tracing
see *note call_tracing(): 1832.
The events have the following meaning:
‘'call'’
A function is called (or some other code block entered). The
global trace function is called; 'arg' is ‘None’; the return
value specifies the local trace function.
‘'line'’
The interpreter is about to execute a new line of code or
re-execute the condition of a loop. The local trace function
is called; 'arg' is ‘None’; the return value specifies the new
local trace function. See ‘Objects/lnotab_notes.txt’ for a
detailed explanation of how this works. Per-line events may
be disabled for a frame by setting *note f_trace_lines: bdb.
to *note False: b37. on that *note frame: 6db.
‘'return'’
A function (or other code block) is about to return. The
local trace function is called; 'arg' is the value that will
be returned, or ‘None’ if the event is caused by an exception
being raised. The trace function’s return value is ignored.
‘'exception'’
An exception has occurred. The local trace function is
called; 'arg' is a tuple ‘(exception, value, traceback)’; the
return value specifies the new local trace function.
‘'opcode'’
The interpreter is about to execute a new opcode (see *note
dis: 38. for opcode details). The local trace function is
called; 'arg' is ‘None’; the return value specifies the new
local trace function. Per-opcode events are not emitted by
default: they must be explicitly requested by setting *note
f_trace_opcodes: bdc. to *note True: c0d. on the *note frame:
6db.
Note that as an exception is propagated down the chain of callers,
an ‘'exception'’ event is generated at each level.
For more fine-grained usage, it’s possible to set a trace function
by assigning ‘frame.f_trace = tracefunc’ explicitly, rather than
relying on it being set indirectly via the return value from an
already installed trace function. This is also required for
activating the trace function on the current frame, which *note
settrace(): 14ca. doesn’t do. Note that in order for this to work,
a global tracing function must have been installed with *note
settrace(): 14ca. in order to enable the runtime tracing machinery,
but it doesn’t need to be the same tracing function (e.g. it could
be a low overhead tracing function that simply returns ‘None’ to
disable itself immediately on each frame).
For more information on code and frame objects, refer to *note The
standard type hierarchy: 1ee7.
Raises an *note auditing event: 18ba. ‘sys.settrace’ with no
arguments.
'CPython implementation detail:' The *note settrace(): 14ca.
function is intended only for implementing debuggers, profilers,
coverage tools and the like. Its behavior is part of the
implementation platform, rather than part of the language
definition, and thus may not be available in all Python
implementations.
Changed in version 3.7: ‘'opcode'’ event type added; *note
f_trace_lines: bdb. and *note f_trace_opcodes: bdc. attributes
added to frames
-- Function: sys.set_asyncgen_hooks ([firstiter] [, finalizer])
Accepts two optional keyword arguments which are callables that
accept an *note asynchronous generator iterator: 4396. as an
argument. The 'firstiter' callable will be called when an
asynchronous generator is iterated for the first time. The
'finalizer' will be called when an asynchronous generator is about
to be garbage collected.
Raises an *note auditing event: 18ba.
‘sys.set_asyncgen_hooks_firstiter’ with no arguments.
Raises an *note auditing event: 18ba.
‘sys.set_asyncgen_hooks_finalizer’ with no arguments.
Two auditing events are raised because the underlying API consists
of two calls, each of which must raise its own event.
Added in version 3.6: See PEP 525(14) for more details, and for a
reference example of a 'finalizer' method see the implementation of
‘asyncio.Loop.shutdown_asyncgens’ in Lib/asyncio/base_events.py(15)
Note: This function has been added on a provisional basis (see
PEP 411(16) for details.)
-- Function: sys.set_coroutine_origin_tracking_depth (depth)
Allows enabling or disabling coroutine origin tracking. When
enabled, the ‘cr_origin’ attribute on coroutine objects will
contain a tuple of (filename, line number, function name) tuples
describing the traceback where the coroutine object was created,
with the most recent call first. When disabled, ‘cr_origin’ will
be ‘None’.
To enable, pass a 'depth' value greater than zero; this sets the
number of frames whose information will be captured. To disable,
pass set 'depth' to zero.
This setting is thread-specific.
Added in version 3.7.
Note: This function has been added on a provisional basis (see
PEP 411(17) for details.) Use it only for debugging purposes.
-- Function: sys.activate_stack_trampoline (backend, /)
Activate the stack profiler trampoline 'backend'. The only
supported backend is ‘"perf"’.
*note Availability: 1d54.: Linux.
Added in version 3.12.
See also
........
* *note Python support for the Linux perf profiler: 18f.
* ‘https://perf.wiki.kernel.org’
-- Function: sys.deactivate_stack_trampoline ()
Deactivate the current stack profiler trampoline backend.
If no stack profiler is activated, this function has no effect.
*note Availability: 1d54.: Linux.
Added in version 3.12.
-- Function: sys.is_stack_trampoline_active ()
Return ‘True’ if a stack profiler trampoline is active.
*note Availability: 1d54.: Linux.
Added in version 3.12.
-- Function: sys._enablelegacywindowsfsencoding ()
Changes the *note filesystem encoding and error handler: 537. to
‘mbcs’ and ‘replace’ respectively, for consistency with versions of
Python prior to 3.6.
This is equivalent to defining the *note
PYTHONLEGACYWINDOWSFSENCODING: 2b1. environment variable before
launching Python.
See also *note sys.getfilesystemencoding(): c59. and *note
sys.getfilesystemencodeerrors(): ce6.
*note Availability: 1d54.: Windows.
Note: Changing the filesystem encoding after Python startup is
risky because the old fsencoding or paths encoded by the old
fsencoding may be cached somewhere. Use *note
PYTHONLEGACYWINDOWSFSENCODING: 2b1. instead.
Added in version 3.6: See PEP 529(18) for more details.
Deprecated since version 3.13, will be removed in version 3.16: Use
*note PYTHONLEGACYWINDOWSFSENCODING: 2b1. instead.
-- Data: sys.stdin
-- Data: sys.stdout
-- Data: sys.stderr
*note File objects: 11b5. used by the interpreter for standard
input, output and errors:
* ‘stdin’ is used for all interactive input (including calls to
*note input(): 12cb.);
* ‘stdout’ is used for the output of *note print(): f70. and
*note expression: 1f85. statements and for the prompts of
*note input(): 12cb.;
* The interpreter’s own prompts and its error messages go to
‘stderr’.
These streams are regular *note text files: 1c86. like those
returned by the *note open(): 517. function. Their parameters are
chosen as follows:
* The encoding and error handling are is initialized from *note
PyConfig.stdio_encoding: 39f. and *note PyConfig.stdio_errors:
43a4.
On Windows, UTF-8 is used for the console device.
Non-character devices such as disk files and pipes use the
system locale encoding (i.e. the ANSI codepage). Non-console
character devices such as NUL (i.e. where ‘isatty()’ returns
‘True’) use the value of the console input and output
codepages at startup, respectively for stdin and
stdout/stderr. This defaults to the system *note locale
encoding: 244. if the process is not initially attached to a
console.
The special behaviour of the console can be overridden by
setting the environment variable PYTHONLEGACYWINDOWSSTDIO
before starting Python. In that case, the console codepages
are used as for any other character device.
Under all platforms, you can override the character encoding
by setting the *note PYTHONIOENCODING: 1003. environment
variable before starting Python or by using the new *note -X:
176. ‘utf8’ command line option and *note PYTHONUTF8: ad8.
environment variable. However, for the Windows console, this
only applies when *note PYTHONLEGACYWINDOWSSTDIO: c5b. is also
set.
* When interactive, the ‘stdout’ stream is line-buffered.
Otherwise, it is block-buffered like regular text files. The
‘stderr’ stream is line-buffered in both cases. You can make
both streams unbuffered by passing the *note -u: 19a3.
command-line option or setting the *note PYTHONUNBUFFERED:
19a4. environment variable.
Changed in version 3.9: Non-interactive ‘stderr’ is now
line-buffered instead of fully buffered.
Note: To write or read binary data from/to the standard
streams, use the underlying binary *note buffer: 1295. object.
For example, to write bytes to *note stdout: ad6, use
‘sys.stdout.buffer.write(b'abc')’.
However, if you are writing a library (and do not control in
which context its code will be executed), be aware that the
standard streams may be replaced with file-like objects like
*note io.StringIO: f22. which do not support the ‘buffer’
attribute.
-- Data: sys.__stdin__
-- Data: sys.__stdout__
-- Data: sys.__stderr__
These objects contain the original values of ‘stdin’, ‘stderr’ and
‘stdout’ at the start of the program. They are used during
finalization, and could be useful to print to the actual standard
stream no matter if the ‘sys.std*’ object has been redirected.
It can also be used to restore the actual files to known working
file objects in case they have been overwritten with a broken
object. However, the preferred way to do this is to explicitly
save the previous stream before replacing it, and restore the saved
object.
Note: Under some conditions ‘stdin’, ‘stdout’ and ‘stderr’ as
well as the original values ‘__stdin__’, ‘__stdout__’ and
‘__stderr__’ can be ‘None’. It is usually the case for
Windows GUI apps that aren’t connected to a console and Python
apps started with ‘pythonw’.
-- Data: sys.stdlib_module_names
A frozenset of strings containing the names of standard library
modules.
It is the same on all platforms. Modules which are not available
on some platforms and modules disabled at Python build are also
listed. All module kinds are listed: pure Python, built-in, frozen
and extension modules. Test modules are excluded.
For packages, only the main package is listed: sub-packages and
sub-modules are not listed. For example, the ‘email’ package is
listed, but the ‘email.mime’ sub-package and the ‘email.message’
sub-module are not listed.
See also the *note sys.builtin_module_names: 1c5f. list.
Added in version 3.10.
-- Data: sys.thread_info
A *note named tuple: 64a. holding information about the thread
implementation.
-- Attribute: thread_info.name
The name of the thread implementation:
* ‘"nt"’: Windows threads
* ‘"pthread"’: POSIX threads
* ‘"pthread-stubs"’: stub POSIX threads (on WebAssembly
platforms without threading support)
* ‘"solaris"’: Solaris threads
-- Attribute: thread_info.lock
The name of the lock implementation:
* ‘"semaphore"’: a lock uses a semaphore
* ‘"mutex+cond"’: a lock uses a mutex and a condition
variable
* ‘None’ if this information is unknown
-- Attribute: thread_info.version
The name and version of the thread library. It is a string,
or ‘None’ if this information is unknown.
Added in version 3.3.
-- Data: sys.tracebacklimit
When this variable is set to an integer value, it determines the
maximum number of levels of traceback information printed when an
unhandled exception occurs. The default is ‘1000’. When set to
‘0’ or less, all traceback information is suppressed and only the
exception type and value are printed.
-- Function: sys.unraisablehook (unraisable, /)
Handle an unraisable exception.
Called when an exception has occurred but there is no way for
Python to handle it. For example, when a destructor raises an
exception or during garbage collection (*note gc.collect(): a39.).
The 'unraisable' argument has the following attributes:
* ‘exc_type’: Exception type.
* ‘exc_value’: Exception value, can be ‘None’.
* ‘exc_traceback’: Exception traceback, can be ‘None’.
* ‘err_msg’: Error message, can be ‘None’.
* ‘object’: Object causing the exception, can be ‘None’.
The default hook formats ‘err_msg’ and ‘object’ as: ‘f'{err_msg}:
{object!r}'’; use “Exception ignored in” error message if ‘err_msg’
is ‘None’.
*note sys.unraisablehook(): 1f9. can be overridden to control how
unraisable exceptions are handled.
See also
........
*note excepthook(): 807. which handles uncaught exceptions.
Warning: Storing ‘exc_value’ using a custom hook can create a
reference cycle. It should be cleared explicitly to break the
reference cycle when the exception is no longer needed.
Storing ‘object’ using a custom hook can resurrect it if it is
set to an object which is being finalized. Avoid storing
‘object’ after the custom hook completes to avoid resurrecting
objects.
Raise an auditing event ‘sys.unraisablehook’ with arguments 'hook',
'unraisable' when an exception that cannot be handled occurs. The
'unraisable' object is the same as what will be passed to the hook.
If no hook has been set, 'hook' may be ‘None’.
Added in version 3.8.
-- Data: sys.version
A string containing the version number of the Python interpreter
plus additional information on the build number and compiler used.
This string is displayed when the interactive interpreter is
started. Do not extract version information out of it, rather, use
*note version_info: 69c. and the functions provided by the *note
platform: aa. module.
-- Data: sys.api_version
The C API version for this interpreter. Programmers may find this
useful when debugging version conflicts between Python and
extension modules.
-- Data: sys.version_info
A tuple containing the five components of the version number:
'major', 'minor', 'micro', 'releaselevel', and 'serial'. All
values except 'releaselevel' are integers; the release level is
‘'alpha'’, ‘'beta'’, ‘'candidate'’, or ‘'final'’. The
‘version_info’ value corresponding to the Python version 2.0 is
‘(2, 0, 0, 'final', 0)’. The components can also be accessed by
name, so ‘sys.version_info[0]’ is equivalent to
‘sys.version_info.major’ and so on.
Changed in version 3.1: Added named component attributes.
-- Data: sys.warnoptions
This is an implementation detail of the warnings framework; do not
modify this value. Refer to the *note warnings: 112. module for
more information on the warnings framework.
-- Data: sys.winver
The version number used to form registry keys on Windows platforms.
This is stored as string resource 1000 in the Python DLL. The value
is normally the major and minor versions of the running Python
interpreter. It is provided in the *note sys: d9. module for
informational purposes; modifying this value has no effect on the
registry keys used by Python.
*note Availability: 1d54.: Windows.
-- Data: sys.monitoring
Namespace containing functions and constants for register callbacks
and controlling monitoring events. See *note sys.monitoring: da.
for details.
-- Data: sys._xoptions
A dictionary of the various implementation-specific flags passed
through the *note -X: 176. command-line option. Option names are
either mapped to their values, if given explicitly, or to *note
True: c0d. Example:
$ ./python -Xa=b -Xc
Python 3.2a3+ (py3k, Oct 16 2010, 20:14:50)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._xoptions
{'a': 'b', 'c': True}
'CPython implementation detail:' This is a CPython-specific way of
accessing options passed through *note -X: 176. Other
implementations may export them through other means, or not at all.
Added in version 3.2.
Citations
.........
(C99) ISO/IEC 9899:1999. “Programming languages – C.” A public draft of
this standard is available at
‘https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf’.
---------- Footnotes ----------
(1) https://peps.python.org/pep-3149/
(2) https://peps.python.org/pep-0578/
(3) https://peps.python.org/pep-0529/
(4)
https://code.activestate.com/recipes/577504-compute-memory-footprint-of-an-object-and-its-cont/
(5) https://peps.python.org/pep-0525/
(6) https://peps.python.org/pep-0411/
(7) https://peps.python.org/pep-0411/
(8) https://peps.python.org/pep-0421/
(9) https://peps.python.org/pep-0421/
(10) https://peps.python.org/pep-0393/
(11) https://peps.python.org/pep-0451/
(12) https://peps.python.org/pep-0302/
(13) https://peps.python.org/pep-0302/
(14) https://peps.python.org/pep-0525/
(15)
https://github.com/python/cpython/tree/3.13/Lib/asyncio/base_events.py
(16) https://peps.python.org/pep-0411/
(17) https://peps.python.org/pep-0411/
(18) https://peps.python.org/pep-0529/
File: python.info, Node: sys monitoring — Execution event monitoring, Next: sysconfig — Provide access to Python’s configuration information, Prev: sys — System-specific parameters and functions, Up: Python Runtime Services
5.30.2 ‘sys.monitoring’ — Execution event monitoring
----------------------------------------------------
Added in version 3.12.
__________________________________________________________________
Note: *note sys.monitoring: da. is a namespace within the *note
sys: d9. module, not an independent module, so there is no need to
‘import sys.monitoring’, simply ‘import sys’ and then use
‘sys.monitoring’.
This namespace provides access to the functions and constants necessary
to activate and control event monitoring.
As programs execute, events occur that might be of interest to tools
that monitor execution. The *note sys.monitoring: da. namespace
provides means to receive callbacks when events of interest occur.
The monitoring API consists of three components:
* *note Tool identifiers: 43ad.
* *note Events: 43ae.
* *note Callbacks: 43af.
* Menu:
* Tool identifiers::
* Events::
* Turning events on and off::
* Registering callback functions::
File: python.info, Node: Tool identifiers, Next: Events, Up: sys monitoring — Execution event monitoring
5.30.2.1 Tool identifiers
.........................
A tool identifier is an integer and the associated name. Tool
identifiers are used to discourage tools from interfering with each
other and to allow multiple tools to operate at the same time.
Currently tools are completely independent and cannot be used to monitor
each other. This restriction may be lifted in the future.
Before registering or activating events, a tool should choose an
identifier. Identifiers are integers in the range 0 to 5 inclusive.
* Menu:
* Registering and using tools::
File: python.info, Node: Registering and using tools, Up: Tool identifiers
5.30.2.2 Registering and using tools
....................................
-- Function: sys.monitoring.use_tool_id (tool_id: int, name: str, /) ->
None
Must be called before 'tool_id' can be used. 'tool_id' must be in
the range 0 to 5 inclusive. Raises a *note ValueError: 204. if
'tool_id' is in use.
-- Function: sys.monitoring.free_tool_id (tool_id: int, /) -> None
Should be called once a tool no longer requires 'tool_id'.
Note: *note free_tool_id(): 43b2. will not disable global or local
events associated with 'tool_id', nor will it unregister any
callback functions. This function is only intended to be used to
notify the VM that the particular 'tool_id' is no longer in use.
-- Function: sys.monitoring.get_tool (tool_id: int, /) -> str | None
Returns the name of the tool if 'tool_id' is in use, otherwise it
returns ‘None’. 'tool_id' must be in the range 0 to 5 inclusive.
All IDs are treated the same by the VM with regard to events, but the
following IDs are pre-defined to make co-operation of tools easier:
sys.monitoring.DEBUGGER_ID = 0
sys.monitoring.COVERAGE_ID = 1
sys.monitoring.PROFILER_ID = 2
sys.monitoring.OPTIMIZER_ID = 5
File: python.info, Node: Events, Next: Turning events on and off, Prev: Tool identifiers, Up: sys monitoring — Execution event monitoring
5.30.2.3 Events
...............
The following events are supported:
-- monitoring-event: sys.monitoring.events.BRANCH
A conditional branch is taken (or not).
-- monitoring-event: sys.monitoring.events.CALL
A call in Python code (event occurs before the call).
-- monitoring-event: sys.monitoring.events.C_RAISE
An exception raised from any callable, except for Python functions
(event occurs after the exit).
-- monitoring-event: sys.monitoring.events.C_RETURN
Return from any callable, except for Python functions (event occurs
after the return).
-- monitoring-event: sys.monitoring.events.EXCEPTION_HANDLED
An exception is handled.
-- monitoring-event: sys.monitoring.events.INSTRUCTION
A VM instruction is about to be executed.
-- monitoring-event: sys.monitoring.events.JUMP
An unconditional jump in the control flow graph is made.
-- monitoring-event: sys.monitoring.events.LINE
An instruction is about to be executed that has a different line
number from the preceding instruction.
-- monitoring-event: sys.monitoring.events.PY_RESUME
Resumption of a Python function (for generator and coroutine
functions), except for ‘throw()’ calls.
-- monitoring-event: sys.monitoring.events.PY_RETURN
Return from a Python function (occurs immediately before the
return, the callee’s frame will be on the stack).
-- monitoring-event: sys.monitoring.events.PY_START
Start of a Python function (occurs immediately after the call, the
callee’s frame will be on the stack)
-- monitoring-event: sys.monitoring.events.PY_THROW
A Python function is resumed by a ‘throw()’ call.
-- monitoring-event: sys.monitoring.events.PY_UNWIND
Exit from a Python function during exception unwinding.
-- monitoring-event: sys.monitoring.events.PY_YIELD
Yield from a Python function (occurs immediately before the yield,
the callee’s frame will be on the stack).
-- monitoring-event: sys.monitoring.events.RAISE
An exception is raised, except those that cause a *note
STOP_ITERATION: 43c1. event.
-- monitoring-event: sys.monitoring.events.RERAISE
An exception is re-raised, for example at the end of a *note
finally: 9ca. block.
-- monitoring-event: sys.monitoring.events.STOP_ITERATION
An artificial *note StopIteration: bfa. is raised; see *note the
STOP_ITERATION event: 43c3.
More events may be added in the future.
These events are attributes of the ‘sys.monitoring.events’ namespace.
Each event is represented as a power-of-2 integer constant. To define a
set of events, simply bitwise OR the individual events together. For
example, to specify both *note PY_RETURN: 43bc. and *note PY_START:
43bd. events, use the expression ‘PY_RETURN | PY_START’.
-- monitoring-event: sys.monitoring.events.NO_EVENTS
An alias for ‘0’ so users can do explicit comparisons like:
if get_events(DEBUGGER_ID) == NO_EVENTS:
...
Events are divided into three groups:
* Menu:
* Local events::
* Ancillary events::
* Other events::
* The STOP_ITERATION event::
File: python.info, Node: Local events, Next: Ancillary events, Up: Events
5.30.2.4 Local events
.....................
Local events are associated with normal execution of the program and
happen at clearly defined locations. All local events can be disabled.
The local events are:
* *note PY_START: 43bd.
* *note PY_RESUME: 43bb.
* *note PY_RETURN: 43bc.
* *note PY_YIELD: 43c0.
* *note CALL: 1646.
* *note LINE: 43ba.
* *note INSTRUCTION: 43b8.
* *note JUMP: 43b9.
* *note BRANCH: 43b4.
* *note STOP_ITERATION: 43c1.
File: python.info, Node: Ancillary events, Next: Other events, Prev: Local events, Up: Events
5.30.2.5 Ancillary events
.........................
Ancillary events can be monitored like other events, but are controlled
by another event:
* *note C_RAISE: 43b5.
* *note C_RETURN: 43b6.
The *note C_RETURN: 43b6. and *note C_RAISE: 43b5. events are controlled
by the *note CALL: 1646. event. *note C_RETURN: 43b6. and *note
C_RAISE: 43b5. events will only be seen if the corresponding *note CALL:
1646. event is being monitored.
File: python.info, Node: Other events, Next: The STOP_ITERATION event, Prev: Ancillary events, Up: Events
5.30.2.6 Other events
.....................
Other events are not necessarily tied to a specific location in the
program and cannot be individually disabled.
The other events that can be monitored are:
* *note PY_THROW: 43be.
* *note PY_UNWIND: 43bf.
* *note RAISE: 15f0.
* *note EXCEPTION_HANDLED: 43b7.
File: python.info, Node: The STOP_ITERATION event, Prev: Other events, Up: Events
5.30.2.7 The STOP_ITERATION event
.................................
PEP 380(1) specifies that a *note StopIteration: bfa. exception is
raised when returning a value from a generator or coroutine. However,
this is a very inefficient way to return a value, so some Python
implementations, notably CPython 3.12+, do not raise an exception unless
it would be visible to other code.
To allow tools to monitor for real exceptions without slowing down
generators and coroutines, the *note STOP_ITERATION: 43c1. event is
provided. *note STOP_ITERATION: 43c1. can be locally disabled, unlike
*note RAISE: 15f0.
---------- Footnotes ----------
(1)
https://peps.python.org/pep-0380/#use-of-stopiteration-to-return-values
File: python.info, Node: Turning events on and off, Next: Registering callback functions, Prev: Events, Up: sys monitoring — Execution event monitoring
5.30.2.8 Turning events on and off
..................................
In order to monitor an event, it must be turned on and a corresponding
callback must be registered. Events can be turned on or off by setting
the events either globally or for a particular code object.
* Menu:
* Setting events globally::
* Per code object events::
* Disabling events::
File: python.info, Node: Setting events globally, Next: Per code object events, Up: Turning events on and off
5.30.2.9 Setting events globally
................................
Events can be controlled globally by modifying the set of events being
monitored.
-- Function: sys.monitoring.get_events (tool_id: int, /) -> int
Returns the ‘int’ representing all the active events.
-- Function: sys.monitoring.set_events (tool_id: int, event_set: int,
/) -> None
Activates all events which are set in 'event_set'. Raises a *note
ValueError: 204. if 'tool_id' is not in use.
No events are active by default.
File: python.info, Node: Per code object events, Next: Disabling events, Prev: Setting events globally, Up: Turning events on and off
5.30.2.10 Per code object events
................................
Events can also be controlled on a per code object basis. The functions
defined below which accept a *note types.CodeType: 2e3. should be
prepared to accept a look-alike object from functions which are not
defined in Python (see *note Monitoring C API: 15c.).
-- Function: sys.monitoring.get_local_events (tool_id: int, code:
CodeType, /) -> int
Returns all the local events for 'code'
-- Function: sys.monitoring.set_local_events (tool_id: int, code:
CodeType, event_set: int, /) -> None
Activates all the local events for 'code' which are set in
'event_set'. Raises a *note ValueError: 204. if 'tool_id' is not
in use.
Local events add to global events, but do not mask them. In other
words, all global events will trigger for a code object, regardless of
the local events.
File: python.info, Node: Disabling events, Prev: Per code object events, Up: Turning events on and off
5.30.2.11 Disabling events
..........................
-- Data: sys.monitoring.DISABLE
A special value that can be returned from a callback function to
disable events for the current code location.
Local events can be disabled for a specific code location by returning
*note sys.monitoring.DISABLE: 43d1. from a callback function. This does
not change which events are set, or any other code locations for the
same event.
Disabling events for specific locations is very important for high
performance monitoring. For example, a program can be run under a
debugger with no overhead if the debugger disables all monitoring except
for a few breakpoints.
-- Function: sys.monitoring.restart_events () -> None
Enable all the events that were disabled by *note
sys.monitoring.DISABLE: 43d1. for all tools.
File: python.info, Node: Registering callback functions, Prev: Turning events on and off, Up: sys monitoring — Execution event monitoring
5.30.2.12 Registering callback functions
........................................
To register a callable for events call
-- Function: sys.monitoring.register_callback (tool_id: int, event:
int, func: Callable | None, /) -> Callable | None
Registers the callable 'func' for the 'event' with the given
'tool_id'
If another callback was registered for the given 'tool_id' and
'event', it is unregistered and returned. Otherwise *note
register_callback(): 41ef. returns ‘None’.
Raises an *note auditing event: 18ba.
‘sys.monitoring.register_callback’ with argument ‘func’.
Functions can be unregistered by calling
‘sys.monitoring.register_callback(tool_id, event, None)’.
Callback functions can be registered and unregistered at any time.
* Menu:
* Callback function arguments::
File: python.info, Node: Callback function arguments, Up: Registering callback functions
5.30.2.13 Callback function arguments
.....................................
-- Data: sys.monitoring.MISSING
A special value that is passed to a callback function to indicate
that there are no arguments to the call.
When an active event occurs, the registered callback function is called.
Different events will provide the callback function with different
arguments, as follows:
* *note PY_START: 43bd. and *note PY_RESUME: 43bb.:
func(code: CodeType, instruction_offset: int) -> DISABLE | Any
* *note PY_RETURN: 43bc. and *note PY_YIELD: 43c0.:
func(code: CodeType, instruction_offset: int, retval: object) -> DISABLE | Any
* *note CALL: 1646, *note C_RAISE: 43b5. and *note C_RETURN: 43b6.:
func(code: CodeType, instruction_offset: int, callable: object, arg0: object | MISSING) -> DISABLE | Any
If there are no arguments, 'arg0' is set to *note
sys.monitoring.MISSING: 43d5.
* *note RAISE: 15f0, *note RERAISE: 43c2, *note EXCEPTION_HANDLED:
43b7, *note PY_UNWIND: 43bf, *note PY_THROW: 43be. and *note
STOP_ITERATION: 43c1.:
func(code: CodeType, instruction_offset: int, exception: BaseException) -> DISABLE | Any
* *note LINE: 43ba.:
func(code: CodeType, line_number: int) -> DISABLE | Any
* *note BRANCH: 43b4. and *note JUMP: 43b9.:
func(code: CodeType, instruction_offset: int, destination_offset: int) -> DISABLE | Any
Note that the 'destination_offset' is where the code will next
execute. For an untaken branch this will be the offset of the
instruction following the branch.
* *note INSTRUCTION: 43b8.:
func(code: CodeType, instruction_offset: int) -> DISABLE | Any
File: python.info, Node: sysconfig — Provide access to Python’s configuration information, Next: builtins — Built-in objects, Prev: sys monitoring — Execution event monitoring, Up: Python Runtime Services
5.30.3 ‘sysconfig’ — Provide access to Python’s configuration information
-------------------------------------------------------------------------
Added in version 3.2.
'Source code:' Lib/sysconfig(1)
__________________________________________________________________
The *note sysconfig: db. module provides access to Python’s
configuration information like the list of installation paths and the
configuration variables relevant for the current platform.
* Menu:
* Configuration variables::
* Installation paths::
* User scheme::
* Home scheme::
* Prefix scheme::
* Installation path functions::
* Other functions: Other functions<3>.
* Command-line usage: Command-line usage<3>.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/sysconfig
File: python.info, Node: Configuration variables, Next: Installation paths, Up: sysconfig — Provide access to Python’s configuration information
5.30.3.1 Configuration variables
................................
A Python distribution contains a ‘Makefile’ and a ‘pyconfig.h’ header
file that are necessary to build both the Python binary itself and
third-party C extensions compiled using ‘setuptools’.
*note sysconfig: db. puts all variables found in these files in a
dictionary that can be accessed using *note get_config_vars(): 1025. or
*note get_config_var(): 1024.
Notice that on Windows, it’s a much smaller set.
-- Function: sysconfig.get_config_vars (*args)
With no arguments, return a dictionary of all configuration
variables relevant for the current platform.
With arguments, return a list of values that result from looking up
each argument in the configuration variable dictionary.
For each argument, if the value is not found, return ‘None’.
-- Function: sysconfig.get_config_var (name)
Return the value of a single variable 'name'. Equivalent to
‘get_config_vars().get(name)’.
If 'name' is not found, return ‘None’.
Example of usage:
>>> import sysconfig
>>> sysconfig.get_config_var('Py_ENABLE_SHARED')
0
>>> sysconfig.get_config_var('LIBDIR')
'/usr/local/lib'
>>> sysconfig.get_config_vars('AR', 'CXX')
['ar', 'g++']
File: python.info, Node: Installation paths, Next: User scheme, Prev: Configuration variables, Up: sysconfig — Provide access to Python’s configuration information
5.30.3.2 Installation paths
...........................
Python uses an installation scheme that differs depending on the
platform and on the installation options. These schemes are stored in
*note sysconfig: db. under unique identifiers based on the value
returned by *note os.name: 2b0e. The schemes are used by package
installers to determine where to copy files to.
Python currently supports nine schemes:
- 'posix_prefix': scheme for POSIX platforms like Linux or macOS.
This is the default scheme used when Python or a component is
installed.
- 'posix_home': scheme for POSIX platforms, when the 'home' option is
used. This scheme defines paths located under a specific home
prefix.
- 'posix_user': scheme for POSIX platforms, when the 'user' option is
used. This scheme defines paths located under the user’s home
directory (*note site.USER_BASE: 130f.).
- 'posix_venv': scheme for *note Python virtual environments: 111. on
POSIX platforms; by default it is the same as 'posix_prefix'.
- 'nt': scheme for Windows. This is the default scheme used when
Python or a component is installed.
- 'nt_user': scheme for Windows, when the 'user' option is used.
- 'nt_venv': scheme for *note Python virtual environments: 111. on
Windows; by default it is the same as 'nt'.
- 'venv': a scheme with values from either 'posix_venv' or 'nt_venv'
depending on the platform Python runs on.
- 'osx_framework_user': scheme for macOS, when the 'user' option is
used.
Each scheme is itself composed of a series of paths and each path has a
unique identifier. Python currently uses eight paths:
- 'stdlib': directory containing the standard Python library files
that are not platform-specific.
- 'platstdlib': directory containing the standard Python library
files that are platform-specific.
- 'platlib': directory for site-specific, platform-specific files.
- 'purelib': directory for site-specific, non-platform-specific files
(‘pure’ Python).
- 'include': directory for non-platform-specific header files for the
Python C-API.
- 'platinclude': directory for platform-specific header files for the
Python C-API.
- 'scripts': directory for script files.
- 'data': directory for data files.
File: python.info, Node: User scheme, Next: Home scheme, Prev: Installation paths, Up: sysconfig — Provide access to Python’s configuration information
5.30.3.3 User scheme
....................
This scheme is designed to be the most convenient solution for users
that don’t have write permission to the global site-packages directory
or don’t want to install into it.
Files will be installed into subdirectories of *note site.USER_BASE:
130f. (written as ‘`userbase'’ hereafter). This scheme installs pure
Python modules and extension modules in the same location (also known as
*note site.USER_SITE: 1d35.).
* Menu:
* posix_user::
* nt_user::
* osx_framework_user::
File: python.info, Node: posix_user, Next: nt_user, Up: User scheme
5.30.3.4 ‘posix_user’
.....................
Path Installation directory
-----------------------------------------------------------------------------------
'stdlib' ‘`userbase'/lib/python`X.Y'’
'platstdlib' ‘`userbase'/lib/python`X.Y'’
'platlib' ‘`userbase'/lib/python`X.Y'/site-packages’
'purelib' ‘`userbase'/lib/python`X.Y'/site-packages’
'include' ‘`userbase'/include/python`X.Y'’
'scripts' ‘`userbase'/bin’
'data' ‘`userbase'’
File: python.info, Node: nt_user, Next: osx_framework_user, Prev: posix_user, Up: User scheme
5.30.3.5 ‘nt_user’
..................
Path Installation directory
-----------------------------------------------------------------------------------
'stdlib' ‘`userbase'\Python`XY'’
'platstdlib' ‘`userbase'\Python`XY'’
'platlib' ‘`userbase'\Python`XY'\site-packages’
'purelib' ‘`userbase'\Python`XY'\site-packages’
'include' ‘`userbase'\Python`XY'\Include’
'scripts' ‘`userbase'\Python`XY'\Scripts’
'data' ‘`userbase'’
File: python.info, Node: osx_framework_user, Prev: nt_user, Up: User scheme
5.30.3.6 ‘osx_framework_user’
.............................
Path Installation directory
-----------------------------------------------------------------------------------
'stdlib' ‘`userbase'/lib/python’
'platstdlib' ‘`userbase'/lib/python’
'platlib' ‘`userbase'/lib/python/site-packages’
'purelib' ‘`userbase'/lib/python/site-packages’
'include' ‘`userbase'/include/python`X.Y'’
'scripts' ‘`userbase'/bin’
'data' ‘`userbase'’
File: python.info, Node: Home scheme, Next: Prefix scheme, Prev: User scheme, Up: sysconfig — Provide access to Python’s configuration information
5.30.3.7 Home scheme
....................
The idea behind the “home scheme” is that you build and maintain a
personal stash of Python modules. This scheme’s name is derived from
the idea of a “home” directory on Unix, since it’s not unusual for a
Unix user to make their home directory have a layout similar to ‘/usr/’
or ‘/usr/local/’. This scheme can be used by anyone, regardless of the
operating system they are installing for.
* Menu:
* posix_home::
File: python.info, Node: posix_home, Up: Home scheme
5.30.3.8 ‘posix_home’
.....................
Path Installation directory
-----------------------------------------------------------------------------------
'stdlib' ‘`home'/lib/python’
'platstdlib' ‘`home'/lib/python’
'platlib' ‘`home'/lib/python’
'purelib' ‘`home'/lib/python’
'include' ‘`home'/include/python’
'platinclude' ‘`home'/include/python’
'scripts' ‘`home'/bin’
'data' ‘`home'’
File: python.info, Node: Prefix scheme, Next: Installation path functions, Prev: Home scheme, Up: sysconfig — Provide access to Python’s configuration information
5.30.3.9 Prefix scheme
......................
The “prefix scheme” is useful when you wish to use one Python
installation to perform the build/install (i.e., to run the setup
script), but install modules into the third-party module directory of a
different Python installation (or something that looks like a different
Python installation). If this sounds a trifle unusual, it is—that’s why
the user and home schemes come before. However, there are at least two
known cases where the prefix scheme will be useful.
First, consider that many Linux distributions put Python in ‘/usr’,
rather than the more traditional ‘/usr/local’. This is entirely
appropriate, since in those cases Python is part of “the system” rather
than a local add-on. However, if you are installing Python modules from
source, you probably want them to go in ‘/usr/local/lib/python2.`X'’
rather than ‘/usr/lib/python2.`X'’.
Another possibility is a network filesystem where the name used to write
to a remote directory is different from the name used to read it: for
example, the Python interpreter accessed as ‘/usr/local/bin/python’
might search for modules in ‘/usr/local/lib/python2.`X'’, but those
modules would have to be installed to, say,
‘/mnt/`@server'/export/lib/python2.`X'’.
* Menu:
* posix_prefix::
* nt::
File: python.info, Node: posix_prefix, Next: nt, Up: Prefix scheme
5.30.3.10 ‘posix_prefix’
........................
Path Installation directory
----------------------------------------------------------------------------------
'stdlib' ‘`prefix'/lib/python`X.Y'’
'platstdlib' ‘`prefix'/lib/python`X.Y'’
'platlib' ‘`prefix'/lib/python`X.Y'/site-packages’
'purelib' ‘`prefix'/lib/python`X.Y'/site-packages’
'include' ‘`prefix'/include/python`X.Y'’
'platinclude' ‘`prefix'/include/python`X.Y'’
'scripts' ‘`prefix'/bin’
'data' ‘`prefix'’
File: python.info, Node: nt, Prev: posix_prefix, Up: Prefix scheme
5.30.3.11 ‘nt’
..............
Path Installation directory
----------------------------------------------------------------------------------
'stdlib' ‘`prefix'\Lib’
'platstdlib' ‘`prefix'\Lib’
'platlib' ‘`prefix'\Lib\site-packages’
'purelib' ‘`prefix'\Lib\site-packages’
'include' ‘`prefix'\Include’
'platinclude' ‘`prefix'\Include’
'scripts' ‘`prefix'\Scripts’
'data' ‘`prefix'’
File: python.info, Node: Installation path functions, Next: Other functions<3>, Prev: Prefix scheme, Up: sysconfig — Provide access to Python’s configuration information
5.30.3.12 Installation path functions
.....................................
*note sysconfig: db. provides some functions to determine these
installation paths.
-- Function: sysconfig.get_scheme_names ()
Return a tuple containing all schemes currently supported in *note
sysconfig: db.
-- Function: sysconfig.get_default_scheme ()
Return the default scheme name for the current platform.
Added in version 3.10: This function was previously named
‘_get_default_scheme()’ and considered an implementation detail.
Changed in version 3.11: When Python runs from a virtual
environment, the 'venv' scheme is returned.
-- Function: sysconfig.get_preferred_scheme (key)
Return a preferred scheme name for an installation layout specified
by 'key'.
'key' must be either ‘"prefix"’, ‘"home"’, or ‘"user"’.
The return value is a scheme name listed in *note
get_scheme_names(): 43e6. It can be passed to *note sysconfig: db.
functions that take a 'scheme' argument, such as *note get_paths():
1237.
Added in version 3.10.
Changed in version 3.11: When Python runs from a virtual
environment and ‘key="prefix"’, the 'venv' scheme is returned.
-- Function: sysconfig._get_preferred_schemes ()
Return a dict containing preferred scheme names on the current
platform. Python implementers and redistributors may add their
preferred schemes to the ‘_INSTALL_SCHEMES’ module-level global
value, and modify this function to return those scheme names, to
e.g. provide different schemes for system and language package
managers to use, so packages installed by either do not mix with
those by the other.
End users should not use this function, but *note
get_default_scheme(): 1823. and *note get_preferred_scheme(): 68c.
instead.
Added in version 3.10.
-- Function: sysconfig.get_path_names ()
Return a tuple containing all path names currently supported in
*note sysconfig: db.
-- Function: sysconfig.get_path (name[, scheme[, vars[, expand]]])
Return an installation path corresponding to the path 'name', from
the install scheme named 'scheme'.
'name' has to be a value from the list returned by *note
get_path_names(): 43e8.
*note sysconfig: db. stores installation paths corresponding to
each path name, for each platform, with variables to be expanded.
For instance the 'stdlib' path for the 'nt' scheme is:
‘{base}/Lib’.
*note get_path(): 1321. will use the variables returned by *note
get_config_vars(): 1025. to expand the path. All variables have
default values for each platform so one may call this function and
get the default value.
If 'scheme' is provided, it must be a value from the list returned
by *note get_scheme_names(): 43e6. Otherwise, the default scheme
for the current platform is used.
If 'vars' is provided, it must be a dictionary of variables that
will update the dictionary returned by *note get_config_vars():
1025.
If 'expand' is set to ‘False’, the path will not be expanded using
the variables.
If 'name' is not found, raise a *note KeyError: 33f.
-- Function: sysconfig.get_paths ([scheme[, vars[, expand]]])
Return a dictionary containing all installation paths corresponding
to an installation scheme. See *note get_path(): 1321. for more
information.
If 'scheme' is not provided, will use the default scheme for the
current platform.
If 'vars' is provided, it must be a dictionary of variables that
will update the dictionary used to expand the paths.
If 'expand' is set to false, the paths will not be expanded.
If 'scheme' is not an existing scheme, *note get_paths(): 1237.
will raise a *note KeyError: 33f.
File: python.info, Node: Other functions<3>, Next: Command-line usage<3>, Prev: Installation path functions, Up: sysconfig — Provide access to Python’s configuration information
5.30.3.13 Other functions
.........................
-- Function: sysconfig.get_python_version ()
Return the ‘MAJOR.MINOR’ Python version number as a string.
Similar to ‘'%d.%d' % sys.version_info[:2]’.
-- Function: sysconfig.get_platform ()
Return a string that identifies the current platform.
This is used mainly to distinguish platform-specific build
directories and platform-specific built distributions. Typically
includes the OS name and version and the architecture (as supplied
by *note os.uname(): 110a.), although the exact information
included depends on the OS; e.g., on Linux, the kernel version
isn’t particularly important.
Examples of returned values:
- linux-i586
- linux-alpha (?)
- solaris-2.6-sun4u
Windows will return one of:
- win-amd64 (64-bit Windows on AMD64, aka x86_64, Intel64, and
EM64T)
- win-arm64 (64-bit Windows on ARM64, aka AArch64)
- win32 (all others - specifically, sys.platform is returned)
macOS can return:
- macosx-10.6-ppc
- macosx-10.4-ppc64
- macosx-10.3-i386
- macosx-10.4-fat
For other non-POSIX platforms, currently just returns *note
sys.platform: a8d.
-- Function: sysconfig.is_python_build ()
Return ‘True’ if the running Python interpreter was built from
source and is being run from its built location, and not from a
location resulting from e.g. running ‘make install’ or installing
via a binary installer.
-- Function: sysconfig.parse_config_h (fp[, vars])
Parse a ‘config.h’-style file.
'fp' is a file-like object pointing to the ‘config.h’-like file.
A dictionary containing name/value pairs is returned. If an
optional dictionary is passed in as the second argument, it is used
instead of a new dictionary, and updated with the values read in
the file.
-- Function: sysconfig.get_config_h_filename ()
Return the path of ‘pyconfig.h’.
-- Function: sysconfig.get_makefile_filename ()
Return the path of ‘Makefile’.
File: python.info, Node: Command-line usage<3>, Prev: Other functions<3>, Up: sysconfig — Provide access to Python’s configuration information
5.30.3.14 Command-line usage
............................
You can use *note sysconfig: db. as a script with Python’s '-m' option:
$ python -m sysconfig
Platform: "macosx-10.4-i386"
Python version: "3.2"
Current installation scheme: "posix_prefix"
Paths:
data = "/usr/local"
include = "/Users/tarek/Dev/svn.python.org/py3k/Include"
platinclude = "."
platlib = "/usr/local/lib/python3.2/site-packages"
platstdlib = "/usr/local/lib/python3.2"
purelib = "/usr/local/lib/python3.2/site-packages"
scripts = "/usr/local/bin"
stdlib = "/usr/local/lib/python3.2"
Variables:
AC_APPLE_UNIVERSAL_BUILD = "0"
AIX_GENUINE_CPLUSPLUS = "0"
AR = "ar"
ARFLAGS = "rc"
...
This call will print in the standard output the information returned by
*note get_platform(): 1235, *note get_python_version(): 1236, *note
get_path(): 1321. and *note get_config_vars(): 1025.
File: python.info, Node: builtins — Built-in objects, Next: __main__ — Top-level code environment, Prev: sysconfig — Provide access to Python’s configuration information, Up: Python Runtime Services
5.30.4 ‘builtins’ — Built-in objects
------------------------------------
__________________________________________________________________
This module provides direct access to all ‘built-in’ identifiers of
Python; for example, ‘builtins.open’ is the full name for the built-in
function *note open(): 517.
This module is not normally accessed explicitly by most applications,
but can be useful in modules that provide objects with the same name as
a built-in value, but in which the built-in of that name is also needed.
For example, in a module that wants to implement an *note open(): 517.
function that wraps the built-in *note open(): 517, this module can be
used directly:
import builtins
def open(path):
f = builtins.open(path, 'r')
return UpperCaser(f)
class UpperCaser:
'''Wrapper around a file that converts output to uppercase.'''
def __init__(self, f):
self._f = f
def read(self, count=-1):
return self._f.read(count).upper()
# ...
As an implementation detail, most modules have the name ‘__builtins__’
made available as part of their globals. The value of ‘__builtins__’ is
normally either this module or the value of this module’s *note
__dict__: 558. attribute. Since this is an implementation detail, it
may not be used by alternate implementations of Python.
See also
........
* *note Built-in Constants: 2184.
* *note Built-in Exceptions: 1883.
* *note Built-in Functions: 1250.
* *note Built-in Types: 218e.
File: python.info, Node: __main__ — Top-level code environment, Next: warnings — Warning control, Prev: builtins — Built-in objects, Up: Python Runtime Services
5.30.5 ‘__main__’ — Top-level code environment
----------------------------------------------
__________________________________________________________________
In Python, the special name ‘__main__’ is used for two important
constructs:
1. the name of the top-level environment of the program, which can be
checked using the ‘__name__ == '__main__'’ expression; and
2. the ‘__main__.py’ file in Python packages.
Both of these mechanisms are related to Python modules; how users
interact with them and how they interact with each other. They are
explained in detail below. If you’re new to Python modules, see the
tutorial section *note Modules: 1c56. for an introduction.
* Menu:
* __name__ == '__main__'::
* __main__.py in Python Packages: __main__ py in Python Packages.
* import __main__::
File: python.info, Node: __name__ == '__main__', Next: __main__ py in Python Packages, Up: __main__ — Top-level code environment
5.30.5.1 ‘__name__ == '__main__'’
.................................
When a Python module or package is imported, ‘__name__’ is set to the
module’s name. Usually, this is the name of the Python file itself
without the ‘.py’ extension:
>>> import configparser
>>> configparser.__name__
'configparser'
If the file is part of a package, ‘__name__’ will also include the
parent package’s path:
>>> from concurrent.futures import process
>>> process.__name__
'concurrent.futures.process'
However, if the module is executed in the top-level code environment,
its ‘__name__’ is set to the string ‘'__main__'’.
* Menu:
* What is the “top-level code environment”?::
* Idiomatic Usage::
* Packaging Considerations::
File: python.info, Node: What is the “top-level code environment”?, Next: Idiomatic Usage, Up: __name__ == '__main__'
5.30.5.2 What is the “top-level code environment”?
..................................................
‘__main__’ is the name of the environment where top-level code is run.
“Top-level code” is the first user-specified Python module that starts
running. It’s “top-level” because it imports all other modules that the
program needs. Sometimes “top-level code” is called an 'entry point' to
the application.
The top-level code environment can be:
* the scope of an interactive prompt:
>>> __name__
'__main__'
* the Python module passed to the Python interpreter as a file
argument:
$ python helloworld.py
Hello, world!
* the Python module or package passed to the Python interpreter with
the *note -m: 5dd. argument:
$ python -m tarfile
usage: tarfile.py [-h] [-v] (...)
* Python code read by the Python interpreter from standard input:
$ echo "import this" | python
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
...
* Python code passed to the Python interpreter with the *note -c:
5dc. argument:
$ python -c "import this"
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
...
In each of these situations, the top-level module’s ‘__name__’ is set to
‘'__main__'’.
As a result, a module can discover whether or not it is running in the
top-level environment by checking its own ‘__name__’, which allows a
common idiom for conditionally executing code when the module is not
initialized from an import statement:
if __name__ == '__main__':
# Execute when the module is not initialized from an import statement.
...
See also
........
For a more detailed look at how ‘__name__’ is set in all situations, see
the tutorial section *note Modules: 1c56.
File: python.info, Node: Idiomatic Usage, Next: Packaging Considerations, Prev: What is the “top-level code environment”?, Up: __name__ == '__main__'
5.30.5.3 Idiomatic Usage
........................
Some modules contain code that is intended for script use only, like
parsing command-line arguments or fetching data from standard input. If
a module like this was imported from a different module, for example to
unit test it, the script code would unintentionally execute as well.
This is where using the ‘if __name__ == '__main__'’ code block comes in
handy. Code within this block won’t run unless the module is executed
in the top-level environment.
Putting as few statements as possible in the block below ‘if __name__ ==
'__main__'’ can improve code clarity and correctness. Most often, a
function named ‘main’ encapsulates the program’s primary behavior:
# echo.py
import shlex
import sys
def echo(phrase: str) -> None:
"""A dummy wrapper around print."""
# for demonstration purposes, you can imagine that there is some
# valuable and reusable logic inside this function
print(phrase)
def main() -> int:
"""Echo the input arguments to standard output"""
phrase = shlex.join(sys.argv)
echo(phrase)
return 0
if __name__ == '__main__':
sys.exit(main()) # next section explains the use of sys.exit
Note that if the module didn’t encapsulate code inside the ‘main’
function but instead put it directly within the ‘if __name__ ==
'__main__'’ block, the ‘phrase’ variable would be global to the entire
module. This is error-prone as other functions within the module could
be unintentionally using the global variable instead of a local name. A
‘main’ function solves this problem.
Using a ‘main’ function has the added benefit of the ‘echo’ function
itself being isolated and importable elsewhere. When ‘echo.py’ is
imported, the ‘echo’ and ‘main’ functions will be defined, but neither
of them will be called, because ‘__name__ != '__main__'’.
File: python.info, Node: Packaging Considerations, Prev: Idiomatic Usage, Up: __name__ == '__main__'
5.30.5.4 Packaging Considerations
.................................
‘main’ functions are often used to create command-line tools by
specifying them as entry points for console scripts. When this is done,
pip(1) inserts the function call into a template script, where the
return value of ‘main’ is passed into *note sys.exit(): 133c. For
example:
sys.exit(main())
Since the call to ‘main’ is wrapped in *note sys.exit(): 133c, the
expectation is that your function will return some value acceptable as
an input to *note sys.exit(): 133c.; typically, an integer or ‘None’
(which is implicitly returned if your function does not have a return
statement).
By proactively following this convention ourselves, our module will have
the same behavior when run directly (i.e. ‘python echo.py’) as it will
have if we later package it as a console script entry-point in a
pip-installable package.
In particular, be careful about returning strings from your ‘main’
function. *note sys.exit(): 133c. will interpret a string argument as a
failure message, so your program will have an exit code of ‘1’,
indicating failure, and the string will be written to *note sys.stderr:
939. The ‘echo.py’ example from earlier exemplifies using the
‘sys.exit(main())’ convention.
See also
........
Python Packaging User Guide(2) contains a collection of tutorials and
references on how to distribute and install Python packages with modern
tools.
---------- Footnotes ----------
(1) https://pip.pypa.io/
(2) https://packaging.python.org/
File: python.info, Node: __main__ py in Python Packages, Next: import __main__, Prev: __name__ == '__main__', Up: __main__ — Top-level code environment
5.30.5.5 ‘__main__.py’ in Python Packages
.........................................
If you are not familiar with Python packages, see section *note
Packages: 1c67. of the tutorial. Most commonly, the ‘__main__.py’ file
is used to provide a command-line interface for a package. Consider the
following hypothetical package, “bandclass”:
bandclass
├── __init__.py
├── __main__.py
└── student.py
‘__main__.py’ will be executed when the package itself is invoked
directly from the command line using the *note -m: 5dd. flag. For
example:
$ python -m bandclass
This command will cause ‘__main__.py’ to run. How you utilize this
mechanism will depend on the nature of the package you are writing, but
in this hypothetical case, it might make sense to allow the teacher to
search for students:
# bandclass/__main__.py
import sys
from .student import search_students
student_name = sys.argv[1] if len(sys.argv) >= 2 else ''
print(f'Found student: {search_students(student_name)}')
Note that ‘from .student import search_students’ is an example of a
relative import. This import style can be used when referencing modules
within a package. For more details, see *note Intra-package References:
1c6c. in the *note Modules: 1c56. section of the tutorial.
* Menu:
* Idiomatic Usage: Idiomatic Usage<2>.
File: python.info, Node: Idiomatic Usage<2>, Up: __main__ py in Python Packages
5.30.5.6 Idiomatic Usage
........................
The content of ‘__main__.py’ typically isn’t fenced with an ‘if __name__
== '__main__'’ block. Instead, those files are kept short and import
functions to execute from other modules. Those other modules can then
be easily unit-tested and are properly reusable.
If used, an ‘if __name__ == '__main__'’ block will still work as
expected for a ‘__main__.py’ file within a package, because its
‘__name__’ attribute will include the package’s path if imported:
>>> import asyncio.__main__
>>> asyncio.__main__.__name__
'asyncio.__main__'
This won’t work for ‘__main__.py’ files in the root directory of a
‘.zip’ file though. Hence, for consistency, a minimal ‘__main__.py’
without a ‘__name__’ check is preferred.
See also
........
See *note venv: 111. for an example of a package with a minimal
‘__main__.py’ in the standard library. It doesn’t contain a ‘if
__name__ == '__main__'’ block. You can invoke it with ‘python -m venv
[directory]’.
See *note runpy: be. for more details on the *note -m: 5dd. flag to the
interpreter executable.
See *note zipapp: 130. for how to run applications packaged as '.zip'
files. In this case Python looks for a ‘__main__.py’ file in the root
directory of the archive.
File: python.info, Node: import __main__, Prev: __main__ py in Python Packages, Up: __main__ — Top-level code environment
5.30.5.7 ‘import __main__’
..........................
Regardless of which module a Python program was started with, other
modules running within that same program can import the top-level
environment’s scope (*note namespace: 1c57.) by importing the ‘__main__’
module. This doesn’t import a ‘__main__.py’ file but rather whichever
module that received the special name ‘'__main__'’.
Here is an example module that consumes the ‘__main__’ namespace:
# namely.py
import __main__
def did_user_define_their_name():
return 'my_name' in dir(__main__)
def print_user_name():
if not did_user_define_their_name():
raise ValueError('Define the variable `my_name`!')
if '__file__' in dir(__main__):
print(__main__.my_name, "found in file", __main__.__file__)
else:
print(__main__.my_name)
Example usage of this module could be as follows:
# start.py
import sys
from namely import print_user_name
# my_name = "Dinsdale"
def main():
try:
print_user_name()
except ValueError as ve:
return str(ve)
if __name__ == "__main__":
sys.exit(main())
Now, if we started our program, the result would look like this:
$ python start.py
Define the variable `my_name`!
The exit code of the program would be 1, indicating an error.
Uncommenting the line with ‘my_name = "Dinsdale"’ fixes the program and
now it exits with status code 0, indicating success:
$ python start.py
Dinsdale found in file /path/to/start.py
Note that importing ‘__main__’ doesn’t cause any issues with
unintentionally running top-level code meant for script use which is put
in the ‘if __name__ == "__main__"’ block of the ‘start’ module. Why
does this work?
Python inserts an empty ‘__main__’ module in *note sys.modules: 1550. at
interpreter startup, and populates it by running top-level code. In our
example this is the ‘start’ module which runs line by line and imports
‘namely’. In turn, ‘namely’ imports ‘__main__’ (which is really
‘start’). That’s an import cycle! Fortunately, since the partially
populated ‘__main__’ module is present in *note sys.modules: 1550,
Python passes that to ‘namely’. See *note Special considerations for
__main__: 200b. in the import system’s reference for details on how this
works.
The Python REPL is another example of a “top-level environment”, so
anything defined in the REPL becomes part of the ‘__main__’ scope:
>>> import namely
>>> namely.did_user_define_their_name()
False
>>> namely.print_user_name()
Traceback (most recent call last):
...
ValueError: Define the variable `my_name`!
>>> my_name = 'Jabberwocky'
>>> namely.did_user_define_their_name()
True
>>> namely.print_user_name()
Jabberwocky
Note that in this case the ‘__main__’ scope doesn’t contain a ‘__file__’
attribute as it’s interactive.
The ‘__main__’ scope is used in the implementation of *note pdb: a5. and
*note rlcompleter: bd.
File: python.info, Node: warnings — Warning control, Next: dataclasses — Data Classes, Prev: __main__ — Top-level code environment, Up: Python Runtime Services
5.30.6 ‘warnings’ — Warning control
-----------------------------------
'Source code:' Lib/warnings.py(1)
__________________________________________________________________
Warning messages are typically issued in situations where it is useful
to alert the user of some condition in a program, where that condition
(normally) doesn’t warrant raising an exception and terminating the
program. For example, one might want to issue a warning when a program
uses an obsolete module.
Python programmers issue warnings by calling the *note warn(): 14e4.
function defined in this module. (C programmers use *note
PyErr_WarnEx(): 1416.; see *note Exception Handling: 43fe. for details).
Warning messages are normally written to *note sys.stderr: 939, but
their disposition can be changed flexibly, from ignoring all warnings to
turning them into exceptions. The disposition of warnings can vary
based on the *note warning category: 22b2, the text of the warning
message, and the source location where it is issued. Repetitions of a
particular warning for the same source location are typically
suppressed.
There are two stages in warning control: first, each time a warning is
issued, a determination is made whether a message should be issued or
not; next, if a message is to be issued, it is formatted and printed
using a user-settable hook.
The determination whether to issue a warning message is controlled by
the *note warning filter: 8c8, which is a sequence of matching rules and
actions. Rules can be added to the filter by calling *note
filterwarnings(): 1472. and reset to its default state by calling *note
resetwarnings(): 43ff.
The printing of warning messages is done by calling *note showwarning():
4400, which may be overridden; the default implementation of this
function formats the message by calling *note formatwarning(): 1a14,
which is also available for use by custom implementations.
See also
........
*note logging.captureWarnings(): 2cb6. allows you to handle all warnings
with the standard logging infrastructure.
* Menu:
* Warning Categories::
* The Warnings Filter::
* Temporarily Suppressing Warnings::
* Testing Warnings::
* Updating Code For New Versions of Dependencies::
* Available Functions::
* Available Context Managers::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/warnings.py
File: python.info, Node: Warning Categories, Next: The Warnings Filter, Up: warnings — Warning control
5.30.6.1 Warning Categories
...........................
There are a number of built-in exceptions that represent warning
categories. This categorization is useful to be able to filter out
groups of warnings.
While these are technically *note built-in exceptions: 22b0, they are
documented here, because conceptually they belong to the warnings
mechanism.
User code can define additional warning categories by subclassing one of
the standard warning categories. A warning category must always be a
subclass of the *note Warning: 22b3. class.
The following warnings category classes are currently defined:
Class Description
-------------------------------------------------------------------------------------------
*note Warning: 22b3. This is the base class of all warning category
classes. It is a subclass of
*note Exception: 9d9.
*note UserWarning: 22b4. The default category for *note warn(): 14e4.
*note DeprecationWarning: 1a5. Base category for warnings about deprecated
features when those warnings are intended for
other Python developers (ignored by default,
unless triggered by code in ‘__main__’).
*note SyntaxWarning: 461. Base category for warnings about dubious
syntactic features.
*note RuntimeWarning: 1bd. Base category for warnings about dubious runtime
features.
*note FutureWarning: 411. Base category for warnings about deprecated
features when those warnings are intended for end
users of applications that are written in Python.
*note PendingDeprecationWarning: 8c7. Base category for warnings about features that
will be deprecated in the future (ignored by
default).
*note ImportWarning: 4f6. Base category for warnings triggered during the
process of importing a module (ignored by
default).
*note UnicodeWarning: 13ef. Base category for warnings related to Unicode.
*note BytesWarning: c23. Base category for warnings related to
*note bytes: 1c2. and *note bytearray: 53a.
*note ResourceWarning: 246. Base category for warnings related to resource
usage (ignored by default).
Changed in version 3.7: Previously *note DeprecationWarning: 1a5. and
*note FutureWarning: 411. were distinguished based on whether a feature
was being removed entirely or changing its behaviour. They are now
distinguished based on their intended audience and the way they’re
handled by the default warnings filters.
File: python.info, Node: The Warnings Filter, Next: Temporarily Suppressing Warnings, Prev: Warning Categories, Up: warnings — Warning control
5.30.6.2 The Warnings Filter
............................
The warnings filter controls whether warnings are ignored, displayed, or
turned into errors (raising an exception).
Conceptually, the warnings filter maintains an ordered list of filter
specifications; any specific warning is matched against each filter
specification in the list in turn until a match is found; the filter
determines the disposition of the match. Each entry is a tuple of the
form ('action', 'message', 'category', 'module', 'lineno'), where:
* 'action' is one of the following strings:
Value Disposition
-----------------------------------------------------------------------
‘"default"’ print the first occurrence of matching warnings
for each location (module + line number) where
the warning is issued
‘"error"’ turn matching warnings into exceptions
‘"ignore"’ never print matching warnings
‘"always"’ always print matching warnings
‘"module"’ print the first occurrence of matching warnings
for each module where the warning is issued
(regardless of line number)
‘"once"’ print only the first occurrence of matching
warnings, regardless of location
* 'message' is a string containing a regular expression that the
start of the warning message must match, case-insensitively. In
*note -W: 8c6. and *note PYTHONWARNINGS: baa, 'message' is a
literal string that the start of the warning message must contain
(case-insensitively), ignoring any whitespace at the start or end
of 'message'.
* 'category' is a class (a subclass of *note Warning: 22b3.) of which
the warning category must be a subclass in order to match.
* 'module' is a string containing a regular expression that the start
of the fully qualified module name must match, case-sensitively.
In *note -W: 8c6. and *note PYTHONWARNINGS: baa, 'module' is a
literal string that the fully qualified module name must be equal
to (case-sensitively), ignoring any whitespace at the start or end
of 'module'.
* 'lineno' is an integer that the line number where the warning
occurred must match, or ‘0’ to match all line numbers.
Since the *note Warning: 22b3. class is derived from the built-in *note
Exception: 9d9. class, to turn a warning into an error we simply raise
‘category(message)’.
If a warning is reported and doesn’t match any registered filter then
the “default” action is applied (hence its name).
* Menu:
* Repeated Warning Suppression Criteria::
* Describing Warning Filters::
* Default Warning Filter::
* Overriding the default filter::
File: python.info, Node: Repeated Warning Suppression Criteria, Next: Describing Warning Filters, Up: The Warnings Filter
5.30.6.3 Repeated Warning Suppression Criteria
..............................................
The filters that suppress repeated warnings apply the following criteria
to determine if a warning is considered a repeat:
- ‘"default"’: A warning is considered a repeat only if the
('message', 'category', 'module', 'lineno') are all the same.
- ‘"module"’: A warning is considered a repeat if the ('message',
'category', 'module') are the same, ignoring the line number.
- ‘"once"’: A warning is considered a repeat if the ('message',
'category') are the same, ignoring the module and line number.
File: python.info, Node: Describing Warning Filters, Next: Default Warning Filter, Prev: Repeated Warning Suppression Criteria, Up: The Warnings Filter
5.30.6.4 Describing Warning Filters
...................................
The warnings filter is initialized by *note -W: 8c6. options passed to
the Python interpreter command line and the *note PYTHONWARNINGS: baa.
environment variable. The interpreter saves the arguments for all
supplied entries without interpretation in *note sys.warnoptions: 3ac.;
the *note warnings: 112. module parses these when it is first imported
(invalid options are ignored, after printing a message to *note
sys.stderr: 939.).
Individual warnings filters are specified as a sequence of fields
separated by colons:
action:message:category:module:line
The meaning of each of these fields is as described in *note The
Warnings Filter: 8c8. When listing multiple filters on a single line
(as for *note PYTHONWARNINGS: baa.), the individual filters are
separated by commas and the filters listed later take precedence over
those listed before them (as they’re applied left-to-right, and the most
recently applied filters take precedence over earlier ones).
Commonly used warning filters apply to either all warnings, warnings in
a particular category, or warnings raised by particular modules or
packages. Some examples:
default # Show all warnings (even those ignored by default)
ignore # Ignore all warnings
error # Convert all warnings to errors
error::ResourceWarning # Treat ResourceWarning messages as errors
default::DeprecationWarning # Show DeprecationWarning messages
ignore,default:::mymodule # Only report warnings triggered by "mymodule"
error:::mymodule # Convert warnings to errors in "mymodule"
File: python.info, Node: Default Warning Filter, Next: Overriding the default filter, Prev: Describing Warning Filters, Up: The Warnings Filter
5.30.6.5 Default Warning Filter
...............................
By default, Python installs several warning filters, which can be
overridden by the *note -W: 8c6. command-line option, the *note
PYTHONWARNINGS: baa. environment variable and calls to *note
filterwarnings(): 1472.
In regular release builds, the default warning filter has the following
entries (in order of precedence):
default::DeprecationWarning:__main__
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ImportWarning
ignore::ResourceWarning
In a *note debug build: 1fb, the list of default warning filters is
empty.
Changed in version 3.2: *note DeprecationWarning: 1a5. is now ignored by
default in addition to *note PendingDeprecationWarning: 8c7.
Changed in version 3.7: *note DeprecationWarning: 1a5. is once again
shown by default when triggered directly by code in ‘__main__’.
Changed in version 3.7: *note BytesWarning: c23. no longer appears in
the default filter list and is instead configured via *note
sys.warnoptions: 3ac. when *note -b: 5df. is specified twice.
File: python.info, Node: Overriding the default filter, Prev: Default Warning Filter, Up: The Warnings Filter
5.30.6.6 Overriding the default filter
......................................
Developers of applications written in Python may wish to hide 'all'
Python level warnings from their users by default, and only display them
when running tests or otherwise working on the application. The *note
sys.warnoptions: 3ac. attribute used to pass filter configurations to
the interpreter can be used as a marker to indicate whether or not
warnings should be disabled:
import sys
if not sys.warnoptions:
import warnings
warnings.simplefilter("ignore")
Developers of test runners for Python code are advised to instead ensure
that 'all' warnings are displayed by default for the code under test,
using code like:
import sys
if not sys.warnoptions:
import os, warnings
warnings.simplefilter("default") # Change the filter in this process
os.environ["PYTHONWARNINGS"] = "default" # Also affect subprocesses
Finally, developers of interactive shells that run user code in a
namespace other than ‘__main__’ are advised to ensure that *note
DeprecationWarning: 1a5. messages are made visible by default, using
code like the following (where ‘user_ns’ is the module used to execute
code entered interactively):
import warnings
warnings.filterwarnings("default", category=DeprecationWarning,
module=user_ns.get("__name__"))
File: python.info, Node: Temporarily Suppressing Warnings, Next: Testing Warnings, Prev: The Warnings Filter, Up: warnings — Warning control
5.30.6.7 Temporarily Suppressing Warnings
.........................................
If you are using code that you know will raise a warning, such as a
deprecated function, but do not want to see the warning (even when
warnings have been explicitly configured via the command line), then it
is possible to suppress the warning using the *note catch_warnings: 581.
context manager:
import warnings
def fxn():
warnings.warn("deprecated", DeprecationWarning)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
fxn()
While within the context manager all warnings will simply be ignored.
This allows you to use known-deprecated code without having to see the
warning while not suppressing the warning for other code that might not
be aware of its use of deprecated code. Note: this can only be
guaranteed in a single-threaded application. If two or more threads use
the *note catch_warnings: 581. context manager at the same time, the
behavior is undefined.
File: python.info, Node: Testing Warnings, Next: Updating Code For New Versions of Dependencies, Prev: Temporarily Suppressing Warnings, Up: warnings — Warning control
5.30.6.8 Testing Warnings
.........................
To test warnings raised by code, use the *note catch_warnings: 581.
context manager. With it you can temporarily mutate the warnings filter
to facilitate your testing. For instance, do the following to capture
all raised warnings to check:
import warnings
def fxn():
warnings.warn("deprecated", DeprecationWarning)
with warnings.catch_warnings(record=True) as w:
# Cause all warnings to always be triggered.
warnings.simplefilter("always")
# Trigger a warning.
fxn()
# Verify some things
assert len(w) == 1
assert issubclass(w[-1].category, DeprecationWarning)
assert "deprecated" in str(w[-1].message)
One can also cause all warnings to be exceptions by using ‘error’
instead of ‘always’. One thing to be aware of is that if a warning has
already been raised because of a ‘once’/‘default’ rule, then no matter
what filters are set the warning will not be seen again unless the
warnings registry related to the warning has been cleared.
Once the context manager exits, the warnings filter is restored to its
state when the context was entered. This prevents tests from changing
the warnings filter in unexpected ways between tests and leading to
indeterminate test results. The *note showwarning(): 4400. function in
the module is also restored to its original value. Note: this can only
be guaranteed in a single-threaded application. If two or more threads
use the *note catch_warnings: 581. context manager at the same time, the
behavior is undefined.
When testing multiple operations that raise the same kind of warning, it
is important to test them in a manner that confirms each operation is
raising a new warning (e.g. set warnings to be raised as exceptions and
check the operations raise exceptions, check that the length of the
warning list continues to increase after each operation, or else delete
the previous entries from the warnings list before each new operation).
File: python.info, Node: Updating Code For New Versions of Dependencies, Next: Available Functions, Prev: Testing Warnings, Up: warnings — Warning control
5.30.6.9 Updating Code For New Versions of Dependencies
.......................................................
Warning categories that are primarily of interest to Python developers
(rather than end users of applications written in Python) are ignored by
default.
Notably, this “ignored by default” list includes *note
DeprecationWarning: 1a5. (for every module except ‘__main__’), which
means developers should make sure to test their code with typically
ignored warnings made visible in order to receive timely notifications
of future breaking API changes (whether in the standard library or third
party packages).
In the ideal case, the code will have a suitable test suite, and the
test runner will take care of implicitly enabling all warnings when
running tests (the test runner provided by the *note unittest: 106.
module does this).
In less ideal cases, applications can be checked for use of deprecated
interfaces by passing *note -Wd: 8c6. to the Python interpreter (this is
shorthand for ‘-W default’) or setting ‘PYTHONWARNINGS=default’ in the
environment. This enables default handling for all warnings, including
those that are ignored by default. To change what action is taken for
encountered warnings you can change what argument is passed to *note -W:
8c6. (e.g. ‘-W error’). See the *note -W: 8c6. flag for more details
on what is possible.
File: python.info, Node: Available Functions, Next: Available Context Managers, Prev: Updating Code For New Versions of Dependencies, Up: warnings — Warning control
5.30.6.10 Available Functions
.............................
-- Function: warnings.warn (message, category=None, stacklevel=1,
source=None, *, skip_file_prefixes=())
Issue a warning, or maybe ignore it or raise an exception. The
'category' argument, if given, must be a *note warning category
class: 22b2.; it defaults to *note UserWarning: 22b4.
Alternatively, 'message' can be a *note Warning: 22b3. instance, in
which case 'category' will be ignored and ‘message.__class__’ will
be used. In this case, the message text will be ‘str(message)’.
This function raises an exception if the particular warning issued
is changed into an error by the *note warnings filter: 8c8. The
'stacklevel' argument can be used by wrapper functions written in
Python, like this:
def deprecated_api(message):
warnings.warn(message, DeprecationWarning, stacklevel=2)
This makes the warning refer to ‘deprecated_api’’s caller, rather
than to the source of ‘deprecated_api’ itself (since the latter
would defeat the purpose of the warning message).
The 'skip_file_prefixes' keyword argument can be used to indicate
which stack frames are ignored when counting stack levels. This
can be useful when you want the warning to always appear at call
sites outside of a package when a constant 'stacklevel' does not
fit all call paths or is otherwise challenging to maintain. If
supplied, it must be a tuple of strings. When prefixes are
supplied, stacklevel is implicitly overridden to be ‘max(2,
stacklevel)’. To cause a warning to be attributed to the caller
from outside of the current package you might write:
# example/lower.py
_warn_skips = (os.path.dirname(__file__),)
def one_way(r_luxury_yacht=None, t_wobbler_mangrove=None):
if r_luxury_yacht:
warnings.warn("Please migrate to t_wobbler_mangrove=.",
skip_file_prefixes=_warn_skips)
# example/higher.py
from . import lower
def another_way(**kw):
lower.one_way(**kw)
This makes the warning refer to both the ‘example.lower.one_way()’
and ‘example.higher.another_way()’ call sites only from calling
code living outside of ‘example’ package.
'source', if supplied, is the destroyed object which emitted a
*note ResourceWarning: 246.
Changed in version 3.6: Added 'source' parameter.
Changed in version 3.12: Added 'skip_file_prefixes'.
-- Function: warnings.warn_explicit (message, category, filename,
lineno, module=None, registry=None, module_globals=None,
source=None)
This is a low-level interface to the functionality of *note warn():
14e4, passing in explicitly the message, category, filename and
line number, and optionally the module name and the registry (which
should be the ‘__warningregistry__’ dictionary of the module). The
module name defaults to the filename with ‘.py’ stripped; if no
registry is passed, the warning is never suppressed. 'message'
must be a string and 'category' a subclass of *note Warning: 22b3.
or 'message' may be a *note Warning: 22b3. instance, in which case
'category' will be ignored.
'module_globals', if supplied, should be the global namespace in
use by the code for which the warning is issued. (This argument is
used to support displaying source for modules found in zipfiles or
other non-filesystem import sources).
'source', if supplied, is the destroyed object which emitted a
*note ResourceWarning: 246.
Changed in version 3.6: Add the 'source' parameter.
-- Function: warnings.showwarning (message, category, filename, lineno,
file=None, line=None)
Write a warning to a file. The default implementation calls
‘formatwarning(message, category, filename, lineno, line)’ and
writes the resulting string to 'file', which defaults to *note
sys.stderr: 939. You may replace this function with any callable
by assigning to ‘warnings.showwarning’. 'line' is a line of source
code to be included in the warning message; if 'line' is not
supplied, *note showwarning(): 4400. will try to read the line
specified by 'filename' and 'lineno'.
-- Function: warnings.formatwarning (message, category, filename,
lineno, line=None)
Format a warning the standard way. This returns a string which may
contain embedded newlines and ends in a newline. 'line' is a line
of source code to be included in the warning message; if 'line' is
not supplied, *note formatwarning(): 1a14. will try to read the
line specified by 'filename' and 'lineno'.
-- Function: warnings.filterwarnings (action, message='',
category=Warning, module='', lineno=0, append=False)
Insert an entry into the list of *note warnings filter
specifications: 8c8. The entry is inserted at the front by
default; if 'append' is true, it is inserted at the end. This
checks the types of the arguments, compiles the 'message' and
'module' regular expressions, and inserts them as a tuple in the
list of warnings filters. Entries closer to the front of the list
override entries later in the list, if both match a particular
warning. Omitted arguments default to a value that matches
everything.
-- Function: warnings.simplefilter (action, category=Warning, lineno=0,
append=False)
Insert a simple entry into the list of *note warnings filter
specifications: 8c8. The meaning of the function parameters is as
for *note filterwarnings(): 1472, but regular expressions are not
needed as the filter inserted always matches any message in any
module as long as the category and line number match.
-- Function: warnings.resetwarnings ()
Reset the warnings filter. This discards the effect of all
previous calls to *note filterwarnings(): 1472, including that of
the *note -W: 8c6. command line options and calls to *note
simplefilter(): 6bd.
-- Function: @warnings.deprecated (msg, *, category=DeprecationWarning,
stacklevel=1)
Decorator to indicate that a class, function or overload is
deprecated.
When this decorator is applied to an object, deprecation warnings
may be emitted at runtime when the object is used. *note static
type checkers: 26f. will also generate a diagnostic on usage of the
deprecated object.
Usage:
from warnings import deprecated
from typing import overload
@deprecated("Use B instead")
class A:
pass
@deprecated("Use g instead")
def f():
pass
@overload
@deprecated("int support is deprecated")
def g(x: int) -> int: ...
@overload
def g(x: str) -> int: ...
The warning specified by 'category' will be emitted at runtime on
use of deprecated objects. For functions, that happens on calls;
for classes, on instantiation and on creation of subclasses. If
the 'category' is ‘None’, no warning is emitted at runtime. The
'stacklevel' determines where the warning is emitted. If it is ‘1’
(the default), the warning is emitted at the direct caller of the
deprecated object; if it is higher, it is emitted further up the
stack. Static type checker behavior is not affected by the
'category' and 'stacklevel' arguments.
The deprecation message passed to the decorator is saved in the
‘__deprecated__’ attribute on the decorated object. If applied to
an overload, the decorator must be after the *note @overload: 3fb3.
decorator for the attribute to exist on the overload as returned by
*note typing.get_overloads(): 6aa.
Added in version 3.13: See PEP 702(1).
---------- Footnotes ----------
(1) https://peps.python.org/pep-0702/
File: python.info, Node: Available Context Managers, Prev: Available Functions, Up: warnings — Warning control
5.30.6.11 Available Context Managers
....................................
-- Class: warnings.catch_warnings (*, record=False, module=None,
action=None, category=Warning, lineno=0, append=False)
A context manager that copies and, upon exit, restores the warnings
filter and the *note showwarning(): 4400. function. If the
'record' argument is *note False: b37. (the default) the context
manager returns *note None: 671. on entry. If 'record' is *note
True: c0d, a list is returned that is progressively populated with
objects as seen by a custom *note showwarning(): 4400. function
(which also suppresses output to ‘sys.stdout’). Each object in the
list has attributes with the same names as the arguments to *note
showwarning(): 4400.
The 'module' argument takes a module that will be used instead of
the module returned when you import *note warnings: 112. whose
filter will be protected. This argument exists primarily for
testing the *note warnings: 112. module itself.
If the 'action' argument is not ‘None’, the remaining arguments are
passed to *note simplefilter(): 6bd. as if it were called
immediately on entering the context.
See *note The Warnings Filter: 8c8. for the meaning of the
'category' and 'lineno' parameters.
Note: The *note catch_warnings: 581. manager works by
replacing and then later restoring the module’s *note
showwarning(): 4400. function and internal list of filter
specifications. This means the context manager is modifying
global state and therefore is not thread-safe.
Changed in version 3.11: Added the 'action', 'category', 'lineno',
and 'append' parameters.
File: python.info, Node: dataclasses — Data Classes, Next: contextlib — Utilities for with-statement contexts, Prev: warnings — Warning control, Up: Python Runtime Services
5.30.7 ‘dataclasses’ — Data Classes
-----------------------------------
'Source code:' Lib/dataclasses.py(1)
__________________________________________________________________
This module provides a decorator and functions for automatically adding
generated *note special methods: 18ab. such as *note __init__(): 6ac.
and *note __repr__(): 618. to user-defined classes. It was originally
described in PEP 557(2).
The member variables to use in these generated methods are defined using
PEP 526(3) type annotations. For example, this code:
from dataclasses import dataclass
@dataclass
class InventoryItem:
"""Class for keeping track of an item in inventory."""
name: str
unit_price: float
quantity_on_hand: int = 0
def total_cost(self) -> float:
return self.unit_price * self.quantity_on_hand
will add, among other things, a ‘__init__()’ that looks like:
def __init__(self, name: str, unit_price: float, quantity_on_hand: int = 0):
self.name = name
self.unit_price = unit_price
self.quantity_on_hand = quantity_on_hand
Note that this method is automatically added to the class: it is not
directly specified in the ‘InventoryItem’ definition shown above.
Added in version 3.7.
* Menu:
* Module contents: Module contents<4>.
* Post-init processing::
* Class variables::
* Init-only variables::
* Frozen instances::
* Inheritance: Inheritance<2>.
* Re-ordering of keyword-only parameters in __init__(): Re-ordering of keyword-only parameters in __init__.
* Default factory functions::
* Mutable default values::
* Descriptor-typed fields::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/dataclasses.py
(2) https://peps.python.org/pep-0557/
(3) https://peps.python.org/pep-0526/
File: python.info, Node: Module contents<4>, Next: Post-init processing, Up: dataclasses — Data Classes
5.30.7.1 Module contents
........................
-- Function: @dataclasses.dataclass (*, init=True, repr=True, eq=True,
order=False, unsafe_hash=False, frozen=False, match_args=True,
kw_only=False, slots=False, weakref_slot=False)
This function is a *note decorator: 72c. that is used to add
generated *note special methods: 18ab. to classes, as described
below.
The ‘@dataclass’ decorator examines the class to find ‘field’s. A
‘field’ is defined as a class variable that has a *note type
annotation: d55. With two exceptions described below, nothing in
‘@dataclass’ examines the type specified in the variable
annotation.
The order of the fields in all of the generated methods is the
order in which they appear in the class definition.
The ‘@dataclass’ decorator will add various “dunder” methods to the
class, described below. If any of the added methods already exist
in the class, the behavior depends on the parameter, as documented
below. The decorator returns the same class that it is called on;
no new class is created.
If ‘@dataclass’ is used just as a simple decorator with no
parameters, it acts as if it has the default values documented in
this signature. That is, these three uses of ‘@dataclass’ are
equivalent:
@dataclass
class C:
...
@dataclass()
class C:
...
@dataclass(init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False,
match_args=True, kw_only=False, slots=False, weakref_slot=False)
class C:
...
The parameters to ‘@dataclass’ are:
- 'init': If true (the default), a *note __init__(): 6ac. method
will be generated.
If the class already defines ‘__init__()’, this parameter is
ignored.
- 'repr': If true (the default), a *note __repr__(): 618. method
will be generated. The generated repr string will have the
class name and the name and repr of each field, in the order
they are defined in the class. Fields that are marked as
being excluded from the repr are not included. For example:
‘InventoryItem(name='widget', unit_price=3.0,
quantity_on_hand=10)’.
If the class already defines ‘__repr__()’, this parameter is
ignored.
- 'eq': If true (the default), an *note __eq__(): afa. method
will be generated. This method compares the class as if it
were a tuple of its fields, in order. Both instances in the
comparison must be of the identical type.
If the class already defines ‘__eq__()’, this parameter is
ignored.
- 'order': If true (the default is ‘False’), *note __lt__():
1292, *note __le__(): 12fe, *note __gt__(): 12ff, and *note
__ge__(): 1300. methods will be generated. These compare the
class as if it were a tuple of its fields, in order. Both
instances in the comparison must be of the identical type. If
'order' is true and 'eq' is false, a *note ValueError: 204. is
raised.
If the class already defines any of ‘__lt__()’, ‘__le__()’,
‘__gt__()’, or ‘__ge__()’, then *note TypeError: 534. is
raised.
- 'unsafe_hash': If true, force ‘dataclasses’ to create a *note
__hash__(): afb. method, even though it may not be safe to do
so. Otherwise, generate a *note __hash__(): afb. method
according to how 'eq' and 'frozen' are set. The default value
is ‘False’.
‘__hash__()’ is used by built-in *note hash(): 5e7, and when
objects are added to hashed collections such as dictionaries
and sets. Having a ‘__hash__()’ implies that instances of the
class are immutable. Mutability is a complicated property
that depends on the programmer’s intent, the existence and
behavior of ‘__eq__()’, and the values of the 'eq' and
'frozen' flags in the ‘@dataclass’ decorator.
By default, ‘@dataclass’ will not implicitly add a *note
__hash__(): afb. method unless it is safe to do so. Neither
will it add or change an existing explicitly defined
‘__hash__()’ method. Setting the class attribute ‘__hash__ =
None’ has a specific meaning to Python, as described in the
‘__hash__()’ documentation.
If ‘__hash__()’ is not explicitly defined, or if it is set to
‘None’, then ‘@dataclass’ 'may' add an implicit ‘__hash__()’
method. Although not recommended, you can force ‘@dataclass’
to create a ‘__hash__()’ method with ‘unsafe_hash=True’. This
might be the case if your class is logically immutable but can
still be mutated. This is a specialized use case and should
be considered carefully.
Here are the rules governing implicit creation of a
‘__hash__()’ method. Note that you cannot both have an
explicit ‘__hash__()’ method in your dataclass and set
‘unsafe_hash=True’; this will result in a *note TypeError:
534.
If 'eq' and 'frozen' are both true, by default ‘@dataclass’
will generate a ‘__hash__()’ method for you. If 'eq' is true
and 'frozen' is false, ‘__hash__()’ will be set to ‘None’,
marking it unhashable (which it is, since it is mutable). If
'eq' is false, ‘__hash__()’ will be left untouched meaning the
‘__hash__()’ method of the superclass will be used (if the
superclass is *note object: a8c, this means it will fall back
to id-based hashing).
- 'frozen': If true (the default is ‘False’), assigning to
fields will generate an exception. This emulates read-only
frozen instances. If *note __setattr__(): 1f2d. or *note
__delattr__(): 1f2e. is defined in the class, then *note
TypeError: 534. is raised. See the discussion below.
- 'match_args': If true (the default is ‘True’), the *note
__match_args__: 18bf. tuple will be created from the list of
non keyword-only parameters to the generated *note __init__():
6ac. method (even if ‘__init__()’ is not generated, see
above). If false, or if ‘__match_args__’ is already defined
in the class, then ‘__match_args__’ will not be generated.
Added in version 3.10.
- 'kw_only': If true (the default value is ‘False’), then all
fields will be marked as keyword-only. If a field is marked
as keyword-only, then the only effect is that the *note
__init__(): 6ac. parameter generated from a keyword-only field
must be specified with a keyword when ‘__init__()’ is called.
See the *note parameter: 1eff. glossary entry for details.
Also see the *note KW_ONLY: 4415. section.
Keyword-only fields are not included in ‘__match_args__’.
Added in version 3.10.
- 'slots': If true (the default is ‘False’), *note __slots__:
14b5. attribute will be generated and new class will be
returned instead of the original one. If ‘__slots__’ is
already defined in the class, then *note TypeError: 534. is
raised.
Warning: Calling no-arg *note super(): 4d7. in
dataclasses using ‘slots=True’ will result in the
following exception being raised: ‘TypeError: super(type,
obj): obj must be an instance or subtype of type’. The
two-arg *note super(): 4d7. is a valid workaround. See
gh-90562(1) for full details.
Warning: Passing parameters to a base class *note
__init_subclass__(): 62d. when using ‘slots=True’ will
result in a *note TypeError: 534. Either use
‘__init_subclass__’ with no parameters or use default
values as a workaround. See gh-91126(2) for full
details.
Added in version 3.10.
Changed in version 3.11: If a field name is already included
in the ‘__slots__’ of a base class, it will not be included in
the generated ‘__slots__’ to prevent *note overriding them:
1f6f. Therefore, do not use ‘__slots__’ to retrieve the field
names of a dataclass. Use *note fields(): 1742. instead. To
be able to determine inherited slots, base class ‘__slots__’
may be any iterable, but 'not' an iterator.
- 'weakref_slot': If true (the default is ‘False’), add a slot
named “__weakref__”, which is required to make an instance
*note weakref-able: fe9. It is an error to specify
‘weakref_slot=True’ without also specifying ‘slots=True’.
Added in version 3.11.
‘field’s may optionally specify a default value, using normal
Python syntax:
@dataclass
class C:
a: int # 'a' has no default value
b: int = 0 # assign a default value for 'b'
In this example, both ‘a’ and ‘b’ will be included in the added
*note __init__(): 6ac. method, which will be defined as:
def __init__(self, a: int, b: int = 0):
*note TypeError: 534. will be raised if a field without a default
value follows a field with a default value. This is true whether
this occurs in a single class, or as a result of class inheritance.
-- Function: dataclasses.field (*, default=MISSING,
default_factory=MISSING, init=True, repr=True, hash=None,
compare=True, metadata=None, kw_only=MISSING)
For common and simple use cases, no other functionality is
required. There are, however, some dataclass features that require
additional per-field information. To satisfy this need for
additional information, you can replace the default field value
with a call to the provided ‘field()’ function. For example:
@dataclass
class C:
mylist: list[int] = field(default_factory=list)
c = C()
c.mylist += [1, 2, 3]
As shown above, the *note MISSING: 4416. value is a sentinel object
used to detect if some parameters are provided by the user. This
sentinel is used because ‘None’ is a valid value for some
parameters with a distinct meaning. No code should directly use
the *note MISSING: 4416. value.
The parameters to ‘field()’ are:
- 'default': If provided, this will be the default value for
this field. This is needed because the ‘field()’ call itself
replaces the normal position of the default value.
- 'default_factory': If provided, it must be a zero-argument
callable that will be called when a default value is needed
for this field. Among other purposes, this can be used to
specify fields with mutable default values, as discussed
below. It is an error to specify both 'default' and
'default_factory'.
- 'init': If true (the default), this field is included as a
parameter to the generated *note __init__(): 6ac. method.
- 'repr': If true (the default), this field is included in the
string returned by the generated *note __repr__(): 618.
method.
- 'hash': This can be a bool or ‘None’. If true, this field is
included in the generated *note __hash__(): afb. method. If
false, this field is excluded from the generated *note
__hash__(): afb. If ‘None’ (the default), use the value of
'compare': this would normally be the expected behavior, since
a field should be included in the hash if it’s used for
comparisons. Setting this value to anything other than ‘None’
is discouraged.
One possible reason to set ‘hash=False’ but ‘compare=True’
would be if a field is expensive to compute a hash value for,
that field is needed for equality testing, and there are other
fields that contribute to the type’s hash value. Even if a
field is excluded from the hash, it will still be used for
comparisons.
- 'compare': If true (the default), this field is included in
the generated equality and comparison methods (*note __eq__():
afa, *note __gt__(): 12ff, et al.).
- 'metadata': This can be a mapping or ‘None’. ‘None’ is
treated as an empty dict. This value is wrapped in *note
MappingProxyType(): 469. to make it read-only, and exposed on
the *note Field: 2255. object. It is not used at all by Data
Classes, and is provided as a third-party extension mechanism.
Multiple third-parties can each have their own key, to use as
a namespace in the metadata.
- 'kw_only': If true, this field will be marked as keyword-only.
This is used when the generated *note __init__(): 6ac.
method’s parameters are computed.
Keyword-only fields are also not included in ‘__match_args__’.
Added in version 3.10.
If the default value of a field is specified by a call to
‘field()’, then the class attribute for this field will be replaced
by the specified 'default' value. If 'default' is not provided,
then the class attribute will be deleted. The intent is that after
the *note @dataclass: 1cb. decorator runs, the class attributes
will all contain the default values for the fields, just as if the
default value itself were specified. For example, after:
@dataclass
class C:
x: int
y: int = field(repr=False)
z: int = field(repr=False, default=10)
t: int = 20
The class attribute ‘C.z’ will be ‘10’, the class attribute ‘C.t’
will be ‘20’, and the class attributes ‘C.x’ and ‘C.y’ will not be
set.
-- Class: dataclasses.Field
‘Field’ objects describe each defined field. These objects are
created internally, and are returned by the *note fields(): 1742.
module-level method (see below). Users should never instantiate a
‘Field’ object directly. Its documented attributes are:
- ‘name’: The name of the field.
- ‘type’: The type of the field.
- ‘default’, ‘default_factory’, ‘init’, ‘repr’, ‘hash’,
‘compare’, ‘metadata’, and ‘kw_only’ have the identical
meaning and values as they do in the *note field(): 1a1f.
function.
Other attributes may exist, but they are private and must not be
inspected or relied on.
-- Class: dataclasses.InitVar
‘InitVar[T]’ type annotations describe variables that are *note
init-only: 4417. Fields annotated with ‘InitVar’ are considered
pseudo-fields, and thus are neither returned by the *note fields():
1742. function nor used in any way except adding them as parameters
to *note __init__(): 6ac. and an optional *note __post_init__():
4418.
-- Function: dataclasses.fields (class_or_instance)
Returns a tuple of *note Field: 2255. objects that define the
fields for this dataclass. Accepts either a dataclass, or an
instance of a dataclass. Raises *note TypeError: 534. if not
passed a dataclass or instance of one. Does not return
pseudo-fields which are ‘ClassVar’ or ‘InitVar’.
-- Function: dataclasses.asdict (obj, *, dict_factory=dict)
Converts the dataclass 'obj' to a dict (by using the factory
function 'dict_factory'). Each dataclass is converted to a dict of
its fields, as ‘name: value’ pairs. dataclasses, dicts, lists, and
tuples are recursed into. Other objects are copied with *note
copy.deepcopy(): b6e.
Example of using ‘asdict()’ on nested dataclasses:
@dataclass
class Point:
x: int
y: int
@dataclass
class C:
mylist: list[Point]
p = Point(10, 20)
assert asdict(p) == {'x': 10, 'y': 20}
c = C([Point(0, 0), Point(10, 4)])
assert asdict(c) == {'mylist': [{'x': 0, 'y': 0}, {'x': 10, 'y': 4}]}
To create a shallow copy, the following workaround may be used:
{field.name: getattr(obj, field.name) for field in fields(obj)}
‘asdict()’ raises *note TypeError: 534. if 'obj' is not a dataclass
instance.
-- Function: dataclasses.astuple (obj, *, tuple_factory=tuple)
Converts the dataclass 'obj' to a tuple (by using the factory
function 'tuple_factory'). Each dataclass is converted to a tuple
of its field values. dataclasses, dicts, lists, and tuples are
recursed into. Other objects are copied with *note
copy.deepcopy(): b6e.
Continuing from the previous example:
assert astuple(p) == (10, 20)
assert astuple(c) == ([(0, 0), (10, 4)],)
To create a shallow copy, the following workaround may be used:
tuple(getattr(obj, field.name) for field in dataclasses.fields(obj))
‘astuple()’ raises *note TypeError: 534. if 'obj' is not a
dataclass instance.
-- Function: dataclasses.make_dataclass (cls_name, fields, *, bases=(),
namespace=None, init=True, repr=True, eq=True, order=False,
unsafe_hash=False, frozen=False, match_args=True,
kw_only=False, slots=False, weakref_slot=False, module=None)
Creates a new dataclass with name 'cls_name', fields as defined in
'fields', base classes as given in 'bases', and initialized with a
namespace as given in 'namespace'. 'fields' is an iterable whose
elements are each either ‘name’, ‘(name, type)’, or ‘(name, type,
Field)’. If just ‘name’ is supplied, *note typing.Any: 6a8. is
used for ‘type’. The values of 'init', 'repr', 'eq', 'order',
'unsafe_hash', 'frozen', 'match_args', 'kw_only', 'slots', and
'weakref_slot' have the same meaning as they do in *note
@dataclass: 1cb.
If 'module' is defined, the ‘__module__’ attribute of the dataclass
is set to that value. By default, it is set to the module name of
the caller.
This function is not strictly required, because any Python
mechanism for creating a new class with ‘__annotations__’ can then
apply the *note @dataclass: 1cb. function to convert that class to
a dataclass. This function is provided as a convenience. For
example:
C = make_dataclass('C',
[('x', int),
'y',
('z', int, field(default=5))],
namespace={'add_one': lambda self: self.x + 1})
Is equivalent to:
@dataclass
class C:
x: int
y: 'typing.Any'
z: int = 5
def add_one(self):
return self.x + 1
-- Function: dataclasses.replace (obj, /, **changes)
Creates a new object of the same type as 'obj', replacing fields
with values from 'changes'. If 'obj' is not a Data Class, raises
*note TypeError: 534. If keys in 'changes' are not field names of
the given dataclass, raises *note TypeError: 534.
The newly returned object is created by calling the *note
__init__(): 6ac. method of the dataclass. This ensures that *note
__post_init__(): 4418, if present, is also called.
Init-only variables without default values, if any exist, must be
specified on the call to ‘replace()’ so that they can be passed to
‘__init__()’ and *note __post_init__(): 4418.
It is an error for 'changes' to contain any fields that are defined
as having ‘init=False’. A *note ValueError: 204. will be raised in
this case.
Be forewarned about how ‘init=False’ fields work during a call to
‘replace()’. They are not copied from the source object, but
rather are initialized in *note __post_init__(): 4418, if they’re
initialized at all. It is expected that ‘init=False’ fields will
be rarely and judiciously used. If they are used, it might be wise
to have alternate class constructors, or perhaps a custom
‘replace()’ (or similarly named) method which handles instance
copying.
Dataclass instances are also supported by generic function *note
copy.replace(): 151.
-- Function: dataclasses.is_dataclass (obj)
Return ‘True’ if its parameter is a dataclass (including subclasses
of a dataclass) or an instance of one, otherwise return ‘False’.
If you need to know if a class is an instance of a dataclass (and
not a dataclass itself), then add a further check for ‘not
isinstance(obj, type)’:
def is_dataclass_instance(obj):
return is_dataclass(obj) and not isinstance(obj, type)
-- Data: dataclasses.MISSING
A sentinel value signifying a missing default or default_factory.
-- Data: dataclasses.KW_ONLY
A sentinel value used as a type annotation. Any fields after a
pseudo-field with the type of ‘KW_ONLY’ are marked as keyword-only
fields. Note that a pseudo-field of type ‘KW_ONLY’ is otherwise
completely ignored. This includes the name of such a field. By
convention, a name of ‘_’ is used for a ‘KW_ONLY’ field.
Keyword-only fields signify *note __init__(): 6ac. parameters that
must be specified as keywords when the class is instantiated.
In this example, the fields ‘y’ and ‘z’ will be marked as
keyword-only fields:
@dataclass
class Point:
x: float
_: KW_ONLY
y: float
z: float
p = Point(0, y=1.5, z=2.0)
In a single dataclass, it is an error to specify more than one
field whose type is ‘KW_ONLY’.
Added in version 3.10.
-- Exception: dataclasses.FrozenInstanceError
Raised when an implicitly defined *note __setattr__(): 1f2d. or
*note __delattr__(): 1f2e. is called on a dataclass which was
defined with ‘frozen=True’. It is a subclass of *note
AttributeError: 348.
---------- Footnotes ----------
(1) https://github.com/python/cpython/issues/90562
(2) https://github.com/python/cpython/issues/91126
File: python.info, Node: Post-init processing, Next: Class variables, Prev: Module contents<4>, Up: dataclasses — Data Classes
5.30.7.2 Post-init processing
.............................
-- Function: dataclasses.__post_init__ ()
When defined on the class, it will be called by the generated *note
__init__(): 6ac, normally as ‘self.__post_init__()’. However, if
any ‘InitVar’ fields are defined, they will also be passed to
‘__post_init__()’ in the order they were defined in the class. If
no ‘__init__()’ method is generated, then ‘__post_init__()’ will
not automatically be called.
Among other uses, this allows for initializing field values that
depend on one or more other fields. For example:
@dataclass
class C:
a: float
b: float
c: float = field(init=False)
def __post_init__(self):
self.c = self.a + self.b
The *note __init__(): 6ac. method generated by *note @dataclass: 1cb.
does not call base class ‘__init__()’ methods. If the base class has an
‘__init__()’ method that has to be called, it is common to call this
method in a *note __post_init__(): 4418. method:
class Rectangle:
def __init__(self, height, width):
self.height = height
self.width = width
@dataclass
class Square(Rectangle):
side: float
def __post_init__(self):
super().__init__(self.side, self.side)
Note, however, that in general the dataclass-generated ‘__init__()’
methods don’t need to be called, since the derived dataclass will take
care of initializing all fields of any base class that is a dataclass
itself.
See the section below on init-only variables for ways to pass parameters
to ‘__post_init__()’. Also see the warning about how *note replace():
16f9. handles ‘init=False’ fields.
File: python.info, Node: Class variables, Next: Init-only variables, Prev: Post-init processing, Up: dataclasses — Data Classes
5.30.7.3 Class variables
........................
One of the few places where *note @dataclass: 1cb. actually inspects the
type of a field is to determine if a field is a class variable as
defined in PEP 526(1). It does this by checking if the type of the
field is *note typing.ClassVar: 268. If a field is a ‘ClassVar’, it is
excluded from consideration as a field and is ignored by the dataclass
mechanisms. Such ‘ClassVar’ pseudo-fields are not returned by the
module-level *note fields(): 1742. function.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0526/
File: python.info, Node: Init-only variables, Next: Frozen instances, Prev: Class variables, Up: dataclasses — Data Classes
5.30.7.4 Init-only variables
............................
Another place where *note @dataclass: 1cb. inspects a type annotation is
to determine if a field is an init-only variable. It does this by
seeing if the type of a field is of type *note InitVar: 1822. If a
field is an *note InitVar: 1822, it is considered a pseudo-field called
an init-only field. As it is not a true field, it is not returned by
the module-level *note fields(): 1742. function. Init-only fields are
added as parameters to the generated *note __init__(): 6ac. method, and
are passed to the optional *note __post_init__(): 4418. method. They
are not otherwise used by dataclasses.
For example, suppose a field will be initialized from a database, if a
value is not provided when creating the class:
@dataclass
class C:
i: int
j: int | None = None
database: InitVar[DatabaseType | None] = None
def __post_init__(self, database):
if self.j is None and database is not None:
self.j = database.lookup('j')
c = C(10, database=my_database)
In this case, *note fields(): 1742. will return *note Field: 2255.
objects for ‘i’ and ‘j’, but not for ‘database’.
File: python.info, Node: Frozen instances, Next: Inheritance<2>, Prev: Init-only variables, Up: dataclasses — Data Classes
5.30.7.5 Frozen instances
.........................
It is not possible to create truly immutable Python objects. However,
by passing ‘frozen=True’ to the *note @dataclass: 1cb. decorator you can
emulate immutability. In that case, dataclasses will add *note
__setattr__(): 1f2d. and *note __delattr__(): 1f2e. methods to the
class. These methods will raise a *note FrozenInstanceError: 4419. when
invoked.
There is a tiny performance penalty when using ‘frozen=True’: *note
__init__(): 6ac. cannot use simple assignment to initialize fields, and
must use ‘object.__setattr__()’.
File: python.info, Node: Inheritance<2>, Next: Re-ordering of keyword-only parameters in __init__, Prev: Frozen instances, Up: dataclasses — Data Classes
5.30.7.6 Inheritance
....................
When the dataclass is being created by the *note @dataclass: 1cb.
decorator, it looks through all of the class’s base classes in reverse
MRO (that is, starting at *note object: a8c.) and, for each dataclass
that it finds, adds the fields from that base class to an ordered
mapping of fields. After all of the base class fields are added, it
adds its own fields to the ordered mapping. All of the generated
methods will use this combined, calculated ordered mapping of fields.
Because the fields are in insertion order, derived classes override base
classes. An example:
@dataclass
class Base:
x: Any = 15.0
y: int = 0
@dataclass
class C(Base):
z: int = 10
x: int = 15
The final list of fields is, in order, ‘x’, ‘y’, ‘z’. The final type of
‘x’ is *note int: 259, as specified in class ‘C’.
The generated *note __init__(): 6ac. method for ‘C’ will look like:
def __init__(self, x: int = 15, y: int = 0, z: int = 10):
File: python.info, Node: Re-ordering of keyword-only parameters in __init__, Next: Default factory functions, Prev: Inheritance<2>, Up: dataclasses — Data Classes
5.30.7.7 Re-ordering of keyword-only parameters in ‘__init__()’
...............................................................
After the parameters needed for *note __init__(): 6ac. are computed, any
keyword-only parameters are moved to come after all regular
(non-keyword-only) parameters. This is a requirement of how
keyword-only parameters are implemented in Python: they must come after
non-keyword-only parameters.
In this example, ‘Base.y’, ‘Base.w’, and ‘D.t’ are keyword-only fields,
and ‘Base.x’ and ‘D.z’ are regular fields:
@dataclass
class Base:
x: Any = 15.0
_: KW_ONLY
y: int = 0
w: int = 1
@dataclass
class D(Base):
z: int = 10
t: int = field(kw_only=True, default=0)
The generated ‘__init__()’ method for ‘D’ will look like:
def __init__(self, x: Any = 15.0, z: int = 10, *, y: int = 0, w: int = 1, t: int = 0):
Note that the parameters have been re-ordered from how they appear in
the list of fields: parameters derived from regular fields are followed
by parameters derived from keyword-only fields.
The relative ordering of keyword-only parameters is maintained in the
re-ordered ‘__init__()’ parameter list.
File: python.info, Node: Default factory functions, Next: Mutable default values, Prev: Re-ordering of keyword-only parameters in __init__, Up: dataclasses — Data Classes
5.30.7.8 Default factory functions
..................................
If a *note field(): 1a1f. specifies a 'default_factory', it is called
with zero arguments when a default value for the field is needed. For
example, to create a new instance of a list, use:
mylist: list = field(default_factory=list)
If a field is excluded from *note __init__(): 6ac. (using ‘init=False’)
and the field also specifies 'default_factory', then the default factory
function will always be called from the generated ‘__init__()’ function.
This happens because there is no other way to give the field an initial
value.
File: python.info, Node: Mutable default values, Next: Descriptor-typed fields, Prev: Default factory functions, Up: dataclasses — Data Classes
5.30.7.9 Mutable default values
...............................
Python stores default member variable values in class attributes.
Consider this example, not using dataclasses:
class C:
x = []
def add(self, element):
self.x.append(element)
o1 = C()
o2 = C()
o1.add(1)
o2.add(2)
assert o1.x == [1, 2]
assert o1.x is o2.x
Note that the two instances of class ‘C’ share the same class variable
‘x’, as expected.
Using dataclasses, 'if' this code was valid:
@dataclass
class D:
x: list = [] # This code raises ValueError
def add(self, element):
self.x.append(element)
it would generate code similar to:
class D:
x = []
def __init__(self, x=x):
self.x = x
def add(self, element):
self.x.append(element)
assert D().x is D().x
This has the same issue as the original example using class ‘C’. That
is, two instances of class ‘D’ that do not specify a value for ‘x’ when
creating a class instance will share the same copy of ‘x’. Because
dataclasses just use normal Python class creation they also share this
behavior. There is no general way for Data Classes to detect this
condition. Instead, the *note @dataclass: 1cb. decorator will raise a
*note ValueError: 204. if it detects an unhashable default parameter.
The assumption is that if a value is unhashable, it is mutable. This is
a partial solution, but it does protect against many common errors.
Using default factory functions is a way to create new instances of
mutable types as default values for fields:
@dataclass
class D:
x: list = field(default_factory=list)
assert D().x is not D().x
Changed in version 3.11: Instead of looking for and disallowing objects
of type *note list: 60d, *note dict: 258, or *note set: 5d5, unhashable
objects are now not allowed as default values. Unhashability is used to
approximate mutability.
File: python.info, Node: Descriptor-typed fields, Prev: Mutable default values, Up: dataclasses — Data Classes
5.30.7.10 Descriptor-typed fields
.................................
Fields that are assigned *note descriptor objects: c4f. as their default
value have the following special behaviors:
* The value for the field passed to the dataclass’s *note __init__():
6ac. method is passed to the descriptor’s *note __set__(): 1f6a.
method rather than overwriting the descriptor object.
* Similarly, when getting or setting the field, the descriptor’s
*note __get__(): 14b2. or ‘__set__()’ method is called rather than
returning or overwriting the descriptor object.
* To determine whether a field contains a default value, *note
@dataclass: 1cb. will call the descriptor’s ‘__get__()’ method
using its class access form: ‘descriptor.__get__(obj=None,
type=cls)’. If the descriptor returns a value in this case, it
will be used as the field’s default. On the other hand, if the
descriptor raises *note AttributeError: 348. in this situation, no
default value will be provided for the field.
class IntConversionDescriptor:
def __init__(self, *, default):
self._default = default
def __set_name__(self, owner, name):
self._name = "_" + name
def __get__(self, obj, type):
if obj is None:
return self._default
return getattr(obj, self._name, self._default)
def __set__(self, obj, value):
setattr(obj, self._name, int(value))
@dataclass
class InventoryItem:
quantity_on_hand: IntConversionDescriptor = IntConversionDescriptor(default=100)
i = InventoryItem()
print(i.quantity_on_hand) # 100
i.quantity_on_hand = 2.5 # calls __set__ with 2.5
print(i.quantity_on_hand) # 2
Note that if a field is annotated with a descriptor type, but is not
assigned a descriptor object as its default value, the field will act
like a normal field.
File: python.info, Node: contextlib — Utilities for with-statement contexts, Next: abc — Abstract Base Classes, Prev: dataclasses — Data Classes, Up: Python Runtime Services
5.30.8 ‘contextlib’ — Utilities for ‘with’-statement contexts
-------------------------------------------------------------
'Source code:' Lib/contextlib.py(1)
__________________________________________________________________
This module provides utilities for common tasks involving the *note
with: 5ce. statement. For more information see also *note Context
Manager Types: 1fbf. and *note With Statement Context Managers: 1fbd.
* Menu:
* Utilities::
* Examples and Recipes: Examples and Recipes<3>.
* Single use, reusable and reentrant context managers: Single use reusable and reentrant context managers.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/contextlib.py
File: python.info, Node: Utilities, Next: Examples and Recipes<3>, Up: contextlib — Utilities for with-statement contexts
5.30.8.1 Utilities
..................
Functions and classes provided:
-- Class: contextlib.AbstractContextManager
An *note abstract base class: 11a8. for classes that implement
*note object.__enter__(): 5c4. and *note object.__exit__(): 12f3.
A default implementation for *note object.__enter__(): 5c4. is
provided which returns ‘self’ while *note object.__exit__(): 12f3.
is an abstract method which by default returns ‘None’. See also
the definition of *note Context Manager Types: 1fbf.
Added in version 3.6.
-- Class: contextlib.AbstractAsyncContextManager
An *note abstract base class: 11a8. for classes that implement
*note object.__aenter__(): 1fd0. and *note object.__aexit__():
162a. A default implementation for *note object.__aenter__():
1fd0. is provided which returns ‘self’ while *note
object.__aexit__(): 162a. is an abstract method which by default
returns ‘None’. See also the definition of *note Asynchronous
Context Managers: 5f8.
Added in version 3.7.
-- Function: @contextlib.contextmanager
This function is a *note decorator: 72c. that can be used to define
a factory function for *note with: 5ce. statement context managers,
without needing to create a class or separate *note __enter__():
5c4. and *note __exit__(): 12f3. methods.
While many objects natively support use in with statements,
sometimes a resource needs to be managed that isn’t a context
manager in its own right, and doesn’t implement a ‘close()’ method
for use with ‘contextlib.closing’.
An abstract example would be the following to ensure correct
resource management:
from contextlib import contextmanager
@contextmanager
def managed_resource(*args, **kwds):
# Code to acquire resource, e.g.:
resource = acquire_resource(*args, **kwds)
try:
yield resource
finally:
# Code to release resource, e.g.:
release_resource(resource)
The function can then be used like this:
>>> with managed_resource(timeout=3600) as resource:
... # Resource is released at the end of this block,
... # even if code in the block raises an exception
The function being decorated must return a *note generator:
105a.-iterator when called. This iterator must yield exactly one
value, which will be bound to the targets in the *note with: 5ce.
statement’s ‘as’ clause, if any.
At the point where the generator yields, the block nested in the
*note with: 5ce. statement is executed. The generator is then
resumed after the block is exited. If an unhandled exception
occurs in the block, it is reraised inside the generator at the
point where the yield occurred. Thus, you can use a *note try:
6e4.…*note except: 18b.…*note finally: 9ca. statement to trap the
error (if any), or ensure that some cleanup takes place. If an
exception is trapped merely in order to log it or to perform some
action (rather than to suppress it entirely), the generator must
reraise that exception. Otherwise the generator context manager
will indicate to the ‘with’ statement that the exception has been
handled, and execution will resume with the statement immediately
following the ‘with’ statement.
*note contextmanager(): 11f4. uses *note ContextDecorator: 11f3. so
the context managers it creates can be used as decorators as well
as in *note with: 5ce. statements. When used as a decorator, a new
generator instance is implicitly created on each function call
(this allows the otherwise “one-shot” context managers created by
*note contextmanager(): 11f4. to meet the requirement that context
managers support multiple invocations in order to be used as
decorators).
Changed in version 3.2: Use of *note ContextDecorator: 11f3.
-- Function: @contextlib.asynccontextmanager
Similar to *note contextmanager(): 11f4, but creates an *note
asynchronous context manager: 5f8.
This function is a *note decorator: 72c. that can be used to define
a factory function for *note async with: 5d1. statement
asynchronous context managers, without needing to create a class or
separate *note __aenter__(): 1fd0. and *note __aexit__(): 162a.
methods. It must be applied to an *note asynchronous generator:
203e. function.
A simple example:
from contextlib import asynccontextmanager
@asynccontextmanager
async def get_connection():
conn = await acquire_db_connection()
try:
yield conn
finally:
await release_db_connection(conn)
async def get_all_users():
async with get_connection() as conn:
return conn.query('SELECT ...')
Added in version 3.7.
Context managers defined with *note asynccontextmanager(): b2a. can
be used either as decorators or with *note async with: 5d1.
statements:
import time
from contextlib import asynccontextmanager
@asynccontextmanager
async def timeit():
now = time.monotonic()
try:
yield
finally:
print(f'it took {time.monotonic() - now}s to run')
@timeit()
async def main():
# ... async code ...
When used as a decorator, a new generator instance is implicitly
created on each function call. This allows the otherwise
“one-shot” context managers created by *note asynccontextmanager():
b2a. to meet the requirement that context managers support multiple
invocations in order to be used as decorators.
Changed in version 3.10: Async context managers created with *note
asynccontextmanager(): b2a. can be used as decorators.
-- Function: contextlib.closing (thing)
Return a context manager that closes 'thing' upon completion of the
block. This is basically equivalent to:
from contextlib import contextmanager
@contextmanager
def closing(thing):
try:
yield thing
finally:
thing.close()
And lets you write code like this:
from contextlib import closing
from urllib.request import urlopen
with closing(urlopen('https://www.python.org')) as page:
for line in page:
print(line)
without needing to explicitly close ‘page’. Even if an error
occurs, ‘page.close()’ will be called when the *note with: 5ce.
block is exited.
Note: Most types managing resources support the *note context
manager: 5d0. protocol, which closes 'thing' on leaving the
*note with: 5ce. statement. As such, ‘closing()’ is most
useful for third party types that don’t support context
managers. This example is purely for illustration purposes,
as *note urlopen(): 295. would normally be used in a context
manager.
-- Function: contextlib.aclosing (thing)
Return an async context manager that calls the ‘aclose()’ method of
'thing' upon completion of the block. This is basically equivalent
to:
from contextlib import asynccontextmanager
@asynccontextmanager
async def aclosing(thing):
try:
yield thing
finally:
await thing.aclose()
Significantly, ‘aclosing()’ supports deterministic cleanup of async
generators when they happen to exit early by *note break: aa9. or
an exception. For example:
from contextlib import aclosing
async with aclosing(my_generator()) as values:
async for value in values:
if value == 42:
break
This pattern ensures that the generator’s async exit code is
executed in the same context as its iterations (so that exceptions
and context variables work as expected, and the exit code isn’t run
after the lifetime of some task it depends on).
Added in version 3.10.
-- Function: contextlib.nullcontext (enter_result=None)
Return a context manager that returns 'enter_result' from
‘__enter__’, but otherwise does nothing. It is intended to be used
as a stand-in for an optional context manager, for example:
def myfunction(arg, ignore_exceptions=False):
if ignore_exceptions:
# Use suppress to ignore all exceptions.
cm = contextlib.suppress(Exception)
else:
# Do not ignore any exceptions, cm has no effect.
cm = contextlib.nullcontext()
with cm:
# Do something
An example using 'enter_result':
def process_file(file_or_path):
if isinstance(file_or_path, str):
# If string, open file
cm = open(file_or_path)
else:
# Caller is responsible for closing file
cm = nullcontext(file_or_path)
with cm as file:
# Perform processing on the file
It can also be used as a stand-in for *note asynchronous context
managers: 5f8.:
async def send_http(session=None):
if not session:
# If no http session, create it with aiohttp
cm = aiohttp.ClientSession()
else:
# Caller is responsible for closing the session
cm = nullcontext(session)
async with cm as session:
# Send http requests with session
Added in version 3.7.
Changed in version 3.10: *note asynchronous context manager: 5d3.
support was added.
-- Function: contextlib.suppress (*exceptions)
Return a context manager that suppresses any of the specified
exceptions if they occur in the body of a ‘with’ statement and then
resumes execution with the first statement following the end of the
‘with’ statement.
As with any other mechanism that completely suppresses exceptions,
this context manager should be used only to cover very specific
errors where silently continuing with program execution is known to
be the right thing to do.
For example:
from contextlib import suppress
with suppress(FileNotFoundError):
os.remove('somefile.tmp')
with suppress(FileNotFoundError):
os.remove('someotherfile.tmp')
This code is equivalent to:
try:
os.remove('somefile.tmp')
except FileNotFoundError:
pass
try:
os.remove('someotherfile.tmp')
except FileNotFoundError:
pass
This context manager is *note reentrant: 442b.
If the code within the ‘with’ block raises a *note
BaseExceptionGroup: 261, suppressed exceptions are removed from the
group. Any exceptions of the group which are not suppressed are
re-raised in a new group which is created using the original
group’s *note derive(): 16c0. method.
Added in version 3.4.
Changed in version 3.12: ‘suppress’ now supports suppressing
exceptions raised as part of a *note BaseExceptionGroup: 261.
-- Function: contextlib.redirect_stdout (new_target)
Context manager for temporarily redirecting *note sys.stdout: ad6.
to another file or file-like object.
This tool adds flexibility to existing functions or classes whose
output is hardwired to stdout.
For example, the output of *note help(): 8d6. normally is sent to
'sys.stdout'. You can capture that output in a string by
redirecting the output to an *note io.StringIO: f22. object. The
replacement stream is returned from the ‘__enter__’ method and so
is available as the target of the *note with: 5ce. statement:
with redirect_stdout(io.StringIO()) as f:
help(pow)
s = f.getvalue()
To send the output of *note help(): 8d6. to a file on disk,
redirect the output to a regular file:
with open('help.txt', 'w') as f:
with redirect_stdout(f):
help(pow)
To send the output of *note help(): 8d6. to 'sys.stderr':
with redirect_stdout(sys.stderr):
help(pow)
Note that the global side effect on *note sys.stdout: ad6. means
that this context manager is not suitable for use in library code
and most threaded applications. It also has no effect on the
output of subprocesses. However, it is still a useful approach for
many utility scripts.
This context manager is *note reentrant: 442b.
Added in version 3.4.
-- Function: contextlib.redirect_stderr (new_target)
Similar to *note redirect_stdout(): de7. but redirecting *note
sys.stderr: 939. to another file or file-like object.
This context manager is *note reentrant: 442b.
Added in version 3.5.
-- Function: contextlib.chdir (path)
Non parallel-safe context manager to change the current working
directory. As this changes a global state, the working directory,
it is not suitable for use in most threaded or async contexts. It
is also not suitable for most non-linear code execution, like
generators, where the program execution is temporarily relinquished
– unless explicitly desired, you should not yield when this context
manager is active.
This is a simple wrapper around *note chdir(): 609, it changes the
current working directory upon entering and restores the old one on
exit.
This context manager is *note reentrant: 442b.
Added in version 3.11.
-- Class: contextlib.ContextDecorator
A base class that enables a context manager to also be used as a
decorator.
Context managers inheriting from ‘ContextDecorator’ have to
implement ‘__enter__’ and ‘__exit__’ as normal. ‘__exit__’ retains
its optional exception handling even when used as a decorator.
‘ContextDecorator’ is used by *note contextmanager(): 11f4, so you
get this functionality automatically.
Example of ‘ContextDecorator’:
from contextlib import ContextDecorator
class mycontext(ContextDecorator):
def __enter__(self):
print('Starting')
return self
def __exit__(self, *exc):
print('Finishing')
return False
The class can then be used like this:
>>> @mycontext()
... def function():
... print('The bit in the middle')
...
>>> function()
Starting
The bit in the middle
Finishing
>>> with mycontext():
... print('The bit in the middle')
...
Starting
The bit in the middle
Finishing
This change is just syntactic sugar for any construct of the
following form:
def f():
with cm():
# Do stuff
‘ContextDecorator’ lets you instead write:
@cm()
def f():
# Do stuff
It makes it clear that the ‘cm’ applies to the whole function,
rather than just a piece of it (and saving an indentation level is
nice, too).
Existing context managers that already have a base class can be
extended by using ‘ContextDecorator’ as a mixin class:
from contextlib import ContextDecorator
class mycontext(ContextBaseClass, ContextDecorator):
def __enter__(self):
return self
def __exit__(self, *exc):
return False
Note: As the decorated function must be able to be called
multiple times, the underlying context manager must support
use in multiple *note with: 5ce. statements. If this is not
the case, then the original construct with the explicit ‘with’
statement inside the function should be used.
Added in version 3.2.
-- Class: contextlib.AsyncContextDecorator
Similar to *note ContextDecorator: 11f3. but only for asynchronous
functions.
Example of ‘AsyncContextDecorator’:
from asyncio import run
from contextlib import AsyncContextDecorator
class mycontext(AsyncContextDecorator):
async def __aenter__(self):
print('Starting')
return self
async def __aexit__(self, *exc):
print('Finishing')
return False
The class can then be used like this:
>>> @mycontext()
... async def function():
... print('The bit in the middle')
...
>>> run(function())
Starting
The bit in the middle
Finishing
>>> async def function():
... async with mycontext():
... print('The bit in the middle')
...
>>> run(function())
Starting
The bit in the middle
Finishing
Added in version 3.10.
-- Class: contextlib.ExitStack
A context manager that is designed to make it easy to
programmatically combine other context managers and cleanup
functions, especially those that are optional or otherwise driven
by input data.
For example, a set of files may easily be handled in a single with
statement as follows:
with ExitStack() as stack:
files = [stack.enter_context(open(fname)) for fname in filenames]
# All opened files will automatically be closed at the end of
# the with statement, even if attempts to open files later
# in the list raise an exception
The *note __enter__(): 5c4. method returns the *note ExitStack:
b29. instance, and performs no additional operations.
Each instance maintains a stack of registered callbacks that are
called in reverse order when the instance is closed (either
explicitly or implicitly at the end of a *note with: 5ce.
statement). Note that callbacks are 'not' invoked implicitly when
the context stack instance is garbage collected.
This stack model is used so that context managers that acquire
their resources in their ‘__init__’ method (such as file objects)
can be handled correctly.
Since registered callbacks are invoked in the reverse order of
registration, this ends up behaving as if multiple nested *note
with: 5ce. statements had been used with the registered set of
callbacks. This even extends to exception handling - if an inner
callback suppresses or replaces an exception, then outer callbacks
will be passed arguments based on that updated state.
This is a relatively low level API that takes care of the details
of correctly unwinding the stack of exit callbacks. It provides a
suitable foundation for higher level context managers that
manipulate the exit stack in application specific ways.
Added in version 3.3.
-- Method: enter_context (cm)
Enters a new context manager and adds its *note __exit__():
12f3. method to the callback stack. The return value is the
result of the context manager’s own *note __enter__(): 5c4.
method.
These context managers may suppress exceptions just as they
normally would if used directly as part of a *note with: 5ce.
statement.
Changed in version 3.11: Raises *note TypeError: 534. instead
of *note AttributeError: 348. if 'cm' is not a context
manager.
-- Method: push (exit)
Adds a context manager’s *note __exit__(): 12f3. method to the
callback stack.
As ‘__enter__’ is 'not' invoked, this method can be used to
cover part of an *note __enter__(): 5c4. implementation with a
context manager’s own *note __exit__(): 12f3. method.
If passed an object that is not a context manager, this method
assumes it is a callback with the same signature as a context
manager’s *note __exit__(): 12f3. method and adds it directly
to the callback stack.
By returning true values, these callbacks can suppress
exceptions the same way context manager *note __exit__():
12f3. methods can.
The passed in object is returned from the function, allowing
this method to be used as a function decorator.
-- Method: callback (callback, /, *args, **kwds)
Accepts an arbitrary callback function and arguments and adds
it to the callback stack.
Unlike the other methods, callbacks added this way cannot
suppress exceptions (as they are never passed the exception
details).
The passed in callback is returned from the function, allowing
this method to be used as a function decorator.
-- Method: pop_all ()
Transfers the callback stack to a fresh *note ExitStack: b29.
instance and returns it. No callbacks are invoked by this
operation - instead, they will now be invoked when the new
stack is closed (either explicitly or implicitly at the end of
a *note with: 5ce. statement).
For example, a group of files can be opened as an “all or
nothing” operation as follows:
with ExitStack() as stack:
files = [stack.enter_context(open(fname)) for fname in filenames]
# Hold onto the close method, but don't call it yet.
close_files = stack.pop_all().close
# If opening any file fails, all previously opened files will be
# closed automatically. If all files are opened successfully,
# they will remain open even after the with statement ends.
# close_files() can then be invoked explicitly to close them all.
-- Method: close ()
Immediately unwinds the callback stack, invoking callbacks in
the reverse order of registration. For any context managers
and exit callbacks registered, the arguments passed in will
indicate that no exception occurred.
-- Class: contextlib.AsyncExitStack
An *note asynchronous context manager: 5f8, similar to *note
ExitStack: b29, that supports combining both synchronous and
asynchronous context managers, as well as having coroutines for
cleanup logic.
The *note close(): 442e. method is not implemented; *note aclose():
442f. must be used instead.
-- Method: async enter_async_context (cm)
Similar to *note ExitStack.enter_context(): 5cf. but expects
an asynchronous context manager.
Changed in version 3.11: Raises *note TypeError: 534. instead
of *note AttributeError: 348. if 'cm' is not an asynchronous
context manager.
-- Method: push_async_exit (exit)
Similar to *note ExitStack.push(): 442c. but expects either an
asynchronous context manager or a coroutine function.
-- Method: push_async_callback (callback, /, *args, **kwds)
Similar to *note ExitStack.callback(): a83. but expects a
coroutine function.
-- Method: async aclose ()
Similar to *note ExitStack.close(): 442e. but properly handles
awaitables.
Continuing the example for *note asynccontextmanager(): b2a.:
async with AsyncExitStack() as stack:
connections = [await stack.enter_async_context(get_connection())
for i in range(5)]
# All opened connections will automatically be released at the end of
# the async with statement, even if attempts to open a connection
# later in the list raise an exception.
Added in version 3.7.
File: python.info, Node: Examples and Recipes<3>, Next: Single use reusable and reentrant context managers, Prev: Utilities, Up: contextlib — Utilities for with-statement contexts
5.30.8.2 Examples and Recipes
.............................
This section describes some examples and recipes for making effective
use of the tools provided by *note contextlib: 23.
* Menu:
* Supporting a variable number of context managers::
* Catching exceptions from __enter__ methods::
* Cleaning up in an __enter__ implementation::
* Replacing any use of try-finally and flag variables::
* Using a context manager as a function decorator::
File: python.info, Node: Supporting a variable number of context managers, Next: Catching exceptions from __enter__ methods, Up: Examples and Recipes<3>
5.30.8.3 Supporting a variable number of context managers
.........................................................
The primary use case for *note ExitStack: b29. is the one given in the
class documentation: supporting a variable number of context managers
and other cleanup operations in a single *note with: 5ce. statement.
The variability may come from the number of context managers needed
being driven by user input (such as opening a user specified collection
of files), or from some of the context managers being optional:
with ExitStack() as stack:
for resource in resources:
stack.enter_context(resource)
if need_special_resource():
special = acquire_special_resource()
stack.callback(release_special_resource, special)
# Perform operations that use the acquired resources
As shown, *note ExitStack: b29. also makes it quite easy to use *note
with: 5ce. statements to manage arbitrary resources that don’t natively
support the context management protocol.
File: python.info, Node: Catching exceptions from __enter__ methods, Next: Cleaning up in an __enter__ implementation, Prev: Supporting a variable number of context managers, Up: Examples and Recipes<3>
5.30.8.4 Catching exceptions from ‘__enter__’ methods
.....................................................
It is occasionally desirable to catch exceptions from an ‘__enter__’
method implementation, 'without' inadvertently catching exceptions from
the *note with: 5ce. statement body or the context manager’s ‘__exit__’
method. By using *note ExitStack: b29. the steps in the context
management protocol can be separated slightly in order to allow this:
stack = ExitStack()
try:
x = stack.enter_context(cm)
except Exception:
# handle __enter__ exception
else:
with stack:
# Handle normal case
Actually needing to do this is likely to indicate that the underlying
API should be providing a direct resource management interface for use
with *note try: 6e4./*note except: 18b./*note finally: 9ca. statements,
but not all APIs are well designed in that regard. When a context
manager is the only resource management API provided, then *note
ExitStack: b29. can make it easier to handle various situations that
can’t be handled directly in a *note with: 5ce. statement.
File: python.info, Node: Cleaning up in an __enter__ implementation, Next: Replacing any use of try-finally and flag variables, Prev: Catching exceptions from __enter__ methods, Up: Examples and Recipes<3>
5.30.8.5 Cleaning up in an ‘__enter__’ implementation
.....................................................
As noted in the documentation of *note ExitStack.push(): 442c, this
method can be useful in cleaning up an already allocated resource if
later steps in the *note __enter__(): 5c4. implementation fail.
Here’s an example of doing this for a context manager that accepts
resource acquisition and release functions, along with an optional
validation function, and maps them to the context management protocol:
from contextlib import contextmanager, AbstractContextManager, ExitStack
class ResourceManager(AbstractContextManager):
def __init__(self, acquire_resource, release_resource, check_resource_ok=None):
self.acquire_resource = acquire_resource
self.release_resource = release_resource
if check_resource_ok is None:
def check_resource_ok(resource):
return True
self.check_resource_ok = check_resource_ok
@contextmanager
def _cleanup_on_error(self):
with ExitStack() as stack:
stack.push(self)
yield
# The validation check passed and didn't raise an exception
# Accordingly, we want to keep the resource, and pass it
# back to our caller
stack.pop_all()
def __enter__(self):
resource = self.acquire_resource()
with self._cleanup_on_error():
if not self.check_resource_ok(resource):
msg = "Failed validation for {!r}"
raise RuntimeError(msg.format(resource))
return resource
def __exit__(self, *exc_details):
# We don't need to duplicate any of our resource release logic
self.release_resource()
File: python.info, Node: Replacing any use of try-finally and flag variables, Next: Using a context manager as a function decorator, Prev: Cleaning up in an __enter__ implementation, Up: Examples and Recipes<3>
5.30.8.6 Replacing any use of ‘try-finally’ and flag variables
..............................................................
A pattern you will sometimes see is a ‘try-finally’ statement with a
flag variable to indicate whether or not the body of the ‘finally’
clause should be executed. In its simplest form (that can’t already be
handled just by using an ‘except’ clause instead), it looks something
like this:
cleanup_needed = True
try:
result = perform_operation()
if result:
cleanup_needed = False
finally:
if cleanup_needed:
cleanup_resources()
As with any ‘try’ statement based code, this can cause problems for
development and review, because the setup code and the cleanup code can
end up being separated by arbitrarily long sections of code.
*note ExitStack: b29. makes it possible to instead register a callback
for execution at the end of a ‘with’ statement, and then later decide to
skip executing that callback:
from contextlib import ExitStack
with ExitStack() as stack:
stack.callback(cleanup_resources)
result = perform_operation()
if result:
stack.pop_all()
This allows the intended cleanup behaviour to be made explicit up front,
rather than requiring a separate flag variable.
If a particular application uses this pattern a lot, it can be
simplified even further by means of a small helper class:
from contextlib import ExitStack
class Callback(ExitStack):
def __init__(self, callback, /, *args, **kwds):
super().__init__()
self.callback(callback, *args, **kwds)
def cancel(self):
self.pop_all()
with Callback(cleanup_resources) as cb:
result = perform_operation()
if result:
cb.cancel()
If the resource cleanup isn’t already neatly bundled into a standalone
function, then it is still possible to use the decorator form of *note
ExitStack.callback(): a83. to declare the resource cleanup in advance:
from contextlib import ExitStack
with ExitStack() as stack:
@stack.callback
def cleanup_resources():
...
result = perform_operation()
if result:
stack.pop_all()
Due to the way the decorator protocol works, a callback function
declared this way cannot take any parameters. Instead, any resources to
be released must be accessed as closure variables.
File: python.info, Node: Using a context manager as a function decorator, Prev: Replacing any use of try-finally and flag variables, Up: Examples and Recipes<3>
5.30.8.7 Using a context manager as a function decorator
........................................................
*note ContextDecorator: 11f3. makes it possible to use a context manager
in both an ordinary ‘with’ statement and also as a function decorator.
For example, it is sometimes useful to wrap functions or groups of
statements with a logger that can track the time of entry and time of
exit. Rather than writing both a function decorator and a context
manager for the task, inheriting from *note ContextDecorator: 11f3.
provides both capabilities in a single definition:
from contextlib import ContextDecorator
import logging
logging.basicConfig(level=logging.INFO)
class track_entry_and_exit(ContextDecorator):
def __init__(self, name):
self.name = name
def __enter__(self):
logging.info('Entering: %s', self.name)
def __exit__(self, exc_type, exc, exc_tb):
logging.info('Exiting: %s', self.name)
Instances of this class can be used as both a context manager:
with track_entry_and_exit('widget loader'):
print('Some time consuming activity goes here')
load_widget()
And also as a function decorator:
@track_entry_and_exit('widget loader')
def activity():
print('Some time consuming activity goes here')
load_widget()
Note that there is one additional limitation when using context managers
as function decorators: there’s no way to access the return value of
*note __enter__(): 5c4. If that value is needed, then it is still
necessary to use an explicit ‘with’ statement.
See also
........
PEP 343(1) - The “with” statement
The specification, background, and examples for the Python *note
with: 5ce. statement.
---------- Footnotes ----------
(1) https://peps.python.org/pep-0343/
File: python.info, Node: Single use reusable and reentrant context managers, Prev: Examples and Recipes<3>, Up: contextlib — Utilities for with-statement contexts
5.30.8.8 Single use, reusable and reentrant context managers
............................................................
Most context managers are written in a way that means they can only be
used effectively in a *note with: 5ce. statement once. These single use
context managers must be created afresh each time they’re used -
attempting to use them a second time will trigger an exception or
otherwise not work correctly.
This common limitation means that it is generally advisable to create
context managers directly in the header of the *note with: 5ce.
statement where they are used (as shown in all of the usage examples
above).
Files are an example of effectively single use context managers, since
the first *note with: 5ce. statement will close the file, preventing any
further IO operations using that file object.
Context managers created using *note contextmanager(): 11f4. are also
single use context managers, and will complain about the underlying
generator failing to yield if an attempt is made to use them a second
time:
>>> from contextlib import contextmanager
>>> @contextmanager
... def singleuse():
... print("Before")
... yield
... print("After")
...
>>> cm = singleuse()
>>> with cm:
... pass
...
Before
After
>>> with cm:
... pass
...
Traceback (most recent call last):
...
RuntimeError: generator didn't yield
* Menu:
* Reentrant context managers::
* Reusable context managers::
File: python.info, Node: Reentrant context managers, Next: Reusable context managers, Up: Single use reusable and reentrant context managers
5.30.8.9 Reentrant context managers
...................................
More sophisticated context managers may be “reentrant”. These context
managers can not only be used in multiple *note with: 5ce. statements,
but may also be used 'inside' a ‘with’ statement that is already using
the same context manager.
*note threading.RLock: 2e2. is an example of a reentrant context
manager, as are *note suppress(): f21, *note redirect_stdout(): de7, and
*note chdir(): 608. Here’s a very simple example of reentrant use:
>>> from contextlib import redirect_stdout
>>> from io import StringIO
>>> stream = StringIO()
>>> write_to_stream = redirect_stdout(stream)
>>> with write_to_stream:
... print("This is written to the stream rather than stdout")
... with write_to_stream:
... print("This is also written to the stream")
...
>>> print("This is written directly to stdout")
This is written directly to stdout
>>> print(stream.getvalue())
This is written to the stream rather than stdout
This is also written to the stream
Real world examples of reentrancy are more likely to involve multiple
functions calling each other and hence be far more complicated than this
example.
Note also that being reentrant is 'not' the same thing as being thread
safe. *note redirect_stdout(): de7, for example, is definitely not
thread safe, as it makes a global modification to the system state by
binding *note sys.stdout: ad6. to a different stream.
File: python.info, Node: Reusable context managers, Prev: Reentrant context managers, Up: Single use reusable and reentrant context managers
5.30.8.10 Reusable context managers
...................................
Distinct from both single use and reentrant context managers are
“reusable” context managers (or, to be completely explicit, “reusable,
but not reentrant” context managers, since reentrant context managers
are also reusable). These context managers support being used multiple
times, but will fail (or otherwise not work correctly) if the specific
context manager instance has already been used in a containing with
statement.
*note threading.Lock: 1677. is an example of a reusable, but not
reentrant, context manager (for a reentrant lock, it is necessary to use
*note threading.RLock: 2e2. instead).
Another example of a reusable, but not reentrant, context manager is
*note ExitStack: b29, as it invokes 'all' currently registered callbacks
when leaving any with statement, regardless of where those callbacks
were added:
>>> from contextlib import ExitStack
>>> stack = ExitStack()
>>> with stack:
... stack.callback(print, "Callback: from first context")
... print("Leaving first context")
...
Leaving first context
Callback: from first context
>>> with stack:
... stack.callback(print, "Callback: from second context")
... print("Leaving second context")
...
Leaving second context
Callback: from second context
>>> with stack:
... stack.callback(print, "Callback: from outer context")
... with stack:
... stack.callback(print, "Callback: from inner context")
... print("Leaving inner context")
... print("Leaving outer context")
...
Leaving inner context
Callback: from inner context
Callback: from outer context
Leaving outer context
As the output from the example shows, reusing a single stack object
across multiple with statements works correctly, but attempting to nest
them will cause the stack to be cleared at the end of the innermost with
statement, which is unlikely to be desirable behaviour.
Using separate *note ExitStack: b29. instances instead of reusing a
single instance avoids that problem:
>>> from contextlib import ExitStack
>>> with ExitStack() as outer_stack:
... outer_stack.callback(print, "Callback: from outer context")
... with ExitStack() as inner_stack:
... inner_stack.callback(print, "Callback: from inner context")
... print("Leaving inner context")
... print("Leaving outer context")
...
Leaving inner context
Callback: from inner context
Leaving outer context
Callback: from outer context
File: python.info, Node: abc — Abstract Base Classes, Next: atexit — Exit handlers, Prev: contextlib — Utilities for with-statement contexts, Up: Python Runtime Services
5.30.9 ‘abc’ — Abstract Base Classes
------------------------------------
'Source code:' Lib/abc.py(1)
__________________________________________________________________
This module provides the infrastructure for defining *note abstract base
classes: 11a8. (ABCs) in Python, as outlined in PEP 3119(2); see the PEP
for why this was added to Python. (See also PEP 3141(3) and the *note
numbers: 9e. module regarding a type hierarchy for numbers based on
ABCs.)
The *note collections: 1d. module has some concrete classes that derive
from ABCs; these can, of course, be further derived. In addition, the
*note collections.abc: 1e. submodule has some ABCs that can be used to
test whether a class or instance provides a particular interface, for
example, if it is *note hashable: 60c. or if it is a *note mapping:
11ae.
This module provides the metaclass *note ABCMeta: f13. for defining ABCs
and a helper class *note ABC: f12. to alternatively define ABCs through
inheritance:
-- Class: abc.ABC
A helper class that has *note ABCMeta: f13. as its metaclass. With
this class, an abstract base class can be created by simply
deriving from ‘ABC’ avoiding sometimes confusing metaclass usage,
for example:
from abc import ABC
class MyABC(ABC):
pass
Note that the type of ‘ABC’ is still *note ABCMeta: f13, therefore
inheriting from ‘ABC’ requires the usual precautions regarding
metaclass usage, as multiple inheritance may lead to metaclass
conflicts. One may also define an abstract base class by passing
the metaclass keyword and using ‘ABCMeta’ directly, for example:
from abc import ABCMeta
class MyABC(metaclass=ABCMeta):
pass
Added in version 3.4.
-- Class: abc.ABCMeta
Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed
directly, and then acts as a mix-in class. You can also register
unrelated concrete classes (even built-in classes) and unrelated
ABCs as “virtual subclasses” – these and their descendants will be
considered subclasses of the registering ABC by the built-in *note
issubclass(): 7bc. function, but the registering ABC won’t show up
in their MRO (Method Resolution Order) nor will method
implementations defined by the registering ABC be callable (not
even via *note super(): 4d7.). (4)
Classes created with a metaclass of ‘ABCMeta’ have the following
method:
-- Method: register (subclass)
Register 'subclass' as a “virtual subclass” of this ABC. For
example:
from abc import ABC
class MyABC(ABC):
pass
MyABC.register(tuple)
assert issubclass(tuple, MyABC)
assert isinstance((), MyABC)
Changed in version 3.3: Returns the registered subclass, to
allow usage as a class decorator.
Changed in version 3.4: To detect calls to ‘register()’, you
can use the *note get_cache_token(): f11. function.
You can also override this method in an abstract base class:
-- Method: __subclasshook__ (subclass)
(Must be defined as a class method.)
Check whether 'subclass' is considered a subclass of this ABC.
This means that you can customize the behavior of *note
issubclass(): 7bc. further without the need to call *note
register(): 1082. on every class you want to consider a
subclass of the ABC. (This class method is called from the
*note __subclasscheck__(): 1f7d. method of the ABC.)
This method should return ‘True’, ‘False’ or *note
NotImplemented: 7cd. If it returns ‘True’, the 'subclass' is
considered a subclass of this ABC. If it returns ‘False’, the
'subclass' is not considered a subclass of this ABC, even if
it would normally be one. If it returns ‘NotImplemented’, the
subclass check is continued with the usual mechanism.
For a demonstration of these concepts, look at this example ABC
definition:
class Foo:
def __getitem__(self, index):
...
def __len__(self):
...
def get_iterator(self):
return iter(self)
class MyIterable(ABC):
@abstractmethod
def __iter__(self):
while False:
yield None
def get_iterator(self):
return self.__iter__()
@classmethod
def __subclasshook__(cls, C):
if cls is MyIterable:
if any("__iter__" in B.__dict__ for B in C.__mro__):
return True
return NotImplemented
MyIterable.register(Foo)
The ABC ‘MyIterable’ defines the standard iterable method, *note
__iter__(): 1f5c, as an abstract method. The implementation given
here can still be called from subclasses. The ‘get_iterator()’
method is also part of the ‘MyIterable’ abstract base class, but it
does not have to be overridden in non-abstract derived classes.
The *note __subclasshook__(): 2531. class method defined here says
that any class that has an *note __iter__(): 1f5c. method in its
*note __dict__: 558. (or in that of one of its base classes,
accessed via the *note __mro__: 1f29. list) is considered a
‘MyIterable’ too.
Finally, the last line makes ‘Foo’ a virtual subclass of
‘MyIterable’, even though it does not define an *note __iter__():
1f5c. method (it uses the old-style iterable protocol, defined in
terms of *note __len__(): 1f63. and *note __getitem__(): 285.).
Note that this will not make ‘get_iterator’ available as a method
of ‘Foo’, so it is provided separately.
The ‘abc’ module also provides the following decorator:
-- Function: @abc.abstractmethod
A decorator indicating abstract methods.
Using this decorator requires that the class’s metaclass is *note
ABCMeta: f13. or is derived from it. A class that has a metaclass
derived from ‘ABCMeta’ cannot be instantiated unless all of its
abstract methods and properties are overridden. The abstract
methods can be called using any of the normal ‘super’ call
mechanisms. ‘abstractmethod()’ may be used to declare abstract
methods for properties and descriptors.
Dynamically adding abstract methods to a class, or attempting to
modify the abstraction status of a method or class once it is
created, are only supported using the *note
update_abstractmethods(): 443d. function. The ‘abstractmethod()’
only affects subclasses derived using regular inheritance; “virtual
subclasses” registered with the ABC’s *note register(): 1082.
method are not affected.
When ‘abstractmethod()’ is applied in combination with other method
descriptors, it should be applied as the innermost decorator, as
shown in the following usage examples:
class C(ABC):
@abstractmethod
def my_abstract_method(self, arg1):
...
@classmethod
@abstractmethod
def my_abstract_classmethod(cls, arg2):
...
@staticmethod
@abstractmethod
def my_abstract_staticmethod(arg3):
...
@property
@abstractmethod
def my_abstract_property(self):
...
@my_abstract_property.setter
@abstractmethod
def my_abstract_property(self, val):
...
@abstractmethod
def _get_x(self):
...
@abstractmethod
def _set_x(self, val):
...
x = property(_get_x, _set_x)
In order to correctly interoperate with the abstract base class
machinery, the descriptor must identify itself as abstract using
‘__isabstractmethod__’. In general, this attribute should be
‘True’ if any of the methods used to compose the descriptor are
abstract. For example, Python’s built-in *note property: 194. does
the equivalent of:
class Descriptor:
...
@property
def __isabstractmethod__(self):
return any(getattr(f, '__isabstractmethod__', False) for
f in (self._fget, self._fset, self._fdel))
Note: Unlike Java abstract methods, these abstract methods may
have an implementation. This implementation can be called via
the *note super(): 4d7. mechanism from the class that
overrides it. This could be useful as an end-point for a
super-call in a framework that uses cooperative
multiple-inheritance.
The ‘abc’ module also supports the following legacy decorators:
-- Function: @abc.abstractclassmethod
Added in version 3.2.
Deprecated since version 3.3: It is now possible to use *note
classmethod: 166. with *note abstractmethod(): 107f, making this
decorator redundant.
A subclass of the built-in *note classmethod(): 166, indicating an
abstract classmethod. Otherwise it is similar to *note
abstractmethod(): 107f.
This special case is deprecated, as the *note classmethod(): 166.
decorator is now correctly identified as abstract when applied to
an abstract method:
class C(ABC):
@classmethod
@abstractmethod
def my_abstract_classmethod(cls, arg):
...
-- Function: @abc.abstractstaticmethod
Added in version 3.2.
Deprecated since version 3.3: It is now possible to use *note
staticmethod: 412. with *note abstractmethod(): 107f, making this
decorator redundant.
A subclass of the built-in *note staticmethod(): 412, indicating an
abstract staticmethod. Otherwise it is similar to *note
abstractmethod(): 107f.
This special case is deprecated, as the *note staticmethod(): 412.
decorator is now correctly identified as abstract when applied to
an abstract method:
class C(ABC):
@staticmethod
@abstractmethod
def my_abstract_staticmethod(arg):
...
-- Function: @abc.abstractproperty
Deprecated since version 3.3: It is now possible to use *note
property: 194, *note property.getter(): 165a, *note
property.setter(): 165b. and *note property.deleter(): 165c. with
*note abstractmethod(): 107f, making this decorator redundant.
A subclass of the built-in *note property(): 194, indicating an
abstract property.
This special case is deprecated, as the *note property(): 194.
decorator is now correctly identified as abstract when applied to
an abstract method:
class C(ABC):
@property
@abstractmethod
def my_abstract_property(self):
...
The above example defines a read-only property; you can also define
a read-write abstract property by appropriately marking one or more
of the underlying methods as abstract:
class C(ABC):
@property
def x(self):
...
@x.setter
@abstractmethod
def x(self, val):
...
If only some components are abstract, only those components need to
be updated to create a concrete property in a subclass:
class D(C):
@C.x.setter
def x(self, val):
...
The ‘abc’ module also provides the following functions:
-- Function: abc.get_cache_token ()
Returns the current abstract base class cache token.
The token is an opaque object (that supports equality testing)
identifying the current version of the abstract base class cache
for virtual subclasses. The token changes with every call to *note
ABCMeta.register(): 1082. on any ABC.
Added in version 3.4.
-- Function: abc.update_abstractmethods (cls)
A function to recalculate an abstract class’s abstraction status.
This function should be called if a class’s abstract methods have
been implemented or changed after it was created. Usually, this
function should be called from within a class decorator.
Returns 'cls', to allow usage as a class decorator.
If 'cls' is not an instance of *note ABCMeta: f13, does nothing.
Note: This function assumes that 'cls'’s superclasses are
already updated. It does not update any subclasses.
Added in version 3.10.
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/abc.py
(2) https://peps.python.org/pep-3119/
(3) https://peps.python.org/pep-3141/
(4) C++ programmers should note that Python’s virtual base class
concept is not the same as C++’s.
File: python.info, Node: atexit — Exit handlers, Next: traceback — Print or retrieve a stack traceback, Prev: abc — Abstract Base Classes, Up: Python Runtime Services
5.30.10 ‘atexit’ — Exit handlers
--------------------------------
__________________________________________________________________
The *note atexit: c. module defines functions to register and unregister
cleanup functions. Functions thus registered are automatically executed
upon normal interpreter termination. *note atexit: c. runs these
functions in the 'reverse' order in which they were registered; if you
register ‘A’, ‘B’, and ‘C’, at interpreter termination time they will be
run in the order ‘C’, ‘B’, ‘A’.
'Note:' The functions registered via this module are not called when the
program is killed by a signal not handled by Python, when a Python fatal
internal error is detected, or when *note os._exit(): 2298. is called.
'Note:' The effect of registering or unregistering functions from within
a cleanup function is undefined.
Changed in version 3.7: When used with C-API subinterpreters, registered
functions are local to the interpreter they were registered in.
-- Function: atexit.register (func, *args, **kwargs)
Register 'func' as a function to be executed at termination. Any
optional arguments that are to be passed to 'func' must be passed
as arguments to *note register(): 87b. It is possible to register
the same function and arguments more than once.
At normal program termination (for instance, if *note sys.exit():
133c. is called or the main module’s execution completes), all
functions registered are called in last in, first out order. The
assumption is that lower level modules will normally be imported
before higher level modules and thus must be cleaned up later.
If an exception is raised during execution of the exit handlers, a
traceback is printed (unless *note SystemExit: d40. is raised) and
the exception information is saved. After all exit handlers have
had a chance to run, the last exception to be raised is re-raised.
This function returns 'func', which makes it possible to use it as
a decorator.
Warning: Starting new threads or calling *note os.fork(): 197.
from a registered function can lead to race condition between
the main Python runtime thread freeing thread states while
internal *note threading: ed. routines or the new process try
to use that state. This can lead to crashes rather than clean
shutdown.
Changed in version 3.12: Attempts to start a new thread or *note
os.fork(): 197. a new process in a registered function now leads to
*note RuntimeError: 195.
-- Function: atexit.unregister (func)
Remove 'func' from the list of functions to be run at interpreter
shutdown. *note unregister(): 4440. silently does nothing if
'func' was not previously registered. If 'func' has been
registered more than once, every occurrence of that function in the
*note atexit: c. call stack will be removed. Equality comparisons
(‘==’) are used internally during unregistration, so function
references do not need to have matching identities.
See also
........
Module *note readline: ba.
Useful example of *note atexit: c. to read and write *note
readline: ba. history files.
* Menu:
* atexit Example::
File: python.info, Node: atexit Example, Up: atexit — Exit handlers
5.30.10.1 ‘atexit’ Example
..........................
The following simple example demonstrates how a module can initialize a
counter from a file when it is imported and save the counter’s updated
value automatically when the program terminates without relying on the
application making an explicit call into this module at termination.
try:
with open('counterfile') as infile:
_count = int(infile.read())
except FileNotFoundError:
_count = 0
def incrcounter(n):
global _count
_count = _count + n
def savecounter():
with open('counterfile', 'w') as outfile:
outfile.write('%d' % _count)
import atexit
atexit.register(savecounter)
Positional and keyword arguments may also be passed to *note register():
87b. to be passed along to the registered function when it is called:
def goodbye(name, adjective):
print('Goodbye %s, it was %s to meet you.' % (name, adjective))
import atexit
atexit.register(goodbye, 'Donny', 'nice')
# or:
atexit.register(goodbye, adjective='nice', name='Donny')
Usage as a *note decorator: 72c.:
import atexit
@atexit.register
def goodbye():
print('You are now leaving the Python sector.')
This only works with functions that can be called without arguments.
File: python.info, Node: traceback — Print or retrieve a stack traceback, Next: __future__ — Future statement definitions, Prev: atexit — Exit handlers, Up: Python Runtime Services
5.30.11 ‘traceback’ — Print or retrieve a stack traceback
---------------------------------------------------------
'Source code:' Lib/traceback.py(1)
__________________________________________________________________
This module provides a standard interface to extract, format and print
stack traces of Python programs. It is more flexible than the
interpreter’s default traceback display, and therefore makes it possible
to configure certain aspects of the output. Finally, it contains a
utility for capturing enough information about an exception to print it
later, without the need to save a reference to the actual exception.
Since exceptions can be the roots of large objects graph, this utility
can significantly improve memory management.
The module uses *note traceback objects: af4. — these are objects of
type *note types.TracebackType: af2, which are assigned to the *note
__traceback__: 12b9. field of *note BaseException: 5b7. instances.
See also
........
Module *note faulthandler: 58.
Used to dump Python tracebacks explicitly, on a fault, after a
timeout, or on a user signal.
Module *note pdb: a5.
Interactive source code debugger for Python programs.
The module’s API can be divided into two parts:
* Module-level functions offering basic functionality, which are
useful for interactive inspection of exceptions and tracebacks.
* *note TracebackException: 25e. class and its helper classes *note
StackSummary: e7e. and *note FrameSummary: e7f. These offer both
more flexibility in the output generated and the ability to store
the information necessary for later formatting without holding
references to actual exception and traceback objects.
* Menu:
* Module-Level Functions: Module-Level Functions<2>.
* TracebackException Objects::
* StackSummary Objects::
* FrameSummary Objects::
* Examples of Using the Module-Level Functions::
* Examples of Using TracebackException::
---------- Footnotes ----------
(1) https://github.com/python/cpython/tree/3.13/Lib/traceback.py
File: python.info, Node: Module-Level Functions<2>, Next: TracebackException Objects, Up: traceback — Print or retrieve a stack traceback
5.30.11.1 Module-Level Functions
................................
-- Function: traceback.print_tb (tb, limit=None, file=None)
Print up to 'limit' stack trace entries from *note traceback
object: af4. 'tb' (starting from the caller’s frame) if 'limit' is
positive. Otherwise, print the last ‘abs(limit)’ entries. If
'limit' is omitted or ‘None’, all entries are printed. If 'file'
is omitted or ‘None’, the output goes to *note sys.stderr: 939.;
otherwise it should be an open *note file: 11b5. or *note file-like
object: 258f. to receive the output.
Note: The meaning of the 'limit' parameter is different than
the meaning of *note sys.tracebacklimit: 15db. A negative
'limit' value corresponds to a positive value of
‘sys.tracebacklimit’, whereas the behaviour of a positive
'limit' value cannot be achieved with ‘sys.tracebacklimit’.
Changed in version 3.5: Added negative 'limit' support.
-- Function: traceback.print_exception (exc, /[, value, tb],
limit=None, file=None, chain=True)
Print exception information and stack trace entries from *note
traceback object: af4. 'tb' to 'file'. This differs from *note
print_tb(): e80. in the following ways:
* if 'tb' is not ‘None’, it prints a header ‘Traceback (most
recent call last):’
* it prints the exception type and 'value' after the stack trace
* if 'type(value)' is *note SyntaxError: 18d. and 'value' has
the appropriate format, it prints the line where the syntax
error occurred with a caret indicating the approximate
position of the error.
Since Python 3.10, instead of passing 'value' and 'tb', an
exception object can be passed as the first argument. If 'value'
and 'tb' are provided, the first argument is ignored in order to
provide backwards compatibility.
The optional 'limit' argument has the same meaning as for *note
print_tb(): e80. If 'chain' is true (the default), then chained
exceptions (the *note __cause__: 12bb. or *note __context__: 12ba.
attributes of the exception) will be printed as well, like the
interpreter itself does when printing an unhandled exception.
Changed in version 3.5: The 'etype' argument is ignored and
inferred from the type of 'value'.
Changed in version 3.10: The 'etype' parameter has been renamed to
'exc' and is now positional-only.
-- Function: traceback.print_exc (limit=None, file=None, chain=True)
This is a shorthand for ‘print_exception(sys.exception(),
limit=limit, file=file, chain=chain)’.
-- Function: traceback.print_last (limit=None, file=None, chain=True)
This is a shorthand for ‘print_exception(sys.last_exc, limit=limit,
file=file, chain=chain)’. In general it will work only after an
exception has reached an interactive prompt (see *note
sys.last_exc: 4ba.).
-- Function: traceback.print_stack (f=None, limit=None, file=None)
Print up to 'limit' stack trace entries (starting from the
invocation point) if 'limit' is positive. Otherwise, print the
last ‘abs(limit)’ entries. If 'limit' is omitted or ‘None’, all
entries are printed. The optional 'f' argument can be used to
specify an alternate *note stack frame: 6db. to start. The
optional 'file' argument has the same meaning as for *note
print_tb(): e80.
Changed in version 3.5: Added negative 'limit' support.
-- Function: traceback.extract_tb (tb, limit=None)
Return a *note StackSummary: e7e. object representing a list of
“pre-processed” stack trace entries extracted from the *note
traceback object: af4. 'tb'. It is useful for alternate formatting
of stack traces. The optional 'limit' argument has the same
meaning as for *note print_tb(): e80. A “pre-processed” stack
trace entry is a *note FrameSummary: e7f. object containing
attributes *note filename: 4448, *note lineno: 4449, *note name:
444a, and *note line: 444b. representing the information that is
usually printed for a stack trace.
-- Function: traceback.extract_stack (f=None, limit=None)
Extract the raw traceback from the current *note stack frame: 6db.
The return value has the same format as for *note extract_tb():
4447. The optional 'f' and 'limit' arguments have the same meaning
as for *note print_stack(): e81.
-- Function: traceback.print_list (extracted_list, file=None)
Print the list of tuples as returned by *note extract_tb(): 4447.
or *note extract_stack(): 444c. as a formatted stack trace to the
given file. If 'file' is ‘None’, the output is written to *note
sys.stderr: 939.
-- Function: traceback.format_list (extracted_list)
Given a list of tuples or *note FrameSummary: e7f. objects as
returned by *note extract_tb(): 4447. or *note extract_stack():
444c, return a list of strings ready for printing. Each string in
the resulting list corresponds to the item with the same index in
the argument list. Each string ends in a newline; the strings may
contain internal newlines as well, for those items whose source
text line is not ‘None’.
-- Function: traceback.format_exception_only (exc, /[, value], *,
show_group=False)
Format the exception part of a traceback using an exception value
such as given by *note sys.last_value: 4bc. The return value is a
list of strings, each ending in a newline. The list contains the
exception’s message, which is normally a single string; however,
for *note SyntaxError: 18d. exceptions, it contains several lines
that (when printed) display detailed information about where the
syntax error occurred. Following the message, the list contains
the exception’s *note notes: 2282.
Since Python 3.10, instead of passing 'value', an exception object
can be passed as the first argument. If 'value' is provided, the
first argument is ignored in order to provide backwards
compatibility.
When 'show_group' is ‘True’, and the exception is an instance of
*note BaseExceptionGroup: 261, the nested exceptions are included
as well, recursively, with indentation relative to their nesting
depth.
Changed in version 3.10: The 'etype' parameter has been renamed to
'exc' and is now positional-only.
Changed in version 3.11: The returned list now includes any *note
notes: 2282. attached to the exception.
Changed in version 3.13: 'show_group' parameter was added.
-- Function: traceback.format_exception (exc, /[, value, tb],
limit=None, chain=True)
Format a stack trace and the exception information. The arguments
have the same meaning as the corresponding arguments to *note
print_exception(): 84b. The return value is a list of strings,
each ending in a newline and some containing internal newlines.
When these lines are concatenated and printed, exactly the same
text is printed as does *note print_exception(): 84b.
Changed in version 3.5: The 'etype' argument is ignored and
inferred from the type of 'value'.
Changed in version 3.10: This function’s behavior and signature
were modified to match *note print_exception(): 84b.
-- Function: traceback.format_exc (limit=None, chain=True)
This is like ‘print_exc(limit)’ but returns a string instead of
printing to a file.
-- Function: traceback.format_tb (tb, limit=None)
A shorthand for ‘format_list(extract_tb(tb, limit))’.
-- Function: traceback.format_stack (f=None, limit=None)
A shorthand for ‘format_list(extract_stack(f, limit))’.
-- Function: traceback.clear_frames (tb)
Clears the local variables of all the stack frames in a *note
traceback: af4. 'tb' by calling the *note clear(): 282. method of
each *note frame object: 6db.
Added in version 3.4.
-- Function: traceback.walk_stack (f)
Walk a stack following *note f.f_back: 787. from the given frame,
yielding the frame and line number for each frame. If 'f' is
‘None’, the current stack is used. This helper is used with *note
StackSummary.extract(): 4451.
Added in version 3.5.
-- Function: traceback.walk_tb (tb)
Walk a traceback following *note tb_next: af3. yielding the frame
and line number for each frame. This helper is used with *note
StackSummary.extract(): 4451.
Added in version 3.5.
File: python.info, Node: TracebackException Objects, Next: StackSummary Objects, Prev: Module-Level Functions<2>, Up: traceback — Print or retrieve a stack traceback
5.30.11.2 ‘TracebackException’ Objects
......................................
Added in version 3.5.
‘TracebackException’ objects are created from actual exceptions to
capture data for later printing. They offer a more lightweight method
of storing this information by avoiding holding references to *note
traceback: af4. and *note frame: 6db. objects. In addition, they expose
more options to configure the output compared to the module-level
functions described above.
-- Class: traceback.TracebackException (exc_type, exc_value,
exc_traceback, *, limit=None, lookup_lines=True,
capture_locals=False, compact=False, max_group_width=15,
max_group_depth=10)
Capture an exception for later rendering. The meaning of 'limit',
'lookup_lines' and 'capture_locals' are as for the *note
StackSummary: e7e. class.
If 'compact' is true, only data that is required by
‘TracebackException’’s *note format(): 61b. method is saved in the
class attributes. In particular, the *note __context__: 4453.
field is calculated only if *note __cause__: 4454. is ‘None’ and
*note __suppress_context__: 4455. is false.
Note that when locals are captured, they are also shown in the
traceback.
'max_group_width' and 'max_group_depth' control the formatting of
exception groups (see *note BaseExceptionGroup: 261.). The depth
refers to the nesting level of the group, and the width refers to
the size of a single exception group’s exceptions array. The
formatted output is truncated when either limit is exceeded.
Changed in version 3.10: Added the 'compact' parameter.
Changed in version 3.11: Added the 'max_group_width' and
'max_group_depth' parameters.
-- Attribute: __cause__
A ‘TracebackException’ of the original *note __cause__: 12bb.
-- Attribute: __context__
A ‘TracebackException’ of the original *note __context__:
12ba.
-- Attribute: exceptions
If ‘self’ represents an *note ExceptionGroup: 1b6, this field
holds a list of ‘TracebackException’ instances representing
the nested exceptions. Otherwise it is ‘None’.
Added in version 3.11.
-- Attribute: __suppress_context__
The *note __suppress_context__: 20b8. value from the original
exception.
-- Attribute: __notes__
The *note __notes__: 2282. value from the original exception,
or ‘None’ if the exception does not have any notes. If it is
not ‘None’ is it formatted in the traceback after the
exception string.
Added in version 3.11.
-- Attribute: stack
A *note StackSummary: e7e. representing the traceback.
-- Attribute: exc_type
The class of the original traceback.
Deprecated since version 3.13.
-- Attribute: exc_type_str
String display of the class of the original exception.
Added in version 3.13.
-- Attribute: filename
For syntax errors - the file name where the error occurred.
-- Attribute: lineno
For syntax errors - the line number where the error occurred.
-- Attribute: end_lineno
For syntax errors - the end line number where the error
occurred. Can be ‘None’ if not present.
Added in version 3.10.
-- Attribute: text
For syntax errors - the text where the error occurred.
-- Attribute: offset
For syntax errors - the offset into the text where the error
occurred.
-- Attribute: end_offset
For syntax errors - the end offset into the text where the
error occurred. Can be ‘None’ if not present.
Added in version 3.10.
-- Attribute: msg
For syntax errors - the compiler error message.
-- Method: classmethod from_exception (exc, *, limit=None,
lookup_lines=True, capture_locals=False)
Capture an exception for later rendering. 'limit',
'lookup_lines' and 'capture_locals' are as for the *note
StackSummary: e7e. class.
Note that when locals are captured, they are also shown in the
traceback.
-- Method: print (*, file=None, chain=True)
Print to 'file' (default ‘sys.stderr’) the exception
information returned by *note format(): 61b.
Added in version 3.11.
-- Method: format (*, chain=True)
Format the exception.
If 'chain' is not ‘True’, *note __cause__: 4454. and *note
__context__: 4453. will not be formatted.
The return value is a generator of strings, each ending in a
newline and some containing internal newlines. *note
print_exception(): 84b. is a wrapper around this method which
just prints the lines to a file.
-- Method: format_exception_only (*, show_group=False)
Format the exception part of the traceback.
The return value is a generator of strings, each ending in a
newline.
When 'show_group' is ‘False’, the generator emits the
exception’s message followed by its notes (if it has any).
The exception message is normally a single string; however,
for *note SyntaxError: 18d. exceptions, it consists of several
lines that (when printed) display detailed information about
where the syntax error occurred.
When 'show_group' is ‘True’, and the exception is an instance
of *note BaseExceptionGroup: 261, the nested exceptions are
included as well, recursively, with indentation relative to
their nesting depth.
Changed in version 3.11: The exception’s *note notes: 2282.
are now included in the output.
Changed in version 3.13: Added the 'show_group' parameter.
File: python.info, Node: StackSummary Objects, Next: FrameSummary Objects, Prev: TracebackException Objects, Up: traceback — Print or retrieve a stack traceback
5.30.11.3 ‘StackSummary’ Objects
................................
Added in version 3.5.
‘StackSummary’ objects represent a call stack ready for formatting.
-- Class: traceback.StackSummary
-- Method: classmethod extract (frame_gen, *, limit=None,
lookup_lines=True, capture_locals=False)
Construct a ‘StackSummary’ object from a frame generator (such
as is returned by *note walk_stack(): e7c. or *note walk_tb():
e7d.).
If 'limit' is supplied, only this many frames are taken from
'frame_gen'. If 'lookup_lines' is ‘False’, the returned *note
FrameSummary: e7f. objects will not have read their lines in
yet, making the cost of creating the ‘StackSummary’ cheaper
(which may be valuable if it may not actually get formatted).
If 'capture_locals' is ‘True’ the local variables in each
‘FrameSummary’ are captured as object representations.
Changed in version 3.12: Exceptions raised from *note repr():
7f9. on a local variable (when 'capture_locals' is ‘True’) are
no longer propagated to the caller.
-- Method: classmethod from_list (a_list)
Construct a ‘StackSummary’ object from a supplied list of
*note FrameSummary: e7f. objects or old-style list of tuples.
Each tuple should be a 4-tuple with 'filename', 'lineno',
'name', 'line' as the elements.
-- Method: format ()
Returns a list of strings ready for printing. Each string in
the resulting list corresponds to a single *note frame: 6db.
from the stack. Each string ends in a newline; the strings
may contain internal newlines as well, for those items with
source text lines.
For long sequences of the same frame and line, the first few
repetitions are shown, followed by a summary line stating the
exact number of further repetitions.
Changed in version 3.6: Long sequences of repeated frames are
now abbreviated.
-- Method: format_frame_summary (frame_summary)
Returns a string for printing one of the *note frames: 6db.
involved in the stack. This method is called for each *note
FrameSummary: e7f. object to be printed by *note
StackSummary.format(): 4464. If it returns ‘None’, the frame
is omitted from the output.
Added in version 3.11.
File: python.info, Node: FrameSummary Objects, Next: Examples of Using the Module-Level Functions, Prev: StackSummary Objects, Up: traceback — Print or retrieve a stack traceback
5.30.11.4 ‘FrameSummary’ Objects
................................
Added in version 3.5.
A ‘FrameSummary’ object represents a single *note frame: 6db. in a *note
traceback: af4.
-- Class: traceback.FrameSummary (filename, lineno, name, *,
lookup_line=True, locals=None, line=None, end_lineno=None,
colno=None, end_colno=None)
Represents a single *note frame: 6db. in the *note traceback: af4.
or stack that is being formatted or printed. It may optionally
have a stringified version of the frame’s locals included in it.
If 'lookup_line' is ‘False’, the source code is not looked up until
the ‘FrameSummary’ has the *note line: 444b. attribute accessed
(which also happens when casting it to a *note tuple: 36b.). *note
line: 444b. may be directly provided, and will prevent line lookups
happening at all. 'locals' is an optional local variable mapping,
and if supplied the variable representations are stored in the
summary for later display.
‘FrameSummary’ instances have the following attributes:
-- Attribute: filename
The filename of the source code for this frame. Equivalent to
accessing *note f.f_code.co_filename: 1359. on a *note frame
object: 6db. 'f'.
-- Attribute: lineno
The line number of the source code for this frame.
-- Attribute: name
Equivalent to accessing *note f.f_code.co_name: 1f36. on a
*note frame object: 6db. 'f'.
-- Attribute: line
A string representing the source code for this frame, with
leading and trailing whitespace stripped. If the source is
not available, it is ‘None’.
-- Attribute: end_lineno
The last line number of the source code for this frame. By
default, it is set to ‘lineno’ and indexation starts from 1.
Changed in version 3.13: The default value changed from ‘None’
to ‘lineno’.
-- Attribute: colno
The column number of the source code for this frame. By
default, it is ‘None’ and indexation starts from 0.
-- Attribute: end_colno
The last column number of the source code for this frame. By
default, it is ‘None’ and indexation starts from 0.
File: python.info, Node: Examples of Using the Module-Level Functions, Next: Examples of Using TracebackException, Prev: FrameSummary Objects, Up: traceback — Print or retrieve a stack traceback
5.30.11.5 Examples of Using the Module-Level Functions
......................................................
This simple example implements a basic read-eval-print loop, similar to
(but less useful than) the standard Python interactive interpreter loop.
For a more complete implementation of the interpreter loop, refer to the
*note code: 1a. module.
import sys, traceback
def run_user_code(envdir):
source = input(">>> ")
try:
exec(source, envdir)
except Exception:
print("Exception in user code:")
print("-"*60)
traceback.print_exc(file=sys.stdout)
print("-"*60)
envdir = {}
while True:
run_user_code(envdir)
The following example demonstrates the different ways to print and
format the exception and traceback:
import sys, traceback
def lumberjack():
bright_side_of_life()
def bright_side_of_life():
return tuple()[0]
try:
lumberjack()
except IndexError as exc:
print("*** print_tb:")
traceback.print_tb(exc.__traceback__, limit=1, file=sys.stdout)
print("*** print_exception:")
traceback.print_exception(exc, limit=2, file=sys.stdout)
print("*** print_exc:")
traceback.print_exc(limit=2, file=sys.stdout)
print("*** format_exc, first and last line:")
formatted_lines = traceback.format_exc().splitlines()
print(formatted_lines[0])
print(formatted_lines[-1])
print("*** format_exception:")
print(repr(traceback.format_exception(exc)))
print("*** extract_tb:")
print(repr(traceback.extract_tb(exc.__traceback__)))
print("*** format_tb:")
print(repr(traceback.format_tb(exc.__traceback__)))
print("*** tb_lineno:", exc.__traceback__.tb_lineno)
The output for the example would look similar to this:
*** print_tb:
File "", line 10, in
lumberjack()
~~~~~~~~~~^^
*** print_exception:
Traceback (most recent call last):
File "", line 10, in
lumberjack()
~~~~~~~~~~^^
File "", line 4, in lumberjack
bright_side_of_life()
~~~~~~~~~~~~~~~~~~~^^
IndexError: tuple index out of range
*** print_exc:
Traceback (most recent call last):
File "", line 10, in
lumberjack()
~~~~~~~~~~^^
File "", line 4, in lumberjack
bright_side_of_life()
~~~~~~~~~~~~~~~~~~~^^
IndexError: tuple index out of range
*** format_exc, first and last line:
Traceback (most recent call last):
IndexError: tuple index out of range
*** format_exception:
['Traceback (most recent call last):\n',
' File "", line 10, in \n lumberjack()\n ~~~~~~~~~~^^\n',
' File "", line 4, in lumberjack\n bright_side_of_life()\n ~~~~~~~~~~~~~~~~~~~^^\n',
' File "", line 7, in bright_side_of_life\n return tuple()[0]\n ~~~~~~~^^^\n',
'IndexError: tuple index out of range\n']
*** extract_tb:
[, line 10 in >,
, line 4 in lumberjack>,
, line 7 in bright_side_of_life>]
*** format_tb:
[' File "", line 10, in \n lumberjack()\n ~~~~~~~~~~^^\n',
' File "", line 4, in lumberjack\n bright_side_of_life()\n ~~~~~~~~~~~~~~~~~~~^^\n',
' File "", line 7, in bright_side_of_life\n return tuple()[0]\n ~~~~~~~^^^\n']
*** tb_lineno: 10
The following example shows the different ways to print and format the
stack:
>>> import traceback
>>> def another_function():
... lumberstack()
...
>>> def lumberstack():
... traceback.print_stack()
... print(repr(traceback.extract_stack()))
... print(repr(traceback.format_stack()))
...
>>> another_function()
File "", line 10, in
another_function()
File "", line 3, in another_function
lumberstack()
File "", line 6, in lumberstack
traceback.print_stack()
[('', 10, '', 'another_function()'),
('', 3, 'another_function', 'lumberstack()'),
('', 7, 'lumberstack', 'print(repr(traceback.extract_stack()))')]
[' File "", line 10, in \n another_function()\n',
' File "