diff options
| author | Craig Jennings <c@cjennings.net> | 2024-04-07 13:41:34 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2024-04-07 13:41:34 -0500 |
| commit | 754bbf7a25a8dda49b5d08ef0d0443bbf5af0e36 (patch) | |
| tree | f1190704f78f04a2b0b4c977d20fe96a828377f1 /devdocs/python~3.12/library%2Fpickle.html | |
new repository
Diffstat (limited to 'devdocs/python~3.12/library%2Fpickle.html')
| -rw-r--r-- | devdocs/python~3.12/library%2Fpickle.html | 397 |
1 files changed, 397 insertions, 0 deletions
diff --git a/devdocs/python~3.12/library%2Fpickle.html b/devdocs/python~3.12/library%2Fpickle.html new file mode 100644 index 00000000..009047f4 --- /dev/null +++ b/devdocs/python~3.12/library%2Fpickle.html @@ -0,0 +1,397 @@ + <span id="pickle-python-object-serialization"></span><h1>pickle — Python object serialization</h1> <p><strong>Source code:</strong> <a class="reference external" href="https://github.com/python/cpython/tree/3.12/Lib/pickle.py">Lib/pickle.py</a></p> <p>The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module implements binary protocols for serializing and de-serializing a Python object structure. <em>“Pickling”</em> is the process whereby a Python object hierarchy is converted into a byte stream, and <em>“unpickling”</em> is the inverse operation, whereby a byte stream (from a <a class="reference internal" href="../glossary#term-binary-file"><span class="xref std std-term">binary file</span></a> or <a class="reference internal" href="../glossary#term-bytes-like-object"><span class="xref std std-term">bytes-like object</span></a>) is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as “serialization”, “marshalling,” <a class="footnote-reference brackets" href="#id7" id="id1">1</a> or “flattening”; however, to avoid confusion, the terms used here are “pickling” and “unpickling”.</p> <div class="admonition warning"> <p class="admonition-title">Warning</p> <p>The <code>pickle</code> module <strong>is not secure</strong>. Only unpickle data you trust.</p> <p>It is possible to construct malicious pickle data which will <strong>execute arbitrary code during unpickling</strong>. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.</p> <p>Consider signing data with <a class="reference internal" href="hmac#module-hmac" title="hmac: Keyed-Hashing for Message Authentication (HMAC) implementation"><code>hmac</code></a> if you need to ensure that it has not been tampered with.</p> <p>Safer serialization formats such as <a class="reference internal" href="json#module-json" title="json: Encode and decode the JSON format."><code>json</code></a> may be more appropriate if you are processing untrusted data. See <a class="reference internal" href="#comparison-with-json"><span class="std std-ref">Comparison with json</span></a>.</p> </div> <section id="relationship-to-other-python-modules"> <h2>Relationship to other Python modules</h2> <section id="comparison-with-marshal"> <h3>Comparison with <code>marshal</code> +</h3> <p>Python has a more primitive serialization module called <a class="reference internal" href="marshal#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code>marshal</code></a>, but in general <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> should always be the preferred way to serialize Python objects. <a class="reference internal" href="marshal#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code>marshal</code></a> exists primarily to support Python’s <code>.pyc</code> files.</p> <p>The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module differs from <a class="reference internal" href="marshal#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code>marshal</code></a> in several significant ways:</p> <ul> <li> +<p>The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module keeps track of the objects it has already serialized, so that later references to the same object won’t be serialized again. <a class="reference internal" href="marshal#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code>marshal</code></a> doesn’t do this.</p> <p>This has implications both for recursive objects and object sharing. Recursive objects are objects that contain references to themselves. These are not handled by marshal, and in fact, attempting to marshal recursive objects will crash your Python interpreter. Object sharing happens when there are multiple references to the same object in different places in the object hierarchy being serialized. <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> stores such objects only once, and ensures that all other references point to the master copy. Shared objects remain shared, which can be very important for mutable objects.</p> </li> <li> +<a class="reference internal" href="marshal#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code>marshal</code></a> cannot be used to serialize user-defined classes and their instances. <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored.</li> <li>The <a class="reference internal" href="marshal#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code>marshal</code></a> serialization format is not guaranteed to be portable across Python versions. Because its primary job in life is to support <code>.pyc</code> files, the Python implementers reserve the right to change the serialization format in non-backwards compatible ways should the need arise. The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> serialization format is guaranteed to be backwards compatible across Python releases provided a compatible pickle protocol is chosen and pickling and unpickling code deals with Python 2 to Python 3 type differences if your data is crossing that unique breaking change language boundary.</li> </ul> </section> <section id="comparison-with-json"> <span id="id2"></span><h3>Comparison with <code>json</code> +</h3> <p>There are fundamental differences between the pickle protocols and <a class="reference external" href="https://json.org">JSON (JavaScript Object Notation)</a>:</p> <ul class="simple"> <li>JSON is a text serialization format (it outputs unicode text, although most of the time it is then encoded to <code>utf-8</code>), while pickle is a binary serialization format;</li> <li>JSON is human-readable, while pickle is not;</li> <li>JSON is interoperable and widely used outside of the Python ecosystem, while pickle is Python-specific;</li> <li>JSON, by default, can only represent a subset of the Python built-in types, and no custom classes; pickle can represent an extremely large number of Python types (many of them automatically, by clever usage of Python’s introspection facilities; complex cases can be tackled by implementing <a class="reference internal" href="#pickle-inst"><span class="std std-ref">specific object APIs</span></a>);</li> <li>Unlike pickle, deserializing untrusted JSON does not in itself create an arbitrary code execution vulnerability.</li> </ul> <div class="admonition seealso"> <p class="admonition-title">See also</p> <p>The <a class="reference internal" href="json#module-json" title="json: Encode and decode the JSON format."><code>json</code></a> module: a standard library module allowing JSON serialization and deserialization.</p> </div> </section> </section> <section id="data-stream-format"> <span id="pickle-protocols"></span><h2>Data stream format</h2> <p id="index-1">The data format used by <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON or XDR (which can’t represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects.</p> <p>By default, the <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> data format uses a relatively compact binary representation. If you need optimal size characteristics, you can efficiently <a class="reference internal" href="archiving"><span class="doc">compress</span></a> pickled data.</p> <p>The module <a class="reference internal" href="pickletools#module-pickletools" title="pickletools: Contains extensive comments about the pickle protocols and pickle-machine opcodes, as well as some useful functions."><code>pickletools</code></a> contains tools for analyzing data streams generated by <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a>. <a class="reference internal" href="pickletools#module-pickletools" title="pickletools: Contains extensive comments about the pickle protocols and pickle-machine opcodes, as well as some useful functions."><code>pickletools</code></a> source code has extensive comments about opcodes used by pickle protocols.</p> <p>There are currently 6 different protocols which can be used for pickling. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.</p> <ul class="simple"> <li>Protocol version 0 is the original “human-readable” protocol and is backwards compatible with earlier versions of Python.</li> <li>Protocol version 1 is an old binary format which is also compatible with earlier versions of Python.</li> <li>Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of <a class="reference internal" href="../glossary#term-new-style-class"><span class="xref std std-term">new-style classes</span></a>. Refer to <span class="target" id="index-2"></span><a class="pep reference external" href="https://peps.python.org/pep-0307/"><strong>PEP 307</strong></a> for information about improvements brought by protocol 2.</li> <li>Protocol version 3 was added in Python 3.0. It has explicit support for <a class="reference internal" href="stdtypes#bytes" title="bytes"><code>bytes</code></a> objects and cannot be unpickled by Python 2.x. This was the default protocol in Python 3.0–3.7.</li> <li>Protocol version 4 was added in Python 3.4. It adds support for very large objects, pickling more kinds of objects, and some data format optimizations. It is the default protocol starting with Python 3.8. Refer to <span class="target" id="index-3"></span><a class="pep reference external" href="https://peps.python.org/pep-3154/"><strong>PEP 3154</strong></a> for information about improvements brought by protocol 4.</li> <li>Protocol version 5 was added in Python 3.8. It adds support for out-of-band data and speedup for in-band data. Refer to <span class="target" id="index-4"></span><a class="pep reference external" href="https://peps.python.org/pep-0574/"><strong>PEP 574</strong></a> for information about improvements brought by protocol 5.</li> </ul> <div class="admonition note"> <p class="admonition-title">Note</p> <p>Serialization is a more primitive notion than persistence; although <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> reads and writes file objects, it does not handle the issue of naming persistent objects, nor the (even more complicated) issue of concurrent access to persistent objects. The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module can transform a complex object into a byte stream and it can transform the byte stream into an object with the same internal structure. Perhaps the most obvious thing to do with these byte streams is to write them onto a file, but it is also conceivable to send them across a network or store them in a database. The <a class="reference internal" href="shelve#module-shelve" title="shelve: Python object persistence."><code>shelve</code></a> module provides a simple interface to pickle and unpickle objects on DBM-style database files.</p> </div> </section> <section id="module-interface"> <h2>Module Interface</h2> <p>To serialize an object hierarchy, you simply call the <a class="reference internal" href="#pickle.dumps" title="pickle.dumps"><code>dumps()</code></a> function. Similarly, to de-serialize a data stream, you call the <a class="reference internal" href="#pickle.loads" title="pickle.loads"><code>loads()</code></a> function. However, if you want more control over serialization and de-serialization, you can create a <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> or an <a class="reference internal" href="#pickle.Unpickler" title="pickle.Unpickler"><code>Unpickler</code></a> object, respectively.</p> <p>The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module provides the following constants:</p> <dl class="py data"> <dt class="sig sig-object py" id="pickle.HIGHEST_PROTOCOL"> +<code>pickle.HIGHEST_PROTOCOL</code> </dt> <dd> +<p>An integer, the highest <a class="reference internal" href="#pickle-protocols"><span class="std std-ref">protocol version</span></a> available. This value can be passed as a <em>protocol</em> value to functions <a class="reference internal" href="#pickle.dump" title="pickle.dump"><code>dump()</code></a> and <a class="reference internal" href="#pickle.dumps" title="pickle.dumps"><code>dumps()</code></a> as well as the <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> constructor.</p> </dd> +</dl> <dl class="py data"> <dt class="sig sig-object py" id="pickle.DEFAULT_PROTOCOL"> +<code>pickle.DEFAULT_PROTOCOL</code> </dt> <dd> +<p>An integer, the default <a class="reference internal" href="#pickle-protocols"><span class="std std-ref">protocol version</span></a> used for pickling. May be less than <a class="reference internal" href="#pickle.HIGHEST_PROTOCOL" title="pickle.HIGHEST_PROTOCOL"><code>HIGHEST_PROTOCOL</code></a>. Currently the default protocol is 4, first introduced in Python 3.4 and incompatible with previous versions.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.0: </span>The default protocol is 3.</p> </div> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>The default protocol is 4.</p> </div> </dd> +</dl> <p>The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module provides the following functions to make the pickling process more convenient:</p> <dl class="py function"> <dt class="sig sig-object py" id="pickle.dump"> +<code>pickle.dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None)</code> </dt> <dd> +<p>Write the pickled representation of the object <em>obj</em> to the open <a class="reference internal" href="../glossary#term-file-object"><span class="xref std std-term">file object</span></a> <em>file</em>. This is equivalent to <code>Pickler(file, protocol).dump(obj)</code>.</p> <p>Arguments <em>file</em>, <em>protocol</em>, <em>fix_imports</em> and <em>buffer_callback</em> have the same meaning as in the <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> constructor.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>The <em>buffer_callback</em> argument was added.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="pickle.dumps"> +<code>pickle.dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None)</code> </dt> <dd> +<p>Return the pickled representation of the object <em>obj</em> as a <a class="reference internal" href="stdtypes#bytes" title="bytes"><code>bytes</code></a> object, instead of writing it to a file.</p> <p>Arguments <em>protocol</em>, <em>fix_imports</em> and <em>buffer_callback</em> have the same meaning as in the <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> constructor.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>The <em>buffer_callback</em> argument was added.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="pickle.load"> +<code>pickle.load(file, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)</code> </dt> <dd> +<p>Read the pickled representation of an object from the open <a class="reference internal" href="../glossary#term-file-object"><span class="xref std std-term">file object</span></a> <em>file</em> and return the reconstituted object hierarchy specified therein. This is equivalent to <code>Unpickler(file).load()</code>.</p> <p>The protocol version of the pickle is detected automatically, so no protocol argument is needed. Bytes past the pickled representation of the object are ignored.</p> <p>Arguments <em>file</em>, <em>fix_imports</em>, <em>encoding</em>, <em>errors</em>, <em>strict</em> and <em>buffers</em> have the same meaning as in the <a class="reference internal" href="#pickle.Unpickler" title="pickle.Unpickler"><code>Unpickler</code></a> constructor.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>The <em>buffers</em> argument was added.</p> </div> </dd> +</dl> <dl class="py function"> <dt class="sig sig-object py" id="pickle.loads"> +<code>pickle.loads(data, /, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)</code> </dt> <dd> +<p>Return the reconstituted object hierarchy of the pickled representation <em>data</em> of an object. <em>data</em> must be a <a class="reference internal" href="../glossary#term-bytes-like-object"><span class="xref std std-term">bytes-like object</span></a>.</p> <p>The protocol version of the pickle is detected automatically, so no protocol argument is needed. Bytes past the pickled representation of the object are ignored.</p> <p>Arguments <em>fix_imports</em>, <em>encoding</em>, <em>errors</em>, <em>strict</em> and <em>buffers</em> have the same meaning as in the <a class="reference internal" href="#pickle.Unpickler" title="pickle.Unpickler"><code>Unpickler</code></a> constructor.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>The <em>buffers</em> argument was added.</p> </div> </dd> +</dl> <p>The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module defines three exceptions:</p> <dl class="py exception"> <dt class="sig sig-object py" id="pickle.PickleError"> +<code>exception pickle.PickleError</code> </dt> <dd> +<p>Common base class for the other pickling exceptions. It inherits <a class="reference internal" href="exceptions#Exception" title="Exception"><code>Exception</code></a>.</p> </dd> +</dl> <dl class="py exception"> <dt class="sig sig-object py" id="pickle.PicklingError"> +<code>exception pickle.PicklingError</code> </dt> <dd> +<p>Error raised when an unpicklable object is encountered by <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a>. It inherits <a class="reference internal" href="#pickle.PickleError" title="pickle.PickleError"><code>PickleError</code></a>.</p> <p>Refer to <a class="reference internal" href="#pickle-picklable"><span class="std std-ref">What can be pickled and unpickled?</span></a> to learn what kinds of objects can be pickled.</p> </dd> +</dl> <dl class="py exception"> <dt class="sig sig-object py" id="pickle.UnpicklingError"> +<code>exception pickle.UnpicklingError</code> </dt> <dd> +<p>Error raised when there is a problem unpickling an object, such as a data corruption or a security violation. It inherits <a class="reference internal" href="#pickle.PickleError" title="pickle.PickleError"><code>PickleError</code></a>.</p> <p>Note that other exceptions may also be raised during unpickling, including (but not necessarily limited to) AttributeError, EOFError, ImportError, and IndexError.</p> </dd> +</dl> <p>The <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module exports three classes, <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a>, <a class="reference internal" href="#pickle.Unpickler" title="pickle.Unpickler"><code>Unpickler</code></a> and <a class="reference internal" href="#pickle.PickleBuffer" title="pickle.PickleBuffer"><code>PickleBuffer</code></a>:</p> <dl class="py class"> <dt class="sig sig-object py" id="pickle.Pickler"> +<code>class pickle.Pickler(file, protocol=None, *, fix_imports=True, buffer_callback=None)</code> </dt> <dd> +<p>This takes a binary file for writing a pickle data stream.</p> <p>The optional <em>protocol</em> argument, an integer, tells the pickler to use the given protocol; supported protocols are 0 to <a class="reference internal" href="#pickle.HIGHEST_PROTOCOL" title="pickle.HIGHEST_PROTOCOL"><code>HIGHEST_PROTOCOL</code></a>. If not specified, the default is <a class="reference internal" href="#pickle.DEFAULT_PROTOCOL" title="pickle.DEFAULT_PROTOCOL"><code>DEFAULT_PROTOCOL</code></a>. If a negative number is specified, <a class="reference internal" href="#pickle.HIGHEST_PROTOCOL" title="pickle.HIGHEST_PROTOCOL"><code>HIGHEST_PROTOCOL</code></a> is selected.</p> <p>The <em>file</em> argument must have a write() method that accepts a single bytes argument. It can thus be an on-disk file opened for binary writing, an <a class="reference internal" href="io#io.BytesIO" title="io.BytesIO"><code>io.BytesIO</code></a> instance, or any other custom object that meets this interface.</p> <p>If <em>fix_imports</em> is true and <em>protocol</em> is less than 3, pickle will try to map the new Python 3 names to the old module names used in Python 2, so that the pickle data stream is readable with Python 2.</p> <p>If <em>buffer_callback</em> is None (the default), buffer views are serialized into <em>file</em> as part of the pickle stream.</p> <p>If <em>buffer_callback</em> is not None, then it can be called any number of times with a buffer view. If the callback returns a false value (such as None), the given buffer is <a class="reference internal" href="#pickle-oob"><span class="std std-ref">out-of-band</span></a>; otherwise the buffer is serialized in-band, i.e. inside the pickle stream.</p> <p>It is an error if <em>buffer_callback</em> is not None and <em>protocol</em> is None or smaller than 5.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>The <em>buffer_callback</em> argument was added.</p> </div> <dl class="py method"> <dt class="sig sig-object py" id="pickle.Pickler.dump"> +<code>dump(obj)</code> </dt> <dd> +<p>Write the pickled representation of <em>obj</em> to the open file object given in the constructor.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="pickle.Pickler.persistent_id"> +<code>persistent_id(obj)</code> </dt> <dd> +<p>Do nothing by default. This exists so a subclass can override it.</p> <p>If <a class="reference internal" href="#pickle.Pickler.persistent_id" title="pickle.Pickler.persistent_id"><code>persistent_id()</code></a> returns <code>None</code>, <em>obj</em> is pickled as usual. Any other value causes <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> to emit the returned value as a persistent ID for <em>obj</em>. The meaning of this persistent ID should be defined by <a class="reference internal" href="#pickle.Unpickler.persistent_load" title="pickle.Unpickler.persistent_load"><code>Unpickler.persistent_load()</code></a>. Note that the value returned by <a class="reference internal" href="#pickle.Pickler.persistent_id" title="pickle.Pickler.persistent_id"><code>persistent_id()</code></a> cannot itself have a persistent ID.</p> <p>See <a class="reference internal" href="#pickle-persistent"><span class="std std-ref">Persistence of External Objects</span></a> for details and examples of uses.</p> </dd> +</dl> <dl class="py attribute"> <dt class="sig sig-object py" id="pickle.Pickler.dispatch_table"> +<code>dispatch_table</code> </dt> <dd> +<p>A pickler object’s dispatch table is a registry of <em>reduction functions</em> of the kind which can be declared using <a class="reference internal" href="copyreg#copyreg.pickle" title="copyreg.pickle"><code>copyreg.pickle()</code></a>. It is a mapping whose keys are classes and whose values are reduction functions. A reduction function takes a single argument of the associated class and should conform to the same interface as a <code>__reduce__()</code> method.</p> <p>By default, a pickler object will not have a <a class="reference internal" href="#pickle.Pickler.dispatch_table" title="pickle.Pickler.dispatch_table"><code>dispatch_table</code></a> attribute, and it will instead use the global dispatch table managed by the <a class="reference internal" href="copyreg#module-copyreg" title="copyreg: Register pickle support functions."><code>copyreg</code></a> module. However, to customize the pickling for a specific pickler object one can set the <a class="reference internal" href="#pickle.Pickler.dispatch_table" title="pickle.Pickler.dispatch_table"><code>dispatch_table</code></a> attribute to a dict-like object. Alternatively, if a subclass of <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> has a <a class="reference internal" href="#pickle.Pickler.dispatch_table" title="pickle.Pickler.dispatch_table"><code>dispatch_table</code></a> attribute then this will be used as the default dispatch table for instances of that class.</p> <p>See <a class="reference internal" href="#pickle-dispatch"><span class="std std-ref">Dispatch Tables</span></a> for usage examples.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.3.</span></p> </div> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="pickle.Pickler.reducer_override"> +<code>reducer_override(obj)</code> </dt> <dd> +<p>Special reducer that can be defined in <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> subclasses. This method has priority over any reducer in the <a class="reference internal" href="#pickle.Pickler.dispatch_table" title="pickle.Pickler.dispatch_table"><code>dispatch_table</code></a>. It should conform to the same interface as a <code>__reduce__()</code> method, and can optionally return <code>NotImplemented</code> to fallback on <a class="reference internal" href="#pickle.Pickler.dispatch_table" title="pickle.Pickler.dispatch_table"><code>dispatch_table</code></a>-registered reducers to pickle <code>obj</code>.</p> <p>For a detailed example, see <a class="reference internal" href="#reducer-override"><span class="std std-ref">Custom Reduction for Types, Functions, and Other Objects</span></a>.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> </dd> +</dl> <dl class="py attribute"> <dt class="sig sig-object py" id="pickle.Pickler.fast"> +<code>fast</code> </dt> <dd> +<p>Deprecated. Enable fast mode if set to a true value. The fast mode disables the usage of memo, therefore speeding the pickling process by not generating superfluous PUT opcodes. It should not be used with self-referential objects, doing otherwise will cause <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> to recurse infinitely.</p> <p>Use <a class="reference internal" href="pickletools#pickletools.optimize" title="pickletools.optimize"><code>pickletools.optimize()</code></a> if you need more compact pickles.</p> </dd> +</dl> </dd> +</dl> <dl class="py class"> <dt class="sig sig-object py" id="pickle.Unpickler"> +<code>class pickle.Unpickler(file, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)</code> </dt> <dd> +<p>This takes a binary file for reading a pickle data stream.</p> <p>The protocol version of the pickle is detected automatically, so no protocol argument is needed.</p> <p>The argument <em>file</em> must have three methods, a read() method that takes an integer argument, a readinto() method that takes a buffer argument and a readline() method that requires no arguments, as in the <a class="reference internal" href="io#io.BufferedIOBase" title="io.BufferedIOBase"><code>io.BufferedIOBase</code></a> interface. Thus <em>file</em> can be an on-disk file opened for binary reading, an <a class="reference internal" href="io#io.BytesIO" title="io.BytesIO"><code>io.BytesIO</code></a> object, or any other custom object that meets this interface.</p> <p>The optional arguments <em>fix_imports</em>, <em>encoding</em> and <em>errors</em> are used to control compatibility support for pickle stream generated by Python 2. If <em>fix_imports</em> is true, pickle will try to map the old Python 2 names to the new names used in Python 3. The <em>encoding</em> and <em>errors</em> tell pickle how to decode 8-bit string instances pickled by Python 2; these default to ‘ASCII’ and ‘strict’, respectively. The <em>encoding</em> can be ‘bytes’ to read these 8-bit string instances as bytes objects. Using <code>encoding='latin1'</code> is required for unpickling NumPy arrays and instances of <a class="reference internal" href="datetime#datetime.datetime" title="datetime.datetime"><code>datetime</code></a>, <a class="reference internal" href="datetime#datetime.date" title="datetime.date"><code>date</code></a> and <a class="reference internal" href="datetime#datetime.time" title="datetime.time"><code>time</code></a> pickled by Python 2.</p> <p>If <em>buffers</em> is None (the default), then all data necessary for deserialization must be contained in the pickle stream. This means that the <em>buffer_callback</em> argument was None when a <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> was instantiated (or when <a class="reference internal" href="#pickle.dump" title="pickle.dump"><code>dump()</code></a> or <a class="reference internal" href="#pickle.dumps" title="pickle.dumps"><code>dumps()</code></a> was called).</p> <p>If <em>buffers</em> is not None, it should be an iterable of buffer-enabled objects that is consumed each time the pickle stream references an <a class="reference internal" href="#pickle-oob"><span class="std std-ref">out-of-band</span></a> buffer view. Such buffers have been given in order to the <em>buffer_callback</em> of a Pickler object.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.8: </span>The <em>buffers</em> argument was added.</p> </div> <dl class="py method"> <dt class="sig sig-object py" id="pickle.Unpickler.load"> +<code>load()</code> </dt> <dd> +<p>Read the pickled representation of an object from the open file object given in the constructor, and return the reconstituted object hierarchy specified therein. Bytes past the pickled representation of the object are ignored.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="pickle.Unpickler.persistent_load"> +<code>persistent_load(pid)</code> </dt> <dd> +<p>Raise an <a class="reference internal" href="#pickle.UnpicklingError" title="pickle.UnpicklingError"><code>UnpicklingError</code></a> by default.</p> <p>If defined, <a class="reference internal" href="#pickle.Unpickler.persistent_load" title="pickle.Unpickler.persistent_load"><code>persistent_load()</code></a> should return the object specified by the persistent ID <em>pid</em>. If an invalid persistent ID is encountered, an <a class="reference internal" href="#pickle.UnpicklingError" title="pickle.UnpicklingError"><code>UnpicklingError</code></a> should be raised.</p> <p>See <a class="reference internal" href="#pickle-persistent"><span class="std std-ref">Persistence of External Objects</span></a> for details and examples of uses.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="pickle.Unpickler.find_class"> +<code>find_class(module, name)</code> </dt> <dd> +<p>Import <em>module</em> if necessary and return the object called <em>name</em> from it, where the <em>module</em> and <em>name</em> arguments are <a class="reference internal" href="stdtypes#str" title="str"><code>str</code></a> objects. Note, unlike its name suggests, <a class="reference internal" href="#pickle.Unpickler.find_class" title="pickle.Unpickler.find_class"><code>find_class()</code></a> is also used for finding functions.</p> <p>Subclasses may override this to gain control over what type of objects and how they can be loaded, potentially reducing security risks. Refer to <a class="reference internal" href="#pickle-restrict"><span class="std std-ref">Restricting Globals</span></a> for details.</p> <p class="audit-hook">Raises an <a class="reference internal" href="sys#auditing"><span class="std std-ref">auditing event</span></a> <code>pickle.find_class</code> with arguments <code>module</code>, <code>name</code>.</p> </dd> +</dl> </dd> +</dl> <dl class="py class"> <dt class="sig sig-object py" id="pickle.PickleBuffer"> +<code>class pickle.PickleBuffer(buffer)</code> </dt> <dd> +<p>A wrapper for a buffer representing picklable data. <em>buffer</em> must be a <a class="reference internal" href="../c-api/buffer#bufferobjects"><span class="std std-ref">buffer-providing</span></a> object, such as a <a class="reference internal" href="../glossary#term-bytes-like-object"><span class="xref std std-term">bytes-like object</span></a> or a N-dimensional array.</p> <p><a class="reference internal" href="#pickle.PickleBuffer" title="pickle.PickleBuffer"><code>PickleBuffer</code></a> is itself a buffer provider, therefore it is possible to pass it to other APIs expecting a buffer-providing object, such as <a class="reference internal" href="stdtypes#memoryview" title="memoryview"><code>memoryview</code></a>.</p> <p><a class="reference internal" href="#pickle.PickleBuffer" title="pickle.PickleBuffer"><code>PickleBuffer</code></a> objects can only be serialized using pickle protocol 5 or higher. They are eligible for <a class="reference internal" href="#pickle-oob"><span class="std std-ref">out-of-band serialization</span></a>.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> <dl class="py method"> <dt class="sig sig-object py" id="pickle.PickleBuffer.raw"> +<code>raw()</code> </dt> <dd> +<p>Return a <a class="reference internal" href="stdtypes#memoryview" title="memoryview"><code>memoryview</code></a> of the memory area underlying this buffer. The returned object is a one-dimensional, C-contiguous memoryview with format <code>B</code> (unsigned bytes). <a class="reference internal" href="exceptions#BufferError" title="BufferError"><code>BufferError</code></a> is raised if the buffer is neither C- nor Fortran-contiguous.</p> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="pickle.PickleBuffer.release"> +<code>release()</code> </dt> <dd> +<p>Release the underlying buffer exposed by the PickleBuffer object.</p> </dd> +</dl> </dd> +</dl> </section> <section id="what-can-be-pickled-and-unpickled"> <span id="pickle-picklable"></span><h2>What can be pickled and unpickled?</h2> <p>The following types can be pickled:</p> <ul class="simple"> <li>built-in constants (<code>None</code>, <code>True</code>, <code>False</code>, <code>Ellipsis</code>, and <code>NotImplemented</code>);</li> <li>integers, floating-point numbers, complex numbers;</li> <li>strings, bytes, bytearrays;</li> <li>tuples, lists, sets, and dictionaries containing only picklable objects;</li> <li>functions (built-in and user-defined) accessible from the top level of a module (using <a class="reference internal" href="../reference/compound_stmts#def"><code>def</code></a>, not <a class="reference internal" href="../reference/expressions#lambda"><code>lambda</code></a>);</li> <li>classes accessible from the top level of a module;</li> <li>instances of such classes whose the result of calling <code>__getstate__()</code> is picklable (see section <a class="reference internal" href="#pickle-inst"><span class="std std-ref">Pickling Class Instances</span></a> for details).</li> </ul> <p>Attempts to pickle unpicklable objects will raise the <a class="reference internal" href="#pickle.PicklingError" title="pickle.PicklingError"><code>PicklingError</code></a> exception; when this happens, an unspecified number of bytes may have already been written to the underlying file. Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a <a class="reference internal" href="exceptions#RecursionError" title="RecursionError"><code>RecursionError</code></a> will be raised in this case. You can carefully raise this limit with <a class="reference internal" href="sys#sys.setrecursionlimit" title="sys.setrecursionlimit"><code>sys.setrecursionlimit()</code></a>.</p> <p>Note that functions (built-in and user-defined) are pickled by fully <a class="reference internal" href="../glossary#term-qualified-name"><span class="xref std std-term">qualified name</span></a>, not by value. <a class="footnote-reference brackets" href="#id8" id="id3">2</a> This means that only the function name is pickled, along with the name of the containing module and classes. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. <a class="footnote-reference brackets" href="#id9" id="id4">3</a></p> <p>Similarly, classes are pickled by fully qualified name, so the same restrictions in the unpickling environment apply. Note that none of the class’s code or data is pickled, so in the following example the class attribute <code>attr</code> is not restored in the unpickling environment:</p> <pre data-language="python">class Foo: + attr = 'A class attribute' + +picklestring = pickle.dumps(Foo) +</pre> <p>These restrictions are why picklable functions and classes must be defined at the top level of a module.</p> <p>Similarly, when class instances are pickled, their class’s code and data are not pickled along with them. Only the instance data are pickled. This is done on purpose, so you can fix bugs in a class or add methods to the class and still load objects that were created with an earlier version of the class. If you plan to have long-lived objects that will see many versions of a class, it may be worthwhile to put a version number in the objects so that suitable conversions can be made by the class’s <code>__setstate__()</code> method.</p> </section> <section id="pickling-class-instances"> <span id="pickle-inst"></span><h2>Pickling Class Instances</h2> <p>In this section, we describe the general mechanisms available to you to define, customize, and control how class instances are pickled and unpickled.</p> <p>In most cases, no additional code is needed to make instances picklable. By default, pickle will retrieve the class and the attributes of an instance via introspection. When a class instance is unpickled, its <code>__init__()</code> method is usually <em>not</em> invoked. The default behaviour first creates an uninitialized instance and then restores the saved attributes. The following code shows an implementation of this behaviour:</p> <pre data-language="python">def save(obj): + return (obj.__class__, obj.__dict__) + +def restore(cls, attributes): + obj = cls.__new__(cls) + obj.__dict__.update(attributes) + return obj +</pre> <p>Classes can alter the default behaviour by providing one or several special methods:</p> <dl class="py method"> <dt class="sig sig-object py" id="object.__getnewargs_ex__"> +<code>object.__getnewargs_ex__()</code> </dt> <dd> +<p>In protocols 2 and newer, classes that implements the <a class="reference internal" href="#object.__getnewargs_ex__" title="object.__getnewargs_ex__"><code>__getnewargs_ex__()</code></a> method can dictate the values passed to the <a class="reference internal" href="../reference/datamodel#object.__new__" title="object.__new__"><code>__new__()</code></a> method upon unpickling. The method must return a pair <code>(args, kwargs)</code> where <em>args</em> is a tuple of positional arguments and <em>kwargs</em> a dictionary of named arguments for constructing the object. Those will be passed to the <a class="reference internal" href="../reference/datamodel#object.__new__" title="object.__new__"><code>__new__()</code></a> method upon unpickling.</p> <p>You should implement this method if the <a class="reference internal" href="../reference/datamodel#object.__new__" title="object.__new__"><code>__new__()</code></a> method of your class requires keyword-only arguments. Otherwise, it is recommended for compatibility to implement <a class="reference internal" href="#object.__getnewargs__" title="object.__getnewargs__"><code>__getnewargs__()</code></a>.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.6: </span><a class="reference internal" href="#object.__getnewargs_ex__" title="object.__getnewargs_ex__"><code>__getnewargs_ex__()</code></a> is now used in protocols 2 and 3.</p> </div> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="object.__getnewargs__"> +<code>object.__getnewargs__()</code> </dt> <dd> +<p>This method serves a similar purpose as <a class="reference internal" href="#object.__getnewargs_ex__" title="object.__getnewargs_ex__"><code>__getnewargs_ex__()</code></a>, but supports only positional arguments. It must return a tuple of arguments <code>args</code> which will be passed to the <a class="reference internal" href="../reference/datamodel#object.__new__" title="object.__new__"><code>__new__()</code></a> method upon unpickling.</p> <p><a class="reference internal" href="#object.__getnewargs__" title="object.__getnewargs__"><code>__getnewargs__()</code></a> will not be called if <a class="reference internal" href="#object.__getnewargs_ex__" title="object.__getnewargs_ex__"><code>__getnewargs_ex__()</code></a> is defined.</p> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.6: </span>Before Python 3.6, <a class="reference internal" href="#object.__getnewargs__" title="object.__getnewargs__"><code>__getnewargs__()</code></a> was called instead of <a class="reference internal" href="#object.__getnewargs_ex__" title="object.__getnewargs_ex__"><code>__getnewargs_ex__()</code></a> in protocols 2 and 3.</p> </div> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="object.__getstate__"> +<code>object.__getstate__()</code> </dt> <dd> +<p>Classes can further influence how their instances are pickled by overriding the method <a class="reference internal" href="#object.__getstate__" title="object.__getstate__"><code>__getstate__()</code></a>. It is called and the returned object is pickled as the contents for the instance, instead of a default state. There are several cases:</p> <ul class="simple"> <li>For a class that has no instance <a class="reference internal" href="stdtypes#object.__dict__" title="object.__dict__"><code>__dict__</code></a> and no <a class="reference internal" href="../reference/datamodel#object.__slots__" title="object.__slots__"><code>__slots__</code></a>, the default state is <code>None</code>.</li> <li>For a class that has an instance <a class="reference internal" href="stdtypes#object.__dict__" title="object.__dict__"><code>__dict__</code></a> and no <a class="reference internal" href="../reference/datamodel#object.__slots__" title="object.__slots__"><code>__slots__</code></a>, the default state is <code>self.__dict__</code>.</li> <li>For a class that has an instance <a class="reference internal" href="stdtypes#object.__dict__" title="object.__dict__"><code>__dict__</code></a> and <a class="reference internal" href="../reference/datamodel#object.__slots__" title="object.__slots__"><code>__slots__</code></a>, the default state is a tuple consisting of two dictionaries: <code>self.__dict__</code>, and a dictionary mapping slot names to slot values. Only slots that have a value are included in the latter.</li> <li>For a class that has <a class="reference internal" href="../reference/datamodel#object.__slots__" title="object.__slots__"><code>__slots__</code></a> and no instance <a class="reference internal" href="stdtypes#object.__dict__" title="object.__dict__"><code>__dict__</code></a>, the default state is a tuple whose first item is <code>None</code> and whose second item is a dictionary mapping slot names to slot values described in the previous bullet.</li> </ul> <div class="versionchanged"> <p><span class="versionmodified changed">Changed in version 3.11: </span>Added the default implementation of the <code>__getstate__()</code> method in the <a class="reference internal" href="functions#object" title="object"><code>object</code></a> class.</p> </div> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="object.__setstate__"> +<code>object.__setstate__(state)</code> </dt> <dd> +<p>Upon unpickling, if the class defines <a class="reference internal" href="#object.__setstate__" title="object.__setstate__"><code>__setstate__()</code></a>, it is called with the unpickled state. In that case, there is no requirement for the state object to be a dictionary. Otherwise, the pickled state must be a dictionary and its items are assigned to the new instance’s dictionary.</p> <div class="admonition note"> <p class="admonition-title">Note</p> <p>If <a class="reference internal" href="#object.__getstate__" title="object.__getstate__"><code>__getstate__()</code></a> returns a false value, the <a class="reference internal" href="#object.__setstate__" title="object.__setstate__"><code>__setstate__()</code></a> method will not be called upon unpickling.</p> </div> </dd> +</dl> <p>Refer to the section <a class="reference internal" href="#pickle-state"><span class="std std-ref">Handling Stateful Objects</span></a> for more information about how to use the methods <code>__getstate__()</code> and <code>__setstate__()</code>.</p> <div class="admonition note"> <p class="admonition-title">Note</p> <p>At unpickling time, some methods like <code>__getattr__()</code>, <code>__getattribute__()</code>, or <code>__setattr__()</code> may be called upon the instance. In case those methods rely on some internal invariant being true, the type should implement <code>__new__()</code> to establish such an invariant, as <code>__init__()</code> is not called when unpickling an instance.</p> </div> <p id="index-5">As we shall see, pickle does not use directly the methods described above. In fact, these methods are part of the copy protocol which implements the <code>__reduce__()</code> special method. The copy protocol provides a unified interface for retrieving the data necessary for pickling and copying objects. <a class="footnote-reference brackets" href="#id10" id="id5">4</a></p> <p>Although powerful, implementing <code>__reduce__()</code> directly in your classes is error prone. For this reason, class designers should use the high-level interface (i.e., <code>__getnewargs_ex__()</code>, <code>__getstate__()</code> and <code>__setstate__()</code>) whenever possible. We will show, however, cases where using <code>__reduce__()</code> is the only option or leads to more efficient pickling or both.</p> <dl class="py method"> <dt class="sig sig-object py" id="object.__reduce__"> +<code>object.__reduce__()</code> </dt> <dd> +<p>The interface is currently defined as follows. The <a class="reference internal" href="#object.__reduce__" title="object.__reduce__"><code>__reduce__()</code></a> method takes no argument and shall return either a string or preferably a tuple (the returned object is often referred to as the “reduce value”).</p> <p>If a string is returned, the string should be interpreted as the name of a global variable. It should be the object’s local name relative to its module; the pickle module searches the module namespace to determine the object’s module. This behaviour is typically useful for singletons.</p> <p>When a tuple is returned, it must be between two and six items long. Optional items can either be omitted, or <code>None</code> can be provided as their value. The semantics of each item are in order:</p> <ul> <li>A callable object that will be called to create the initial version of the object.</li> <li>A tuple of arguments for the callable object. An empty tuple must be given if the callable does not accept any argument.</li> <li>Optionally, the object’s state, which will be passed to the object’s <a class="reference internal" href="#object.__setstate__" title="object.__setstate__"><code>__setstate__()</code></a> method as previously described. If the object has no such method then, the value must be a dictionary and it will be added to the object’s <a class="reference internal" href="stdtypes#object.__dict__" title="object.__dict__"><code>__dict__</code></a> attribute.</li> <li>Optionally, an iterator (and not a sequence) yielding successive items. These items will be appended to the object either using <code>obj.append(item)</code> or, in batch, using <code>obj.extend(list_of_items)</code>. This is primarily used for list subclasses, but may be used by other classes as long as they have <code>append()</code> and <code>extend()</code> methods with the appropriate signature. (Whether <code>append()</code> or <code>extend()</code> is used depends on which pickle protocol version is used as well as the number of items to append, so both must be supported.)</li> <li>Optionally, an iterator (not a sequence) yielding successive key-value pairs. These items will be stored to the object using <code>obj[key] = +value</code>. This is primarily used for dictionary subclasses, but may be used by other classes as long as they implement <a class="reference internal" href="../reference/datamodel#object.__setitem__" title="object.__setitem__"><code>__setitem__()</code></a>.</li> <li> +<p>Optionally, a callable with a <code>(obj, state)</code> signature. This callable allows the user to programmatically control the state-updating behavior of a specific object, instead of using <code>obj</code>’s static <a class="reference internal" href="#object.__setstate__" title="object.__setstate__"><code>__setstate__()</code></a> method. If not <code>None</code>, this callable will have priority over <code>obj</code>’s <a class="reference internal" href="#object.__setstate__" title="object.__setstate__"><code>__setstate__()</code></a>.</p> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8: </span>The optional sixth tuple item, <code>(obj, state)</code>, was added.</p> </div> </li> </ul> </dd> +</dl> <dl class="py method"> <dt class="sig sig-object py" id="object.__reduce_ex__"> +<code>object.__reduce_ex__(protocol)</code> </dt> <dd> +<p>Alternatively, a <a class="reference internal" href="#object.__reduce_ex__" title="object.__reduce_ex__"><code>__reduce_ex__()</code></a> method may be defined. The only difference is this method should take a single integer argument, the protocol version. When defined, pickle will prefer it over the <a class="reference internal" href="#object.__reduce__" title="object.__reduce__"><code>__reduce__()</code></a> method. In addition, <a class="reference internal" href="#object.__reduce__" title="object.__reduce__"><code>__reduce__()</code></a> automatically becomes a synonym for the extended version. The main use for this method is to provide backwards-compatible reduce values for older Python releases.</p> </dd> +</dl> <section id="persistence-of-external-objects"> <span id="pickle-persistent"></span><h3>Persistence of External Objects</h3> <p id="index-6">For the benefit of object persistence, the <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module supports the notion of a reference to an object outside the pickled data stream. Such objects are referenced by a persistent ID, which should be either a string of alphanumeric characters (for protocol 0) <a class="footnote-reference brackets" href="#id11" id="id6">5</a> or just an arbitrary object (for any newer protocol).</p> <p>The resolution of such persistent IDs is not defined by the <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module; it will delegate this resolution to the user-defined methods on the pickler and unpickler, <a class="reference internal" href="#pickle.Pickler.persistent_id" title="pickle.Pickler.persistent_id"><code>persistent_id()</code></a> and <a class="reference internal" href="#pickle.Unpickler.persistent_load" title="pickle.Unpickler.persistent_load"><code>persistent_load()</code></a> respectively.</p> <p>To pickle objects that have an external persistent ID, the pickler must have a custom <a class="reference internal" href="#pickle.Pickler.persistent_id" title="pickle.Pickler.persistent_id"><code>persistent_id()</code></a> method that takes an object as an argument and returns either <code>None</code> or the persistent ID for that object. When <code>None</code> is returned, the pickler simply pickles the object as normal. When a persistent ID string is returned, the pickler will pickle that object, along with a marker so that the unpickler will recognize it as a persistent ID.</p> <p>To unpickle external objects, the unpickler must have a custom <a class="reference internal" href="#pickle.Unpickler.persistent_load" title="pickle.Unpickler.persistent_load"><code>persistent_load()</code></a> method that takes a persistent ID object and returns the referenced object.</p> <p>Here is a comprehensive example presenting how persistent ID can be used to pickle external objects by reference.</p> <pre data-language="python"># Simple example presenting how persistent ID can be used to pickle +# external objects by reference. + +import pickle +import sqlite3 +from collections import namedtuple + +# Simple class representing a record in our database. +MemoRecord = namedtuple("MemoRecord", "key, task") + +class DBPickler(pickle.Pickler): + + def persistent_id(self, obj): + # Instead of pickling MemoRecord as a regular class instance, we emit a + # persistent ID. + if isinstance(obj, MemoRecord): + # Here, our persistent ID is simply a tuple, containing a tag and a + # key, which refers to a specific record in the database. + return ("MemoRecord", obj.key) + else: + # If obj does not have a persistent ID, return None. This means obj + # needs to be pickled as usual. + return None + + +class DBUnpickler(pickle.Unpickler): + + def __init__(self, file, connection): + super().__init__(file) + self.connection = connection + + def persistent_load(self, pid): + # This method is invoked whenever a persistent ID is encountered. + # Here, pid is the tuple returned by DBPickler. + cursor = self.connection.cursor() + type_tag, key_id = pid + if type_tag == "MemoRecord": + # Fetch the referenced record from the database and return it. + cursor.execute("SELECT * FROM memos WHERE key=?", (str(key_id),)) + key, task = cursor.fetchone() + return MemoRecord(key, task) + else: + # Always raises an error if you cannot return the correct object. + # Otherwise, the unpickler will think None is the object referenced + # by the persistent ID. + raise pickle.UnpicklingError("unsupported persistent object") + + +def main(): + import io + import pprint + + # Initialize and populate our database. + conn = sqlite3.connect(":memory:") + cursor = conn.cursor() + cursor.execute("CREATE TABLE memos(key INTEGER PRIMARY KEY, task TEXT)") + tasks = ( + 'give food to fish', + 'prepare group meeting', + 'fight with a zebra', + ) + for task in tasks: + cursor.execute("INSERT INTO memos VALUES(NULL, ?)", (task,)) + + # Fetch the records to be pickled. + cursor.execute("SELECT * FROM memos") + memos = [MemoRecord(key, task) for key, task in cursor] + # Save the records using our custom DBPickler. + file = io.BytesIO() + DBPickler(file).dump(memos) + + print("Pickled records:") + pprint.pprint(memos) + + # Update a record, just for good measure. + cursor.execute("UPDATE memos SET task='learn italian' WHERE key=1") + + # Load the records from the pickle data stream. + file.seek(0) + memos = DBUnpickler(file, conn).load() + + print("Unpickled records:") + pprint.pprint(memos) + + +if __name__ == '__main__': + main() +</pre> </section> <section id="dispatch-tables"> <span id="pickle-dispatch"></span><h3>Dispatch Tables</h3> <p>If one wants to customize pickling of some classes without disturbing any other code which depends on pickling, then one can create a pickler with a private dispatch table.</p> <p>The global dispatch table managed by the <a class="reference internal" href="copyreg#module-copyreg" title="copyreg: Register pickle support functions."><code>copyreg</code></a> module is available as <code>copyreg.dispatch_table</code>. Therefore, one may choose to use a modified copy of <code>copyreg.dispatch_table</code> as a private dispatch table.</p> <p>For example</p> <pre data-language="python">f = io.BytesIO() +p = pickle.Pickler(f) +p.dispatch_table = copyreg.dispatch_table.copy() +p.dispatch_table[SomeClass] = reduce_SomeClass +</pre> <p>creates an instance of <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>pickle.Pickler</code></a> with a private dispatch table which handles the <code>SomeClass</code> class specially. Alternatively, the code</p> <pre data-language="python">class MyPickler(pickle.Pickler): + dispatch_table = copyreg.dispatch_table.copy() + dispatch_table[SomeClass] = reduce_SomeClass +f = io.BytesIO() +p = MyPickler(f) +</pre> <p>does the same but all instances of <code>MyPickler</code> will by default share the private dispatch table. On the other hand, the code</p> <pre data-language="python">copyreg.pickle(SomeClass, reduce_SomeClass) +f = io.BytesIO() +p = pickle.Pickler(f) +</pre> <p>modifies the global dispatch table shared by all users of the <a class="reference internal" href="copyreg#module-copyreg" title="copyreg: Register pickle support functions."><code>copyreg</code></a> module.</p> </section> <section id="handling-stateful-objects"> <span id="pickle-state"></span><h3>Handling Stateful Objects</h3> <p id="index-7">Here’s an example that shows how to modify pickling behavior for a class. The <code>TextReader</code> class opens a text file, and returns the line number and line contents each time its <code>readline()</code> method is called. If a <code>TextReader</code> instance is pickled, all attributes <em>except</em> the file object member are saved. When the instance is unpickled, the file is reopened, and reading resumes from the last location. The <code>__setstate__()</code> and <code>__getstate__()</code> methods are used to implement this behavior.</p> <pre data-language="python">class TextReader: + """Print and number lines in a text file.""" + + def __init__(self, filename): + self.filename = filename + self.file = open(filename) + self.lineno = 0 + + def readline(self): + self.lineno += 1 + line = self.file.readline() + if not line: + return None + if line.endswith('\n'): + line = line[:-1] + return "%i: %s" % (self.lineno, line) + + def __getstate__(self): + # Copy the object's state from self.__dict__ which contains + # all our instance attributes. Always use the dict.copy() + # method to avoid modifying the original state. + state = self.__dict__.copy() + # Remove the unpicklable entries. + del state['file'] + return state + + def __setstate__(self, state): + # Restore instance attributes (i.e., filename and lineno). + self.__dict__.update(state) + # Restore the previously opened file's state. To do so, we need to + # reopen it and read from it until the line count is restored. + file = open(self.filename) + for _ in range(self.lineno): + file.readline() + # Finally, save the file. + self.file = file +</pre> <p>A sample usage might be something like this:</p> <pre data-language="python">>>> reader = TextReader("hello.txt") +>>> reader.readline() +'1: Hello world!' +>>> reader.readline() +'2: I am line number two.' +>>> new_reader = pickle.loads(pickle.dumps(reader)) +>>> new_reader.readline() +'3: Goodbye!' +</pre> </section> </section> <section id="custom-reduction-for-types-functions-and-other-objects"> <span id="reducer-override"></span><h2>Custom Reduction for Types, Functions, and Other Objects</h2> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> <p>Sometimes, <a class="reference internal" href="#pickle.Pickler.dispatch_table" title="pickle.Pickler.dispatch_table"><code>dispatch_table</code></a> may not be flexible enough. In particular we may want to customize pickling based on another criterion than the object’s type, or we may want to customize the pickling of functions and classes.</p> <p>For those cases, it is possible to subclass from the <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> class and implement a <a class="reference internal" href="#pickle.Pickler.reducer_override" title="pickle.Pickler.reducer_override"><code>reducer_override()</code></a> method. This method can return an arbitrary reduction tuple (see <code>__reduce__()</code>). It can alternatively return <code>NotImplemented</code> to fallback to the traditional behavior.</p> <p>If both the <a class="reference internal" href="#pickle.Pickler.dispatch_table" title="pickle.Pickler.dispatch_table"><code>dispatch_table</code></a> and <a class="reference internal" href="#pickle.Pickler.reducer_override" title="pickle.Pickler.reducer_override"><code>reducer_override()</code></a> are defined, then <a class="reference internal" href="#pickle.Pickler.reducer_override" title="pickle.Pickler.reducer_override"><code>reducer_override()</code></a> method takes priority.</p> <div class="admonition note"> <p class="admonition-title">Note</p> <p>For performance reasons, <a class="reference internal" href="#pickle.Pickler.reducer_override" title="pickle.Pickler.reducer_override"><code>reducer_override()</code></a> may not be called for the following objects: <code>None</code>, <code>True</code>, <code>False</code>, and exact instances of <a class="reference internal" href="functions#int" title="int"><code>int</code></a>, <a class="reference internal" href="functions#float" title="float"><code>float</code></a>, <a class="reference internal" href="stdtypes#bytes" title="bytes"><code>bytes</code></a>, <a class="reference internal" href="stdtypes#str" title="str"><code>str</code></a>, <a class="reference internal" href="stdtypes#dict" title="dict"><code>dict</code></a>, <a class="reference internal" href="stdtypes#set" title="set"><code>set</code></a>, <a class="reference internal" href="stdtypes#frozenset" title="frozenset"><code>frozenset</code></a>, <a class="reference internal" href="stdtypes#list" title="list"><code>list</code></a> and <a class="reference internal" href="stdtypes#tuple" title="tuple"><code>tuple</code></a>.</p> </div> <p>Here is a simple example where we allow pickling and reconstructing a given class:</p> <pre data-language="python">import io +import pickle + +class MyClass: + my_attribute = 1 + +class MyPickler(pickle.Pickler): + def reducer_override(self, obj): + """Custom reducer for MyClass.""" + if getattr(obj, "__name__", None) == "MyClass": + return type, (obj.__name__, obj.__bases__, + {'my_attribute': obj.my_attribute}) + else: + # For any other object, fallback to usual reduction + return NotImplemented + +f = io.BytesIO() +p = MyPickler(f) +p.dump(MyClass) + +del MyClass + +unpickled_class = pickle.loads(f.getvalue()) + +assert isinstance(unpickled_class, type) +assert unpickled_class.__name__ == "MyClass" +assert unpickled_class.my_attribute == 1 +</pre> </section> <section id="out-of-band-buffers"> <span id="pickle-oob"></span><h2>Out-of-band Buffers</h2> <div class="versionadded"> <p><span class="versionmodified added">New in version 3.8.</span></p> </div> <p>In some contexts, the <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module is used to transfer massive amounts of data. Therefore, it can be important to minimize the number of memory copies, to preserve performance and resource consumption. However, normal operation of the <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module, as it transforms a graph-like structure of objects into a sequential stream of bytes, intrinsically involves copying data to and from the pickle stream.</p> <p>This constraint can be eschewed if both the <em>provider</em> (the implementation of the object types to be transferred) and the <em>consumer</em> (the implementation of the communications system) support the out-of-band transfer facilities provided by pickle protocol 5 and higher.</p> <section id="provider-api"> <h3>Provider API</h3> <p>The large data objects to be pickled must implement a <code>__reduce_ex__()</code> method specialized for protocol 5 and higher, which returns a <a class="reference internal" href="#pickle.PickleBuffer" title="pickle.PickleBuffer"><code>PickleBuffer</code></a> instance (instead of e.g. a <a class="reference internal" href="stdtypes#bytes" title="bytes"><code>bytes</code></a> object) for any large data.</p> <p>A <a class="reference internal" href="#pickle.PickleBuffer" title="pickle.PickleBuffer"><code>PickleBuffer</code></a> object <em>signals</em> that the underlying buffer is eligible for out-of-band data transfer. Those objects remain compatible with normal usage of the <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module. However, consumers can also opt-in to tell <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> that they will handle those buffers by themselves.</p> </section> <section id="consumer-api"> <h3>Consumer API</h3> <p>A communications system can enable custom handling of the <a class="reference internal" href="#pickle.PickleBuffer" title="pickle.PickleBuffer"><code>PickleBuffer</code></a> objects generated when serializing an object graph.</p> <p>On the sending side, it needs to pass a <em>buffer_callback</em> argument to <a class="reference internal" href="#pickle.Pickler" title="pickle.Pickler"><code>Pickler</code></a> (or to the <a class="reference internal" href="#pickle.dump" title="pickle.dump"><code>dump()</code></a> or <a class="reference internal" href="#pickle.dumps" title="pickle.dumps"><code>dumps()</code></a> function), which will be called with each <a class="reference internal" href="#pickle.PickleBuffer" title="pickle.PickleBuffer"><code>PickleBuffer</code></a> generated while pickling the object graph. Buffers accumulated by the <em>buffer_callback</em> will not see their data copied into the pickle stream, only a cheap marker will be inserted.</p> <p>On the receiving side, it needs to pass a <em>buffers</em> argument to <a class="reference internal" href="#pickle.Unpickler" title="pickle.Unpickler"><code>Unpickler</code></a> (or to the <a class="reference internal" href="#pickle.load" title="pickle.load"><code>load()</code></a> or <a class="reference internal" href="#pickle.loads" title="pickle.loads"><code>loads()</code></a> function), which is an iterable of the buffers which were passed to <em>buffer_callback</em>. That iterable should produce buffers in the same order as they were passed to <em>buffer_callback</em>. Those buffers will provide the data expected by the reconstructors of the objects whose pickling produced the original <a class="reference internal" href="#pickle.PickleBuffer" title="pickle.PickleBuffer"><code>PickleBuffer</code></a> objects.</p> <p>Between the sending side and the receiving side, the communications system is free to implement its own transfer mechanism for out-of-band buffers. Potential optimizations include the use of shared memory or datatype-dependent compression.</p> </section> <section id="example"> <h3>Example</h3> <p>Here is a trivial example where we implement a <a class="reference internal" href="stdtypes#bytearray" title="bytearray"><code>bytearray</code></a> subclass able to participate in out-of-band buffer pickling:</p> <pre data-language="python">class ZeroCopyByteArray(bytearray): + + def __reduce_ex__(self, protocol): + if protocol >= 5: + return type(self)._reconstruct, (PickleBuffer(self),), None + else: + # PickleBuffer is forbidden with pickle protocols <= 4. + return type(self)._reconstruct, (bytearray(self),) + + @classmethod + def _reconstruct(cls, obj): + with memoryview(obj) as m: + # Get a handle over the original buffer object + obj = m.obj + if type(obj) is cls: + # Original buffer object is a ZeroCopyByteArray, return it + # as-is. + return obj + else: + return cls(obj) +</pre> <p>The reconstructor (the <code>_reconstruct</code> class method) returns the buffer’s providing object if it has the right type. This is an easy way to simulate zero-copy behaviour on this toy example.</p> <p>On the consumer side, we can pickle those objects the usual way, which when unserialized will give us a copy of the original object:</p> <pre data-language="python">b = ZeroCopyByteArray(b"abc") +data = pickle.dumps(b, protocol=5) +new_b = pickle.loads(data) +print(b == new_b) # True +print(b is new_b) # False: a copy was made +</pre> <p>But if we pass a <em>buffer_callback</em> and then give back the accumulated buffers when unserializing, we are able to get back the original object:</p> <pre data-language="python">b = ZeroCopyByteArray(b"abc") +buffers = [] +data = pickle.dumps(b, protocol=5, buffer_callback=buffers.append) +new_b = pickle.loads(data, buffers=buffers) +print(b == new_b) # True +print(b is new_b) # True: no copy was made +</pre> <p>This example is limited by the fact that <a class="reference internal" href="stdtypes#bytearray" title="bytearray"><code>bytearray</code></a> allocates its own memory: you cannot create a <a class="reference internal" href="stdtypes#bytearray" title="bytearray"><code>bytearray</code></a> instance that is backed by another object’s memory. However, third-party datatypes such as NumPy arrays do not have this limitation, and allow use of zero-copy pickling (or making as few copies as possible) when transferring between distinct processes or systems.</p> <div class="admonition seealso"> <p class="admonition-title">See also</p> <p><span class="target" id="index-8"></span><a class="pep reference external" href="https://peps.python.org/pep-0574/"><strong>PEP 574</strong></a> – Pickle protocol 5 with out-of-band data</p> </div> </section> </section> <section id="restricting-globals"> <span id="pickle-restrict"></span><h2>Restricting Globals</h2> <p id="index-9">By default, unpickling will import any class or function that it finds in the pickle data. For many applications, this behaviour is unacceptable as it permits the unpickler to import and invoke arbitrary code. Just consider what this hand-crafted pickle data stream does when loaded:</p> <pre data-language="python">>>> import pickle +>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.") +hello world +0 +</pre> <p>In this example, the unpickler imports the <a class="reference internal" href="os#os.system" title="os.system"><code>os.system()</code></a> function and then apply the string argument “echo hello world”. Although this example is inoffensive, it is not difficult to imagine one that could damage your system.</p> <p>For this reason, you may want to control what gets unpickled by customizing <a class="reference internal" href="#pickle.Unpickler.find_class" title="pickle.Unpickler.find_class"><code>Unpickler.find_class()</code></a>. Unlike its name suggests, <a class="reference internal" href="#pickle.Unpickler.find_class" title="pickle.Unpickler.find_class"><code>Unpickler.find_class()</code></a> is called whenever a global (i.e., a class or a function) is requested. Thus it is possible to either completely forbid globals or restrict them to a safe subset.</p> <p>Here is an example of an unpickler allowing only few safe classes from the <a class="reference internal" href="builtins#module-builtins" title="builtins: The module that provides the built-in namespace."><code>builtins</code></a> module to be loaded:</p> <pre data-language="python">import builtins +import io +import pickle + +safe_builtins = { + 'range', + 'complex', + 'set', + 'frozenset', + 'slice', +} + +class RestrictedUnpickler(pickle.Unpickler): + + def find_class(self, module, name): + # Only allow safe classes from builtins. + if module == "builtins" and name in safe_builtins: + return getattr(builtins, name) + # Forbid everything else. + raise pickle.UnpicklingError("global '%s.%s' is forbidden" % + (module, name)) + +def restricted_loads(s): + """Helper function analogous to pickle.loads().""" + return RestrictedUnpickler(io.BytesIO(s)).load() +</pre> <p>A sample usage of our unpickler working as intended:</p> <pre data-language="python">>>> restricted_loads(pickle.dumps([1, 2, range(15)])) +[1, 2, range(0, 15)] +>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.") +Traceback (most recent call last): + ... +pickle.UnpicklingError: global 'os.system' is forbidden +>>> restricted_loads(b'cbuiltins\neval\n' +... b'(S\'getattr(__import__("os"), "system")' +... b'("echo hello world")\'\ntR.') +Traceback (most recent call last): + ... +pickle.UnpicklingError: global 'builtins.eval' is forbidden +</pre> <p>As our examples shows, you have to be careful with what you allow to be unpickled. Therefore if security is a concern, you may want to consider alternatives such as the marshalling API in <a class="reference internal" href="xmlrpc.client#module-xmlrpc.client" title="xmlrpc.client: XML-RPC client access."><code>xmlrpc.client</code></a> or third-party solutions.</p> </section> <section id="performance"> <h2>Performance</h2> <p>Recent versions of the pickle protocol (from protocol 2 and upwards) feature efficient binary encodings for several common features and built-in types. Also, the <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a> module has a transparent optimizer written in C.</p> </section> <section id="examples"> <span id="pickle-example"></span><h2>Examples</h2> <p>For the simplest code, use the <a class="reference internal" href="#pickle.dump" title="pickle.dump"><code>dump()</code></a> and <a class="reference internal" href="#pickle.load" title="pickle.load"><code>load()</code></a> functions.</p> <pre data-language="python">import pickle + +# An arbitrary collection of objects supported by pickle. +data = { + 'a': [1, 2.0, 3+4j], + 'b': ("character string", b"byte string"), + 'c': {None, True, False} +} + +with open('data.pickle', 'wb') as f: + # Pickle the 'data' dictionary using the highest protocol available. + pickle.dump(data, f, pickle.HIGHEST_PROTOCOL) +</pre> <p>The following example reads the resulting pickled data.</p> <pre data-language="python">import pickle + +with open('data.pickle', 'rb') as f: + # The protocol version used is detected automatically, so we do not + # have to specify it. + data = pickle.load(f) +</pre> <div class="admonition seealso"> <p class="admonition-title">See also</p> <dl class="simple"> <dt> +<code>Module</code> <a class="reference internal" href="copyreg#module-copyreg" title="copyreg: Register pickle support functions."><code>copyreg</code></a> +</dt> +<dd> +<p>Pickle interface constructor registration for extension types.</p> </dd> <dt> +<code>Module</code> <a class="reference internal" href="pickletools#module-pickletools" title="pickletools: Contains extensive comments about the pickle protocols and pickle-machine opcodes, as well as some useful functions."><code>pickletools</code></a> +</dt> +<dd> +<p>Tools for working with and analyzing pickled data.</p> </dd> <dt> +<code>Module</code> <a class="reference internal" href="shelve#module-shelve" title="shelve: Python object persistence."><code>shelve</code></a> +</dt> +<dd> +<p>Indexed databases of objects; uses <a class="reference internal" href="#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code>pickle</code></a>.</p> </dd> <dt> +<code>Module</code> <a class="reference internal" href="copy#module-copy" title="copy: Shallow and deep copy operations."><code>copy</code></a> +</dt> +<dd> +<p>Shallow and deep object copying.</p> </dd> <dt> +<code>Module</code> <a class="reference internal" href="marshal#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code>marshal</code></a> +</dt> +<dd> +<p>High-performance serialization of built-in types.</p> </dd> </dl> </div> <h4 class="rubric">Footnotes</h4> <dl class="footnote brackets"> <dt class="label" id="id7"> +<code>1</code> </dt> <dd> +<p>Don’t confuse this with the <a class="reference internal" href="marshal#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code>marshal</code></a> module</p> </dd> <dt class="label" id="id8"> +<code>2</code> </dt> <dd> +<p>This is why <a class="reference internal" href="../reference/expressions#lambda"><code>lambda</code></a> functions cannot be pickled: all <code>lambda</code> functions share the same name: <code><lambda></code>.</p> </dd> <dt class="label" id="id9"> +<code>3</code> </dt> <dd> +<p>The exception raised will likely be an <a class="reference internal" href="exceptions#ImportError" title="ImportError"><code>ImportError</code></a> or an <a class="reference internal" href="exceptions#AttributeError" title="AttributeError"><code>AttributeError</code></a> but it could be something else.</p> </dd> <dt class="label" id="id10"> +<code>4</code> </dt> <dd> +<p>The <a class="reference internal" href="copy#module-copy" title="copy: Shallow and deep copy operations."><code>copy</code></a> module uses this protocol for shallow and deep copying operations.</p> </dd> <dt class="label" id="id11"> +<code>5</code> </dt> <dd> +<p>The limitation on alphanumeric characters is due to the fact that persistent IDs in protocol 0 are delimited by the newline character. Therefore if any kind of newline characters occurs in persistent IDs, the resulting pickled data will become unreadable.</p> </dd> </dl> </section> <div class="_attribution"> + <p class="_attribution-p"> + © 2001–2023 Python Software Foundation<br>Licensed under the PSF License.<br> + <a href="https://docs.python.org/3.12/library/pickle.html" class="_attribution-link">https://docs.python.org/3.12/library/pickle.html</a> + </p> +</div> |
