summaryrefslogtreecommitdiff
path: root/devdocs/go/compress%2Fflate%2Findex.html
diff options
context:
space:
mode:
authorCraig Jennings <c@cjennings.net>2024-04-07 13:41:34 -0500
committerCraig Jennings <c@cjennings.net>2024-04-07 13:41:34 -0500
commit754bbf7a25a8dda49b5d08ef0d0443bbf5af0e36 (patch)
treef1190704f78f04a2b0b4c977d20fe96a828377f1 /devdocs/go/compress%2Fflate%2Findex.html
new repository
Diffstat (limited to 'devdocs/go/compress%2Fflate%2Findex.html')
-rw-r--r--devdocs/go/compress%2Fflate%2Findex.html307
1 files changed, 307 insertions, 0 deletions
diff --git a/devdocs/go/compress%2Fflate%2Findex.html b/devdocs/go/compress%2Fflate%2Findex.html
new file mode 100644
index 00000000..88631066
--- /dev/null
+++ b/devdocs/go/compress%2Fflate%2Findex.html
@@ -0,0 +1,307 @@
+<h1> Package flate </h1> <ul id="short-nav">
+<li><code>import "compress/flate"</code></li>
+<li><a href="#pkg-overview" class="overviewLink">Overview</a></li>
+<li><a href="#pkg-index" class="indexLink">Index</a></li>
+<li><a href="#pkg-examples" class="examplesLink">Examples</a></li>
+</ul> <h2 id="pkg-overview">Overview </h2> <p>Package flate implements the DEFLATE compressed data format, described in RFC 1951. The gzip and zlib packages implement access to DEFLATE-based file formats. </p> <h4 id="example__dictionary"> <span class="text">Example (Dictionary)</span>
+</h4> <p>A preset dictionary can be used to improve the compression ratio. The downside to using a dictionary is that the compressor and decompressor must agree in advance what dictionary to use. </p> <p>Code:</p> <pre class="code" data-language="go">// The dictionary is a string of bytes. When compressing some input data,
+// the compressor will attempt to substitute substrings with matches found
+// in the dictionary. As such, the dictionary should only contain substrings
+// that are expected to be found in the actual data stream.
+const dict = `&lt;?xml version="1.0"?&gt;` + `&lt;book&gt;` + `&lt;data&gt;` + `&lt;meta name="` + `" content="`
+
+// The data to compress should (but is not required to) contain frequent
+// substrings that match those in the dictionary.
+const data = `&lt;?xml version="1.0"?&gt;
+&lt;book&gt;
+ &lt;meta name="title" content="The Go Programming Language"/&gt;
+ &lt;meta name="authors" content="Alan Donovan and Brian Kernighan"/&gt;
+ &lt;meta name="published" content="2015-10-26"/&gt;
+ &lt;meta name="isbn" content="978-0134190440"/&gt;
+ &lt;data&gt;...&lt;/data&gt;
+&lt;/book&gt;
+`
+
+var b bytes.Buffer
+
+// Compress the data using the specially crafted dictionary.
+zw, err := flate.NewWriterDict(&amp;b, flate.DefaultCompression, []byte(dict))
+if err != nil {
+ log.Fatal(err)
+}
+if _, err := io.Copy(zw, strings.NewReader(data)); err != nil {
+ log.Fatal(err)
+}
+if err := zw.Close(); err != nil {
+ log.Fatal(err)
+}
+
+// The decompressor must use the same dictionary as the compressor.
+// Otherwise, the input may appear as corrupted.
+fmt.Println("Decompressed output using the dictionary:")
+zr := flate.NewReaderDict(bytes.NewReader(b.Bytes()), []byte(dict))
+if _, err := io.Copy(os.Stdout, zr); err != nil {
+ log.Fatal(err)
+}
+if err := zr.Close(); err != nil {
+ log.Fatal(err)
+}
+
+fmt.Println()
+
+// Substitute all of the bytes in the dictionary with a '#' to visually
+// demonstrate the approximate effectiveness of using a preset dictionary.
+fmt.Println("Substrings matched by the dictionary are marked with #:")
+hashDict := []byte(dict)
+for i := range hashDict {
+ hashDict[i] = '#'
+}
+zr = flate.NewReaderDict(&amp;b, hashDict)
+if _, err := io.Copy(os.Stdout, zr); err != nil {
+ log.Fatal(err)
+}
+if err := zr.Close(); err != nil {
+ log.Fatal(err)
+}
+
+</pre> <p>Output:</p> <pre class="output" data-language="go">Decompressed output using the dictionary:
+&lt;?xml version="1.0"?&gt;
+&lt;book&gt;
+ &lt;meta name="title" content="The Go Programming Language"/&gt;
+ &lt;meta name="authors" content="Alan Donovan and Brian Kernighan"/&gt;
+ &lt;meta name="published" content="2015-10-26"/&gt;
+ &lt;meta name="isbn" content="978-0134190440"/&gt;
+ &lt;data&gt;...&lt;/data&gt;
+&lt;/book&gt;
+
+Substrings matched by the dictionary are marked with #:
+#####################
+######
+ ############title###########The Go Programming Language"/#
+ ############authors###########Alan Donovan and Brian Kernighan"/#
+ ############published###########2015-10-26"/#
+ ############isbn###########978-0134190440"/#
+ ######...&lt;/#####
+&lt;/#####
+</pre> <h4 id="example__reset"> <span class="text">Example (Reset)</span>
+</h4> <p>In performance critical applications, Reset can be used to discard the current compressor or decompressor state and reinitialize them quickly by taking advantage of previously allocated memory. </p> <p>Code:</p> <pre class="code" data-language="go">proverbs := []string{
+ "Don't communicate by sharing memory, share memory by communicating.\n",
+ "Concurrency is not parallelism.\n",
+ "The bigger the interface, the weaker the abstraction.\n",
+ "Documentation is for users.\n",
+}
+
+var r strings.Reader
+var b bytes.Buffer
+buf := make([]byte, 32&lt;&lt;10)
+
+zw, err := flate.NewWriter(nil, flate.DefaultCompression)
+if err != nil {
+ log.Fatal(err)
+}
+zr := flate.NewReader(nil)
+
+for _, s := range proverbs {
+ r.Reset(s)
+ b.Reset()
+
+ // Reset the compressor and encode from some input stream.
+ zw.Reset(&amp;b)
+ if _, err := io.CopyBuffer(zw, &amp;r, buf); err != nil {
+ log.Fatal(err)
+ }
+ if err := zw.Close(); err != nil {
+ log.Fatal(err)
+ }
+
+ // Reset the decompressor and decode to some output stream.
+ if err := zr.(flate.Resetter).Reset(&amp;b, nil); err != nil {
+ log.Fatal(err)
+ }
+ if _, err := io.CopyBuffer(os.Stdout, zr, buf); err != nil {
+ log.Fatal(err)
+ }
+ if err := zr.Close(); err != nil {
+ log.Fatal(err)
+ }
+}
+
+</pre> <p>Output:</p> <pre class="output" data-language="go">Don't communicate by sharing memory, share memory by communicating.
+Concurrency is not parallelism.
+The bigger the interface, the weaker the abstraction.
+Documentation is for users.
+</pre> <h4 id="example__synchronization"> <span class="text">Example (Synchronization)</span>
+</h4> <p>DEFLATE is suitable for transmitting compressed data across the network. </p> <p>Code:</p> <pre class="code" data-language="go">var wg sync.WaitGroup
+defer wg.Wait()
+
+// Use io.Pipe to simulate a network connection.
+// A real network application should take care to properly close the
+// underlying connection.
+rp, wp := io.Pipe()
+
+// Start a goroutine to act as the transmitter.
+wg.Add(1)
+go func() {
+ defer wg.Done()
+
+ zw, err := flate.NewWriter(wp, flate.BestSpeed)
+ if err != nil {
+ log.Fatal(err)
+ }
+
+ b := make([]byte, 256)
+ for _, m := range strings.Fields("A long time ago in a galaxy far, far away...") {
+ // We use a simple framing format where the first byte is the
+ // message length, followed the message itself.
+ b[0] = uint8(copy(b[1:], m))
+
+ if _, err := zw.Write(b[:1+len(m)]); err != nil {
+ log.Fatal(err)
+ }
+
+ // Flush ensures that the receiver can read all data sent so far.
+ if err := zw.Flush(); err != nil {
+ log.Fatal(err)
+ }
+ }
+
+ if err := zw.Close(); err != nil {
+ log.Fatal(err)
+ }
+}()
+
+// Start a goroutine to act as the receiver.
+wg.Add(1)
+go func() {
+ defer wg.Done()
+
+ zr := flate.NewReader(rp)
+
+ b := make([]byte, 256)
+ for {
+ // Read the message length.
+ // This is guaranteed to return for every corresponding
+ // Flush and Close on the transmitter side.
+ if _, err := io.ReadFull(zr, b[:1]); err != nil {
+ if err == io.EOF {
+ break // The transmitter closed the stream
+ }
+ log.Fatal(err)
+ }
+
+ // Read the message content.
+ n := int(b[0])
+ if _, err := io.ReadFull(zr, b[:n]); err != nil {
+ log.Fatal(err)
+ }
+
+ fmt.Printf("Received %d bytes: %s\n", n, b[:n])
+ }
+ fmt.Println()
+
+ if err := zr.Close(); err != nil {
+ log.Fatal(err)
+ }
+}()
+
+</pre> <p>Output:</p> <pre class="output" data-language="go">Received 1 bytes: A
+Received 4 bytes: long
+Received 4 bytes: time
+Received 3 bytes: ago
+Received 2 bytes: in
+Received 1 bytes: a
+Received 6 bytes: galaxy
+Received 4 bytes: far,
+Received 3 bytes: far
+Received 7 bytes: away...
+</pre> <h2 id="pkg-index">Index </h2> <ul id="manual-nav">
+<li><a href="#pkg-constants">Constants</a></li>
+<li><a href="#NewReader">func NewReader(r io.Reader) io.ReadCloser</a></li>
+<li><a href="#NewReaderDict">func NewReaderDict(r io.Reader, dict []byte) io.ReadCloser</a></li>
+<li><a href="#CorruptInputError">type CorruptInputError</a></li>
+<li> <a href="#CorruptInputError.Error">func (e CorruptInputError) Error() string</a>
+</li>
+<li><a href="#InternalError">type InternalError</a></li>
+<li> <a href="#InternalError.Error">func (e InternalError) Error() string</a>
+</li>
+<li><a href="#ReadError">type ReadError</a></li>
+<li> <a href="#ReadError.Error">func (e *ReadError) Error() string</a>
+</li>
+<li><a href="#Reader">type Reader</a></li>
+<li><a href="#Resetter">type Resetter</a></li>
+<li><a href="#WriteError">type WriteError</a></li>
+<li> <a href="#WriteError.Error">func (e *WriteError) Error() string</a>
+</li>
+<li><a href="#Writer">type Writer</a></li>
+<li> <a href="#NewWriter">func NewWriter(w io.Writer, level int) (*Writer, error)</a>
+</li>
+<li> <a href="#NewWriterDict">func NewWriterDict(w io.Writer, level int, dict []byte) (*Writer, error)</a>
+</li>
+<li> <a href="#Writer.Close">func (w *Writer) Close() error</a>
+</li>
+<li> <a href="#Writer.Flush">func (w *Writer) Flush() error</a>
+</li>
+<li> <a href="#Writer.Reset">func (w *Writer) Reset(dst io.Writer)</a>
+</li>
+<li> <a href="#Writer.Write">func (w *Writer) Write(data []byte) (n int, err error)</a>
+</li>
+</ul> <div id="pkg-examples"> <h3>Examples</h3> <dl> <dd><a class="exampleLink" href="#example__dictionary">Package (Dictionary)</a></dd> <dd><a class="exampleLink" href="#example__reset">Package (Reset)</a></dd> <dd><a class="exampleLink" href="#example__synchronization">Package (Synchronization)</a></dd> </dl> </div> <h3>Package files</h3> <p> <span>deflate.go</span> <span>deflatefast.go</span> <span>dict_decoder.go</span> <span>huffman_bit_writer.go</span> <span>huffman_code.go</span> <span>inflate.go</span> <span>token.go</span> </p> <h2 id="pkg-constants">Constants</h2> <pre data-language="go">const (
+ NoCompression = 0
+ BestSpeed = 1
+ BestCompression = 9
+ DefaultCompression = -1
+
+ // HuffmanOnly disables Lempel-Ziv match searching and only performs Huffman
+ // entropy encoding. This mode is useful in compressing data that has
+ // already been compressed with an LZ style algorithm (e.g. Snappy or LZ4)
+ // that lacks an entropy encoder. Compression gains are achieved when
+ // certain bytes in the input stream occur more frequently than others.
+ //
+ // Note that HuffmanOnly produces a compressed output that is
+ // RFC 1951 compliant. That is, any valid DEFLATE decompressor will
+ // continue to be able to decompress this output.
+ HuffmanOnly = -2
+)</pre> <h2 id="NewReader">func <span>NewReader</span> </h2> <pre data-language="go">func NewReader(r io.Reader) io.ReadCloser</pre> <p>NewReader returns a new ReadCloser that can be used to read the uncompressed version of r. If r does not also implement <span>io.ByteReader</span>, the decompressor may read more data than necessary from r. The reader returns <span>io.EOF</span> after the final block in the DEFLATE stream has been encountered. Any trailing data after the final block is ignored. </p>
+<p>The <span>io.ReadCloser</span> returned by NewReader also implements <a href="#Resetter">Resetter</a>. </p>
+<h2 id="NewReaderDict">func <span>NewReaderDict</span> </h2> <pre data-language="go">func NewReaderDict(r io.Reader, dict []byte) io.ReadCloser</pre> <p>NewReaderDict is like <a href="#NewReader">NewReader</a> but initializes the reader with a preset dictionary. The returned <a href="#Reader">Reader</a> behaves as if the uncompressed data stream started with the given dictionary, which has already been read. NewReaderDict is typically used to read data compressed by NewWriterDict. </p>
+<p>The ReadCloser returned by NewReaderDict also implements <a href="#Resetter">Resetter</a>. </p>
+<h2 id="CorruptInputError">type <span>CorruptInputError</span> </h2> <p>A CorruptInputError reports the presence of corrupt input at a given offset. </p>
+<pre data-language="go">type CorruptInputError int64</pre> <h3 id="CorruptInputError.Error">func (CorruptInputError) <span>Error</span> </h3> <pre data-language="go">func (e CorruptInputError) Error() string</pre> <h2 id="InternalError">type <span>InternalError</span> </h2> <p>An InternalError reports an error in the flate code itself. </p>
+<pre data-language="go">type InternalError string</pre> <h3 id="InternalError.Error">func (InternalError) <span>Error</span> </h3> <pre data-language="go">func (e InternalError) Error() string</pre> <h2 id="ReadError">type <span>ReadError</span> </h2> <p>A ReadError reports an error encountered while reading input. </p>
+<p>Deprecated: No longer returned. </p>
+<pre data-language="go">type ReadError struct {
+ Offset int64 // byte offset where error occurred
+ Err error // error returned by underlying Read
+}
+</pre> <h3 id="ReadError.Error">func (*ReadError) <span>Error</span> </h3> <pre data-language="go">func (e *ReadError) Error() string</pre> <h2 id="Reader">type <span>Reader</span> </h2> <p>The actual read interface needed by <a href="#NewReader">NewReader</a>. If the passed in io.Reader does not also have ReadByte, the <a href="#NewReader">NewReader</a> will introduce its own buffering. </p>
+<pre data-language="go">type Reader interface {
+ io.Reader
+ io.ByteReader
+}</pre> <h2 id="Resetter">type <span>Resetter</span> <span title="Added in Go 1.4">1.4</span> </h2> <p>Resetter resets a ReadCloser returned by <a href="#NewReader">NewReader</a> or <a href="#NewReaderDict">NewReaderDict</a> to switch to a new underlying <a href="#Reader">Reader</a>. This permits reusing a ReadCloser instead of allocating a new one. </p>
+<pre data-language="go">type Resetter interface {
+ // Reset discards any buffered data and resets the Resetter as if it was
+ // newly initialized with the given reader.
+ Reset(r io.Reader, dict []byte) error
+}</pre> <h2 id="WriteError">type <span>WriteError</span> </h2> <p>A WriteError reports an error encountered while writing output. </p>
+<p>Deprecated: No longer returned. </p>
+<pre data-language="go">type WriteError struct {
+ Offset int64 // byte offset where error occurred
+ Err error // error returned by underlying Write
+}
+</pre> <h3 id="WriteError.Error">func (*WriteError) <span>Error</span> </h3> <pre data-language="go">func (e *WriteError) Error() string</pre> <h2 id="Writer">type <span>Writer</span> </h2> <p>A Writer takes data written to it and writes the compressed form of that data to an underlying writer (see <a href="#NewWriter">NewWriter</a>). </p>
+<pre data-language="go">type Writer struct {
+ // contains filtered or unexported fields
+}
+</pre> <h3 id="NewWriter">func <span>NewWriter</span> </h3> <pre data-language="go">func NewWriter(w io.Writer, level int) (*Writer, error)</pre> <p>NewWriter returns a new <a href="#Writer">Writer</a> compressing data at the given level. Following zlib, levels range from 1 (<a href="#BestSpeed">BestSpeed</a>) to 9 (<a href="#BestCompression">BestCompression</a>); higher levels typically run slower but compress more. Level 0 (<a href="#NoCompression">NoCompression</a>) does not attempt any compression; it only adds the necessary DEFLATE framing. Level -1 (<a href="#DefaultCompression">DefaultCompression</a>) uses the default compression level. Level -2 (<a href="#HuffmanOnly">HuffmanOnly</a>) will use Huffman compression only, giving a very fast compression for all types of input, but sacrificing considerable compression efficiency. </p>
+<p>If level is in the range [-2, 9] then the error returned will be nil. Otherwise the error returned will be non-nil. </p>
+<h3 id="NewWriterDict">func <span>NewWriterDict</span> </h3> <pre data-language="go">func NewWriterDict(w io.Writer, level int, dict []byte) (*Writer, error)</pre> <p>NewWriterDict is like <a href="#NewWriter">NewWriter</a> but initializes the new <a href="#Writer">Writer</a> with a preset dictionary. The returned <a href="#Writer">Writer</a> behaves as if the dictionary had been written to it without producing any compressed output. The compressed data written to w can only be decompressed by a <a href="#Reader">Reader</a> initialized with the same dictionary. </p>
+<h3 id="Writer.Close">func (*Writer) <span>Close</span> </h3> <pre data-language="go">func (w *Writer) Close() error</pre> <p>Close flushes and closes the writer. </p>
+<h3 id="Writer.Flush">func (*Writer) <span>Flush</span> </h3> <pre data-language="go">func (w *Writer) Flush() error</pre> <p>Flush flushes any pending data to the underlying writer. It is useful mainly in compressed network protocols, to ensure that a remote reader has enough data to reconstruct a packet. Flush does not return until the data has been written. Calling Flush when there is no pending data still causes the <a href="#Writer">Writer</a> to emit a sync marker of at least 4 bytes. If the underlying writer returns an error, Flush returns that error. </p>
+<p>In the terminology of the zlib library, Flush is equivalent to Z_SYNC_FLUSH. </p>
+<h3 id="Writer.Reset">func (*Writer) <span>Reset</span> <span title="Added in Go 1.2">1.2</span> </h3> <pre data-language="go">func (w *Writer) Reset(dst io.Writer)</pre> <p>Reset discards the writer's state and makes it equivalent to the result of <a href="#NewWriter">NewWriter</a> or <a href="#NewWriterDict">NewWriterDict</a> called with dst and w's level and dictionary. </p>
+<h3 id="Writer.Write">func (*Writer) <span>Write</span> </h3> <pre data-language="go">func (w *Writer) Write(data []byte) (n int, err error)</pre> <p>Write writes data to w, which will eventually write the compressed form of data to its underlying writer. </p><div class="_attribution">
+ <p class="_attribution-p">
+ &copy; Google, Inc.<br>Licensed under the Creative Commons Attribution License 3.0.<br>
+ <a href="http://golang.org/pkg/compress/flate/" class="_attribution-link">http://golang.org/pkg/compress/flate/</a>
+ </p>
+</div>