diff options
| author | Craig Jennings <c@cjennings.net> | 2024-04-07 13:41:34 -0500 |
|---|---|---|
| committer | Craig Jennings <c@cjennings.net> | 2024-04-07 13:41:34 -0500 |
| commit | 754bbf7a25a8dda49b5d08ef0d0443bbf5af0e36 (patch) | |
| tree | f1190704f78f04a2b0b4c977d20fe96a828377f1 /devdocs/go/compress%2Fflate%2Findex.html | |
new repository
Diffstat (limited to 'devdocs/go/compress%2Fflate%2Findex.html')
| -rw-r--r-- | devdocs/go/compress%2Fflate%2Findex.html | 307 |
1 files changed, 307 insertions, 0 deletions
diff --git a/devdocs/go/compress%2Fflate%2Findex.html b/devdocs/go/compress%2Fflate%2Findex.html new file mode 100644 index 00000000..88631066 --- /dev/null +++ b/devdocs/go/compress%2Fflate%2Findex.html @@ -0,0 +1,307 @@ +<h1> Package flate </h1> <ul id="short-nav"> +<li><code>import "compress/flate"</code></li> +<li><a href="#pkg-overview" class="overviewLink">Overview</a></li> +<li><a href="#pkg-index" class="indexLink">Index</a></li> +<li><a href="#pkg-examples" class="examplesLink">Examples</a></li> +</ul> <h2 id="pkg-overview">Overview </h2> <p>Package flate implements the DEFLATE compressed data format, described in RFC 1951. The gzip and zlib packages implement access to DEFLATE-based file formats. </p> <h4 id="example__dictionary"> <span class="text">Example (Dictionary)</span> +</h4> <p>A preset dictionary can be used to improve the compression ratio. The downside to using a dictionary is that the compressor and decompressor must agree in advance what dictionary to use. </p> <p>Code:</p> <pre class="code" data-language="go">// The dictionary is a string of bytes. When compressing some input data, +// the compressor will attempt to substitute substrings with matches found +// in the dictionary. As such, the dictionary should only contain substrings +// that are expected to be found in the actual data stream. +const dict = `<?xml version="1.0"?>` + `<book>` + `<data>` + `<meta name="` + `" content="` + +// The data to compress should (but is not required to) contain frequent +// substrings that match those in the dictionary. +const data = `<?xml version="1.0"?> +<book> + <meta name="title" content="The Go Programming Language"/> + <meta name="authors" content="Alan Donovan and Brian Kernighan"/> + <meta name="published" content="2015-10-26"/> + <meta name="isbn" content="978-0134190440"/> + <data>...</data> +</book> +` + +var b bytes.Buffer + +// Compress the data using the specially crafted dictionary. +zw, err := flate.NewWriterDict(&b, flate.DefaultCompression, []byte(dict)) +if err != nil { + log.Fatal(err) +} +if _, err := io.Copy(zw, strings.NewReader(data)); err != nil { + log.Fatal(err) +} +if err := zw.Close(); err != nil { + log.Fatal(err) +} + +// The decompressor must use the same dictionary as the compressor. +// Otherwise, the input may appear as corrupted. +fmt.Println("Decompressed output using the dictionary:") +zr := flate.NewReaderDict(bytes.NewReader(b.Bytes()), []byte(dict)) +if _, err := io.Copy(os.Stdout, zr); err != nil { + log.Fatal(err) +} +if err := zr.Close(); err != nil { + log.Fatal(err) +} + +fmt.Println() + +// Substitute all of the bytes in the dictionary with a '#' to visually +// demonstrate the approximate effectiveness of using a preset dictionary. +fmt.Println("Substrings matched by the dictionary are marked with #:") +hashDict := []byte(dict) +for i := range hashDict { + hashDict[i] = '#' +} +zr = flate.NewReaderDict(&b, hashDict) +if _, err := io.Copy(os.Stdout, zr); err != nil { + log.Fatal(err) +} +if err := zr.Close(); err != nil { + log.Fatal(err) +} + +</pre> <p>Output:</p> <pre class="output" data-language="go">Decompressed output using the dictionary: +<?xml version="1.0"?> +<book> + <meta name="title" content="The Go Programming Language"/> + <meta name="authors" content="Alan Donovan and Brian Kernighan"/> + <meta name="published" content="2015-10-26"/> + <meta name="isbn" content="978-0134190440"/> + <data>...</data> +</book> + +Substrings matched by the dictionary are marked with #: +##################### +###### + ############title###########The Go Programming Language"/# + ############authors###########Alan Donovan and Brian Kernighan"/# + ############published###########2015-10-26"/# + ############isbn###########978-0134190440"/# + ######...</##### +</##### +</pre> <h4 id="example__reset"> <span class="text">Example (Reset)</span> +</h4> <p>In performance critical applications, Reset can be used to discard the current compressor or decompressor state and reinitialize them quickly by taking advantage of previously allocated memory. </p> <p>Code:</p> <pre class="code" data-language="go">proverbs := []string{ + "Don't communicate by sharing memory, share memory by communicating.\n", + "Concurrency is not parallelism.\n", + "The bigger the interface, the weaker the abstraction.\n", + "Documentation is for users.\n", +} + +var r strings.Reader +var b bytes.Buffer +buf := make([]byte, 32<<10) + +zw, err := flate.NewWriter(nil, flate.DefaultCompression) +if err != nil { + log.Fatal(err) +} +zr := flate.NewReader(nil) + +for _, s := range proverbs { + r.Reset(s) + b.Reset() + + // Reset the compressor and encode from some input stream. + zw.Reset(&b) + if _, err := io.CopyBuffer(zw, &r, buf); err != nil { + log.Fatal(err) + } + if err := zw.Close(); err != nil { + log.Fatal(err) + } + + // Reset the decompressor and decode to some output stream. + if err := zr.(flate.Resetter).Reset(&b, nil); err != nil { + log.Fatal(err) + } + if _, err := io.CopyBuffer(os.Stdout, zr, buf); err != nil { + log.Fatal(err) + } + if err := zr.Close(); err != nil { + log.Fatal(err) + } +} + +</pre> <p>Output:</p> <pre class="output" data-language="go">Don't communicate by sharing memory, share memory by communicating. +Concurrency is not parallelism. +The bigger the interface, the weaker the abstraction. +Documentation is for users. +</pre> <h4 id="example__synchronization"> <span class="text">Example (Synchronization)</span> +</h4> <p>DEFLATE is suitable for transmitting compressed data across the network. </p> <p>Code:</p> <pre class="code" data-language="go">var wg sync.WaitGroup +defer wg.Wait() + +// Use io.Pipe to simulate a network connection. +// A real network application should take care to properly close the +// underlying connection. +rp, wp := io.Pipe() + +// Start a goroutine to act as the transmitter. +wg.Add(1) +go func() { + defer wg.Done() + + zw, err := flate.NewWriter(wp, flate.BestSpeed) + if err != nil { + log.Fatal(err) + } + + b := make([]byte, 256) + for _, m := range strings.Fields("A long time ago in a galaxy far, far away...") { + // We use a simple framing format where the first byte is the + // message length, followed the message itself. + b[0] = uint8(copy(b[1:], m)) + + if _, err := zw.Write(b[:1+len(m)]); err != nil { + log.Fatal(err) + } + + // Flush ensures that the receiver can read all data sent so far. + if err := zw.Flush(); err != nil { + log.Fatal(err) + } + } + + if err := zw.Close(); err != nil { + log.Fatal(err) + } +}() + +// Start a goroutine to act as the receiver. +wg.Add(1) +go func() { + defer wg.Done() + + zr := flate.NewReader(rp) + + b := make([]byte, 256) + for { + // Read the message length. + // This is guaranteed to return for every corresponding + // Flush and Close on the transmitter side. + if _, err := io.ReadFull(zr, b[:1]); err != nil { + if err == io.EOF { + break // The transmitter closed the stream + } + log.Fatal(err) + } + + // Read the message content. + n := int(b[0]) + if _, err := io.ReadFull(zr, b[:n]); err != nil { + log.Fatal(err) + } + + fmt.Printf("Received %d bytes: %s\n", n, b[:n]) + } + fmt.Println() + + if err := zr.Close(); err != nil { + log.Fatal(err) + } +}() + +</pre> <p>Output:</p> <pre class="output" data-language="go">Received 1 bytes: A +Received 4 bytes: long +Received 4 bytes: time +Received 3 bytes: ago +Received 2 bytes: in +Received 1 bytes: a +Received 6 bytes: galaxy +Received 4 bytes: far, +Received 3 bytes: far +Received 7 bytes: away... +</pre> <h2 id="pkg-index">Index </h2> <ul id="manual-nav"> +<li><a href="#pkg-constants">Constants</a></li> +<li><a href="#NewReader">func NewReader(r io.Reader) io.ReadCloser</a></li> +<li><a href="#NewReaderDict">func NewReaderDict(r io.Reader, dict []byte) io.ReadCloser</a></li> +<li><a href="#CorruptInputError">type CorruptInputError</a></li> +<li> <a href="#CorruptInputError.Error">func (e CorruptInputError) Error() string</a> +</li> +<li><a href="#InternalError">type InternalError</a></li> +<li> <a href="#InternalError.Error">func (e InternalError) Error() string</a> +</li> +<li><a href="#ReadError">type ReadError</a></li> +<li> <a href="#ReadError.Error">func (e *ReadError) Error() string</a> +</li> +<li><a href="#Reader">type Reader</a></li> +<li><a href="#Resetter">type Resetter</a></li> +<li><a href="#WriteError">type WriteError</a></li> +<li> <a href="#WriteError.Error">func (e *WriteError) Error() string</a> +</li> +<li><a href="#Writer">type Writer</a></li> +<li> <a href="#NewWriter">func NewWriter(w io.Writer, level int) (*Writer, error)</a> +</li> +<li> <a href="#NewWriterDict">func NewWriterDict(w io.Writer, level int, dict []byte) (*Writer, error)</a> +</li> +<li> <a href="#Writer.Close">func (w *Writer) Close() error</a> +</li> +<li> <a href="#Writer.Flush">func (w *Writer) Flush() error</a> +</li> +<li> <a href="#Writer.Reset">func (w *Writer) Reset(dst io.Writer)</a> +</li> +<li> <a href="#Writer.Write">func (w *Writer) Write(data []byte) (n int, err error)</a> +</li> +</ul> <div id="pkg-examples"> <h3>Examples</h3> <dl> <dd><a class="exampleLink" href="#example__dictionary">Package (Dictionary)</a></dd> <dd><a class="exampleLink" href="#example__reset">Package (Reset)</a></dd> <dd><a class="exampleLink" href="#example__synchronization">Package (Synchronization)</a></dd> </dl> </div> <h3>Package files</h3> <p> <span>deflate.go</span> <span>deflatefast.go</span> <span>dict_decoder.go</span> <span>huffman_bit_writer.go</span> <span>huffman_code.go</span> <span>inflate.go</span> <span>token.go</span> </p> <h2 id="pkg-constants">Constants</h2> <pre data-language="go">const ( + NoCompression = 0 + BestSpeed = 1 + BestCompression = 9 + DefaultCompression = -1 + + // HuffmanOnly disables Lempel-Ziv match searching and only performs Huffman + // entropy encoding. This mode is useful in compressing data that has + // already been compressed with an LZ style algorithm (e.g. Snappy or LZ4) + // that lacks an entropy encoder. Compression gains are achieved when + // certain bytes in the input stream occur more frequently than others. + // + // Note that HuffmanOnly produces a compressed output that is + // RFC 1951 compliant. That is, any valid DEFLATE decompressor will + // continue to be able to decompress this output. + HuffmanOnly = -2 +)</pre> <h2 id="NewReader">func <span>NewReader</span> </h2> <pre data-language="go">func NewReader(r io.Reader) io.ReadCloser</pre> <p>NewReader returns a new ReadCloser that can be used to read the uncompressed version of r. If r does not also implement <span>io.ByteReader</span>, the decompressor may read more data than necessary from r. The reader returns <span>io.EOF</span> after the final block in the DEFLATE stream has been encountered. Any trailing data after the final block is ignored. </p> +<p>The <span>io.ReadCloser</span> returned by NewReader also implements <a href="#Resetter">Resetter</a>. </p> +<h2 id="NewReaderDict">func <span>NewReaderDict</span> </h2> <pre data-language="go">func NewReaderDict(r io.Reader, dict []byte) io.ReadCloser</pre> <p>NewReaderDict is like <a href="#NewReader">NewReader</a> but initializes the reader with a preset dictionary. The returned <a href="#Reader">Reader</a> behaves as if the uncompressed data stream started with the given dictionary, which has already been read. NewReaderDict is typically used to read data compressed by NewWriterDict. </p> +<p>The ReadCloser returned by NewReaderDict also implements <a href="#Resetter">Resetter</a>. </p> +<h2 id="CorruptInputError">type <span>CorruptInputError</span> </h2> <p>A CorruptInputError reports the presence of corrupt input at a given offset. </p> +<pre data-language="go">type CorruptInputError int64</pre> <h3 id="CorruptInputError.Error">func (CorruptInputError) <span>Error</span> </h3> <pre data-language="go">func (e CorruptInputError) Error() string</pre> <h2 id="InternalError">type <span>InternalError</span> </h2> <p>An InternalError reports an error in the flate code itself. </p> +<pre data-language="go">type InternalError string</pre> <h3 id="InternalError.Error">func (InternalError) <span>Error</span> </h3> <pre data-language="go">func (e InternalError) Error() string</pre> <h2 id="ReadError">type <span>ReadError</span> </h2> <p>A ReadError reports an error encountered while reading input. </p> +<p>Deprecated: No longer returned. </p> +<pre data-language="go">type ReadError struct { + Offset int64 // byte offset where error occurred + Err error // error returned by underlying Read +} +</pre> <h3 id="ReadError.Error">func (*ReadError) <span>Error</span> </h3> <pre data-language="go">func (e *ReadError) Error() string</pre> <h2 id="Reader">type <span>Reader</span> </h2> <p>The actual read interface needed by <a href="#NewReader">NewReader</a>. If the passed in io.Reader does not also have ReadByte, the <a href="#NewReader">NewReader</a> will introduce its own buffering. </p> +<pre data-language="go">type Reader interface { + io.Reader + io.ByteReader +}</pre> <h2 id="Resetter">type <span>Resetter</span> <span title="Added in Go 1.4">1.4</span> </h2> <p>Resetter resets a ReadCloser returned by <a href="#NewReader">NewReader</a> or <a href="#NewReaderDict">NewReaderDict</a> to switch to a new underlying <a href="#Reader">Reader</a>. This permits reusing a ReadCloser instead of allocating a new one. </p> +<pre data-language="go">type Resetter interface { + // Reset discards any buffered data and resets the Resetter as if it was + // newly initialized with the given reader. + Reset(r io.Reader, dict []byte) error +}</pre> <h2 id="WriteError">type <span>WriteError</span> </h2> <p>A WriteError reports an error encountered while writing output. </p> +<p>Deprecated: No longer returned. </p> +<pre data-language="go">type WriteError struct { + Offset int64 // byte offset where error occurred + Err error // error returned by underlying Write +} +</pre> <h3 id="WriteError.Error">func (*WriteError) <span>Error</span> </h3> <pre data-language="go">func (e *WriteError) Error() string</pre> <h2 id="Writer">type <span>Writer</span> </h2> <p>A Writer takes data written to it and writes the compressed form of that data to an underlying writer (see <a href="#NewWriter">NewWriter</a>). </p> +<pre data-language="go">type Writer struct { + // contains filtered or unexported fields +} +</pre> <h3 id="NewWriter">func <span>NewWriter</span> </h3> <pre data-language="go">func NewWriter(w io.Writer, level int) (*Writer, error)</pre> <p>NewWriter returns a new <a href="#Writer">Writer</a> compressing data at the given level. Following zlib, levels range from 1 (<a href="#BestSpeed">BestSpeed</a>) to 9 (<a href="#BestCompression">BestCompression</a>); higher levels typically run slower but compress more. Level 0 (<a href="#NoCompression">NoCompression</a>) does not attempt any compression; it only adds the necessary DEFLATE framing. Level -1 (<a href="#DefaultCompression">DefaultCompression</a>) uses the default compression level. Level -2 (<a href="#HuffmanOnly">HuffmanOnly</a>) will use Huffman compression only, giving a very fast compression for all types of input, but sacrificing considerable compression efficiency. </p> +<p>If level is in the range [-2, 9] then the error returned will be nil. Otherwise the error returned will be non-nil. </p> +<h3 id="NewWriterDict">func <span>NewWriterDict</span> </h3> <pre data-language="go">func NewWriterDict(w io.Writer, level int, dict []byte) (*Writer, error)</pre> <p>NewWriterDict is like <a href="#NewWriter">NewWriter</a> but initializes the new <a href="#Writer">Writer</a> with a preset dictionary. The returned <a href="#Writer">Writer</a> behaves as if the dictionary had been written to it without producing any compressed output. The compressed data written to w can only be decompressed by a <a href="#Reader">Reader</a> initialized with the same dictionary. </p> +<h3 id="Writer.Close">func (*Writer) <span>Close</span> </h3> <pre data-language="go">func (w *Writer) Close() error</pre> <p>Close flushes and closes the writer. </p> +<h3 id="Writer.Flush">func (*Writer) <span>Flush</span> </h3> <pre data-language="go">func (w *Writer) Flush() error</pre> <p>Flush flushes any pending data to the underlying writer. It is useful mainly in compressed network protocols, to ensure that a remote reader has enough data to reconstruct a packet. Flush does not return until the data has been written. Calling Flush when there is no pending data still causes the <a href="#Writer">Writer</a> to emit a sync marker of at least 4 bytes. If the underlying writer returns an error, Flush returns that error. </p> +<p>In the terminology of the zlib library, Flush is equivalent to Z_SYNC_FLUSH. </p> +<h3 id="Writer.Reset">func (*Writer) <span>Reset</span> <span title="Added in Go 1.2">1.2</span> </h3> <pre data-language="go">func (w *Writer) Reset(dst io.Writer)</pre> <p>Reset discards the writer's state and makes it equivalent to the result of <a href="#NewWriter">NewWriter</a> or <a href="#NewWriterDict">NewWriterDict</a> called with dst and w's level and dictionary. </p> +<h3 id="Writer.Write">func (*Writer) <span>Write</span> </h3> <pre data-language="go">func (w *Writer) Write(data []byte) (n int, err error)</pre> <p>Write writes data to w, which will eventually write the compressed form of data to its underlying writer. </p><div class="_attribution"> + <p class="_attribution-p"> + © Google, Inc.<br>Licensed under the Creative Commons Attribution License 3.0.<br> + <a href="http://golang.org/pkg/compress/flate/" class="_attribution-link">http://golang.org/pkg/compress/flate/</a> + </p> +</div> |
