README
¶
gozlib
gozlib is a GC-friendly compression package with a low memory allocation footprint. It provides various interfaces into the zlib library, using cgo for interoperability. See https://github.com/madler/zlib for details about zlib.
Requirements
Using this package requires cgo and a gnu compiler (clang or gcc), as well as the development version of zlib installed. By default, it expect the zlib header and so files to be in the standard include and library path. If not, you can override it by setting the appropriate paths in the environment variables CGO_CFLAGS and CGO_LDFLAGS.
Implementation and usage
Internally gozlib utilizes dxpool as an off-heap memory pool to maximize memory usage. At this moment, allocated memory is never returned to the system so gozlib is best used when gzip operations are frequent and constant.
This pool is also available for use in the Go code as a way to allocate and reuse byte slices.
See NativeSlicePool for details.
Compression and uncompression components
gozlib supports 3 different mechanisms for compressing and uncompressing data, each ideal to different use cases.
- Single step, in memory using
GoGZipCompressBuffer/GoUncompressBuffer - Event based with
GoGZipCompressStream/GoUncompressStream - Stream based, implementing
io.Reader/io.Writercreated throughNewGoZLibCompressorandNewGoZLibUncompressor. The returned object can be used as a drop in replacement to the standard library gzip implementation (or anything compatible with theiointerfaces).
Single step and event based possible through stateless functions while the stream based option keeps states through the returned object.
Like the standard library gzip implementation, it's possible to flush and reset gozlib's compressor and uncompressor so that they can be pooled and reused.
See the documentation and test files for usage examples and details.
Remember that the stream based, stateful use of gozlib require Close() to be invoked to avoid memory leaks.
Performance
While gozlib is designed to be GC friendly in that it keeps heap allocations to a minimum, managed and unmanaged.
For memory intensive applications, like web services that have to support http compression, an overall performance gain can be obtained just by reducing the frequency and duration of GC runs and pauses.
It's important to note that there is a penalty calling cgo and while the underlying zlib implementation can be faster, benchmarking and carefully measuring performance is necessary to decided on the best approach for each use case.
General case benchmarks show that gozlib performs better in terms of speed and memory utilization. See results below.
HTTP request compression benchmark results
gozlib
Requests/sec: 259.1624
stdlib gzip
Requests/sec: 139.3972
Response time distribution
GC Pause time, as described in the runtime documentation
PauseTotalNs is the cumulative nanoseconds in GC stop-the-world pauses since the program started. During a stop-the-world pause, all goroutines are paused and only the garbage collector can run.
Cumulative number of GC runs
Documentation
¶
Overview ¶
Using this package requires cgo and a gnu compiler (clang or gcc), as well as the development version of zlib installed By default, it expect the zlib header and so files to be in the standard include and library path. If not, you can override it by setting the appropriate paths in the environment variables CGO_CFLAGS and CGO_LDFLAGS Internally gozlib utilizes an off-heap memory pool to maximize memory usage. At this moment, allocated memory is never returned to the system so gozlib is best used when gzip operations are frequent and constant. This pool is also available for use in the Go code as a way to allocate and reuse byte slices. See NativeSlicePool for details
Index ¶
- Variables
- func Flush(compressor io.WriteCloser) error
- func GoGZipCompressBuffer(level CompressionLevel, input []byte, output []byte) (uint64, error)
- func GoGZipCompressStream(level CompressionLevel, inputBufferSize uint32, outputBufferSize uint32, ...) (uint64, error)
- func GoStreamDataInputHandler(ptr unsafe.Pointer, buffer unsafe.Pointer, bufferLength uint32) uint32
- func GoStreamDataOutputHandler(ptr unsafe.Pointer, buffer unsafe.Pointer, bufferLength uint32) uint32
- func GoUncompressBuffer(input []byte, output []byte) (uint64, error)
- func GoUncompressStream(inputBufferSize uint32, outputBufferSize uint32, ...) (uint64, error)
- func NewGoGZipCompressor(output io.Writer, level CompressionLevel, bufferSize uint32) (io.WriteCloser, error)
- func NewGoZLibUncompressor(input io.Reader, bufferSize uint32) (io.ReadCloser, error)
- func ResetCompressor(output io.Writer, compressor io.WriteCloser)
- func ResetUncompressor(input io.Reader, uncompressor io.ReadCloser)
- type CompressionLevel
- type DataStreamEventHandler
- type NativeSlicePool
- type TransformMode
Constants ¶
This section is empty.
Variables ¶
var ( // transformer TransformerUncompressionError = errors.New("error uncompressing data") TransformerInitializationError = errors.New("error initializing transformer") TransformerCompressionError = errors.New("error compressing data") // streaming StreamCompressError = errors.New("error streaming compressed data") StreamUncompressError = errors.New("error streaming uncompressed data") // buffer to buffer OutputBufferTooSmallError = errors.New("output buffer too small") BufferCompressError = errors.New("error compressing buffer") BufferUncompressError = errors.New("error uncompressing buffer") )
Functions ¶
func Flush ¶
func Flush(compressor io.WriteCloser) error
Flush is a helper function to flush a compressor given an interface
func GoGZipCompressBuffer ¶
func GoGZipCompressBuffer(level CompressionLevel, input []byte, output []byte) (uint64, error)
GoGZipCompressBuffer compresses data in gzip format, reading from input and writing to a pre allocated output buffer. If the output is too small to contain the compressed data, an error is returned
func GoGZipCompressStream ¶
func GoGZipCompressStream(level CompressionLevel, inputBufferSize uint32, outputBufferSize uint32, inputReader DataStreamEventHandler, outputWriter DataStreamEventHandler) (uint64, error)
GoGZipCompressStream compresses a stream of data The compression level can be CompressionLevelBestCompression or CompressionLevelBestSpeed `inputReader` is a function used to read uncompressed data `outputWriter` is a function that takes the compressed data `inputBufferSize` and `outputBufferSize` are the sizes of the internal work buffers. For best performance, use large enough power of 2 sizes The function returns the number of bytes written to the output stream and an error, if any.
func GoUncompressBuffer ¶
GoUncompressBuffer uncompresses zipb or an input buffer writing to a pre allocated output if the output is too small to contain the compressed data, an error is returned
func GoUncompressStream ¶
func GoUncompressStream(inputBufferSize uint32, outputBufferSize uint32, inputReader DataStreamEventHandler, outputWriter DataStreamEventHandler) (uint64, error)
GoUncompressStream uncompresses a stream of data in gzip or standard zlib format `inputReader` is a function used to read compressed data `outputWriter` is a function that takes the uncompressed data `inputBufferSize` and `outputBufferSize` are the sizes of the internal work buffers. For best performance, use large enough power of 2 sizes The function returns the number of bytes written to the output stream and an error, if any.
func NewGoGZipCompressor ¶
func NewGoGZipCompressor(output io.Writer, level CompressionLevel, bufferSize uint32) (io.WriteCloser, error)
NewGoGZipCompressor creates a new gzip compressor The compressor writes compressed data to the provided output Writer. The level parameter specifies the compression level. It can be set to CompressionLevelBestCompression or CompressionLevelBestSpeed The bufferSize parameter specifies the size of the buffer used by the compressor. For best performance, set it to a size that's power 2, large enough for the expected input. Returns an io.WriteCloser for writing compressed data and an error, if any.
func NewGoZLibUncompressor ¶
NewGoZLibUncompressor creates a new uncompressor that supports zlib or gzip inputs The input parameter is the io.Reader providing the compressed data to be uncompressed, and the bufferSize parameter is the size of the buffer to use in the internal compression transformer. For best performance, set it to a size that's power 2, large enough for the expected input.
func ResetCompressor ¶
func ResetCompressor(output io.Writer, compressor io.WriteCloser)
ResetCompressor is a helper function that can be used when pooling compressors The compressor will use the given output to write data to
func ResetUncompressor ¶
func ResetUncompressor(input io.Reader, uncompressor io.ReadCloser)
ResetUncompressor is a helper function that can be used when pooling uncompressors the uncompressor will use the given input to read data from
Types ¶
type CompressionLevel ¶
type CompressionLevel int
const ( CompressionLevelBestCompression CompressionLevel = C.Z_BEST_COMPRESSION CompressionLevelBestSpeed CompressionLevel = C.Z_BEST_SPEED )
type DataStreamEventHandler ¶
type NativeSlicePool ¶
type NativeSlicePool struct {
// contains filtered or unexported fields
}
NativeSlicePool is a byte slice pool manager where memory allocated for each slice is allocated off-heap The pool allows for slices of various types to be allocated and returned but given the way memory is internally tracked slices of sizes that are power of 2 provide an optimal memory utilization.
func NewNativeSlicePool ¶
func NewNativeSlicePool() *NativeSlicePool
NewNativeSlicePool creates a new slice pool. Manually call NewNativeSlicePool.Free() to release the resouces allocated by the returned NewNativeSlicePool.
func (*NativeSlicePool) Acquire ¶
func (nsp *NativeSlicePool) Acquire(size int) []byte
Acquire acquires a new byte array. For optimal memory utilization use sizes that are power of 2 The maximum size of a slice is limited to 4Mb and the returned slice cannot have its capacity changed. The returned slice is not zeroed out and it has length zero but capacity equals to size
func (*NativeSlicePool) Free ¶
func (nsp *NativeSlicePool) Free()
Free releases the resources allocated by this pool It must be invoked once the pool is not in use anymore to avoid resource leaks
func (*NativeSlicePool) Return ¶
func (nsp *NativeSlicePool) Return(slice []byte)
Return returns the slice to the pool.
type TransformMode ¶
type TransformMode int
const ( TransformModeZLib TransformMode = 0 TransformModeGZip TransformMode = 1 TransformModeUncompress TransformMode = 2 )
Source Files
Directories