README
¶
go-reader-writer
Description
go-reader-writer (aka GRW) is a simple library for reading and writing compressed resources by uri. GRW supports the following compression algorithms.
| Algorithm | Read | Write |
|---|---|---|
| bzip2 | ✓ | - |
| flate | ✓ | ✓ |
| gzip | ✓ | ✓ |
| snappy | ✓ | ✓ |
| zip | ✓ | - |
| zlib | ✓ | ✓ |
Using cross compilers, this library can also be called by other languages. This library is cross compiled into a Shared Object file (*.so). The Shared Object file can be called by C, C++, and Python on Linux machines. See the examples folder for patterns that you can use. This library is also compiled to pure JavaScript using GopherJS.
Platforms
The following platforms are supported. Pull requests to support other platforms are welcome!
| GOOS | 386 | amd64 | arm | arm64 |
|---|---|---|---|---|
| darwin | - | ✓ | - | - |
| freebsd | ✓ | ✓ | ✓ | - |
| linux | ✓ | ✓ | ✓ | ✓ |
| openbsd | ✓ | ✓ | - | - |
| solaris | - | ✓ | - | - |
| windows | ✓ | ✓ | - | - |
Releases
Find releases for the supported platforms at https://github.com/spatialcurrent/go-reader-writer/releases. See the Building section below to build for another platform from source.
Usage
CLI
The command line tool, grw, can be used to easily read and write compressed resources by uri. See the CLI.md document for detailed usage and examples.
Go
You can install the go-reader-writer packages with.
go get -u -d github.com/spatialcurrent/go-reader-writer/...
See grw in GoDoc for information on how to use Go API.
C
A variant of the reader and writer functions are exported in a Shared Object file (*.so), which can be called by C, C++, and Python programs on Linux machines. For complete patterns for C, C++, and Python, see the examples folder in this repo.
Releases
go-reader-writer is currently in alpha. Find releases at https://github.com/spatialcurrent/go-reader-writer/releases. See the Building section below to build from scratch.
Darwin
grw_darwin_amd64- CLI for Darwin on amd64 (includesmacOSandiOSplatforms)
Linux
grw_linux_amd64- CLI for Linux on amd64grw_linux_amd64- CLI for Linux on arm64grw_linux_amd64.h,grw_linuxamd64.so- Shared Object for Linux on amd64grw_linux_armv7.h,grw_linux_armv7.so- Shared Object for Linux on ARMv7grw_linux_armv8.h,grw_linux_armv8.so- Shared Object for Linux on ARMv8
Windows
grw_windows_amd64.exe- CLI for Windows on amd64
Examples
For the CLI, see the examples in the CLI.md document.
For Go, see the examples in the grw package documentation.
Building
Use make help to see help information for each target.
CLI
The make build_cli script is used to build executables for Linux and Windows.
JavaScript
You can compile GRW to pure JavaScript with the make build_javascript script.
Shared Object
The make build_so script is used to build Shared Objects (*.so), which can be called by C, C++, and Python on Linux machines.
Testing
CLI
To run CLI testes use make test_cli, which uses shUnit2. If you recive a shunit2:FATAL Please declare TMPDIR with path on partition with exec permission. error, you can modify the TMPDIR environment variable in line or with export TMPDIR=<YOUR TEMP DIRECTORY HERE>. For example:
TMPDIR="/usr/local/tmp" make test_cli
To test SFTP support, set following environment variables as shown below.
GRW_TESTDATA_SFTP=sftp://[email protected] OUTPUT_PRIVATE_KEY=~/sftp-test-1.pem INPUT_PRIVATE_KEY=~/sftp-test-1.pem OUTPUT_OVERWRITE=1 make test_cli
Go
To run Go tests use make test_go (or bash scripts/test.sh), which runs unit tests, go vet, go vet with shadow, errcheck, ineffassign, staticcheck, and misspell.
JavaScript
To run JavaScript tests, first install Jest using make deps_javascript, use Yarn, or another method. Then, build the JavaScript module with make build_javascript. To run tests, use make test_javascript. You can also use the scripts in the package.json.
Contributing
Spatial Current, Inc. is currently accepting pull requests for this repository. We'd love to have your contributions! Please see Contributing.md for how to get started.
License
This work is distributed under the MIT License. See LICENSE file.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
grw
grw is the command line version of go-reader-writer (GRW) that is used for reading/writing to multiple compression/archive formats
|
grw is the command line version of go-reader-writer (GRW) that is used for reading/writing to multiple compression/archive formats |
|
pkg
|
|
|
alg
Package alg includes the algorithms supported in this project
|
Package alg includes the algorithms supported in this project |
|
android
Package android includes functions for use with Android native applications.
|
Package android includes functions for use with Android native applications. |
|
archive/zip
Package zip provides a reader that propagate calls to Close.
|
Package zip provides a reader that propagate calls to Close. |
|
bufio
Package bufio provides a buffered reader and writer that propagate calls to Flush and Close.
|
Package bufio provides a buffered reader and writer that propagate calls to Flush and Close. |
|
bytes
Package bytes provides functions for decompressing byte streams.
|
Package bytes provides functions for decompressing byte streams. |
|
cache
Package cache includes a struct for caching reads.
|
Package cache includes a struct for caching reads. |
|
cli
Package cli includes functions for initializing the CLI.
|
Package cli includes functions for initializing the CLI. |
|
compress/bzip2
Package bzip2 provides a reader and writer that propagate calls to Close.
|
Package bzip2 provides a reader and writer that propagate calls to Close. |
|
compress/flate
Package flate provides a reader and writer that propagate calls to Flush and Close.
|
Package flate provides a reader and writer that propagate calls to Flush and Close. |
|
compress/gzip
Package gzip provides a writer that embeds the standard library gzip.Writer and propagates calls to Flush and Close.
|
Package gzip provides a writer that embeds the standard library gzip.Writer and propagates calls to Flush and Close. |
|
compress/snappy
Package snappy provides a reader and writer that propagate calls to Flush and Close.
|
Package snappy provides a reader and writer that propagate calls to Flush and Close. |
|
compress/zlib
Package zlib provides a writer that embeds the standard library zlib.Writer and propagates calls to Flush and Close.
|
Package zlib provides a writer that embeds the standard library zlib.Writer and propagates calls to Flush and Close. |
|
grw
Package grw provides the interfaces, embedded structs, and implementing code for normalizing the reading/writing of a stream of bytes from archive/compressed files.
|
Package grw provides the interfaces, embedded structs, and implementing code for normalizing the reading/writing of a stream of bytes from archive/compressed files. |
|
io
Package io supplements the interfaces provided by the standard library io package and provides functions for using those interfaces.
|
Package io supplements the interfaces provided by the standard library io package and provides functions for using those interfaces. |
|
net/ftp
Package ftp provides functions for interacting with a FTP server.
|
Package ftp provides functions for interacting with a FTP server. |
|
net/http
Package http provides functions for interacting with a HTTP server.
|
Package http provides functions for interacting with a HTTP server. |
|
net/sftp2
Package sftp2 provides functions for interacting with a SFTP server.
|
Package sftp2 provides functions for interacting with a SFTP server. |
|
net/ssh2
Package ssh2 provides functions for interacting with a SSH server.
|
Package ssh2 provides functions for interacting with a SSH server. |
|
nop
Package nop provides functions for creating no operation close methods.
|
Package nop provides functions for creating no operation close methods. |
|
os
Package os provides a few functions used throughout the grw package for interfacing with the local file system.
|
Package os provides a few functions used throughout the grw package for interfacing with the local file system. |
|
schemes
Package schemes includes the schemes supported in this project
|
Package schemes includes the schemes supported in this project |
|
splitter
Package splitter provides functions for splitting various pieces of a Uniform Resource Identifier (URI).
|
Package splitter provides functions for splitting various pieces of a Uniform Resource Identifier (URI). |
|
stat
Package stat provides structs for describing file into that can be used across file systems.
|
Package stat provides structs for describing file into that can be used across file systems. |
|
plugins
|
|
|
grw
grw.so creates a shared library of Go that can be called by C, C++, or Python - https://godoc.org/cmd/cgo - https://blog.golang.org/c-go-cgo
|
grw.so creates a shared library of Go that can be called by C, C++, or Python - https://godoc.org/cmd/cgo - https://blog.golang.org/c-go-cgo |