README
¶
httpreaderat
Go package httpreaderat implements io.ReaderAt that makes HTTP Range Requests.
It can be used for example with "archive/zip" package in Go standard library. Together they can be used to access remote (HTTP accessible) ZIP archives without needing to download the whole archive file.
HTTP Range Requests (see RFC 7233) are used to retrieve the requested byte range. There is an optional fallback mechanism which can be used to download the whole file and buffer it locally if the server does not support Range Requests.
When using this package with "archive/zip", it is a good idea to also use "github.com/avvmoto/buf-readerat" which implements a buffered io.ReaderAt "proxy". It reduces the amount of small HTTP requests significantly. 1 MB is a good buffer size to use. See the example below for details.
If you need io.ReadSeeker (with Read() and Seek() methods) to be used for example with "archive/tar", you can wrap HTTPReaderAt with io.SectionReader.
Example
The following example outputs a file list of a remote zip archive without downloading the whole archive:
package main
import (
"archive/zip"
"fmt"
"github.com/avvmoto/buf-readerat"
"github.com/snabb/httpreaderat"
"net/http"
)
func main() {
req, _ := http.NewRequest("GET", "https://dl.google.com/go/go1.10.windows-amd64.zip", nil)
htrdr, err := httpreaderat.New(nil, req, nil)
if err != nil {
panic(err)
}
bhtrdr := bufra.NewBufReaderAt(htrdr, 1024*1024)
zrdr, err := zip.NewReader(bhtrdr, htrdr.Size())
if err != nil {
panic(err)
}
for _, f := range zrdr.File {
fmt.Println(f.Name)
}
}
License
MIT
Documentation
¶
Overview ¶
Package httpreaderat implements io.ReaderAt that makes HTTP Range Requests.
It can be used for example with "archive/zip" package in Go standard library. Together they can be used to access remote (HTTP accessible) ZIP archives without needing to download the whole archive file.
HTTP Range Requests (see RFC 7233) are used to retrieve the requested byte range.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNoRange = errors.New("server does not support range requests")
ErrNoRange error is returned if the server does not support range requests and there is no Store defined for buffering the file.
var ErrStoreLimit = errors.New("store size limit reached")
ErrStoreLimit error is returned when LimitedStore's limit is reached and there is no secondary fallback Store defined.
var ErrValidationFailed = errors.New("validation failed")
ErrValidationFailed error is returned if the file changed under our feet.
Functions ¶
This section is empty.
Types ¶
type HTTPReaderAt ¶
type HTTPReaderAt struct {
// contains filtered or unexported fields
}
HTTPReaderAt is io.ReaderAt implementation that makes HTTP Range Requests. New instances must be created with the New() function. It is safe for concurrent use.
func New ¶
New creates a new HTTPReaderAt. If nil is passed as http.Client, then http.DefaultClient is used. The supplied http.Request is used as a prototype for requests. It is copied before making the actual request. It is an error to specify any other HTTP method than "GET". A Store can be supplied to enable fallback mechanism in case the server does not support HTTP Range Requests.
func (*HTTPReaderAt) ContentType ¶
func (ra *HTTPReaderAt) ContentType() string
ContentType returns "Content-Type" header contents.
func (*HTTPReaderAt) LastModified ¶
func (ra *HTTPReaderAt) LastModified() string
LastModified returns "Last-Modified" header contents.
func (*HTTPReaderAt) ReadAt ¶
func (ra *HTTPReaderAt) ReadAt(p []byte, off int64) (n int, err error)
ReadAt reads len(b) bytes from the remote file starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF. It is safe for concurrent use.
It tries to notice if the file changes by tracking the size as well as Content-Type, Last-Modified and ETag headers between consecutive ReadAt calls. In case any change is detected, ErrValidationFailed is returned.
type LimitedStore ¶
type LimitedStore struct {
// contains filtered or unexported fields
}
LimitedStore stores to a primary store up to a size limit. If the size limit is exceeded, a secondary store is used. If secondary store is nil, error is returned if the size limit is exceeded.
func NewLimitedStore ¶
func NewLimitedStore(primary Store, limit int64, secondary Store) *LimitedStore
NewLimitedStore creates a new LimitedStore with the specified settings.
func (*LimitedStore) Close ¶
func (s *LimitedStore) Close() error
type Store ¶
Store is the interface to a temporary byte storage. Calling ReadFrom with io.Reader reads data to a temporary storage and allows it to be read back with ReadAt. A Store must be Closed to free up the space when it is no longer needed. A Store can be reused by filling it with new data. ReadFrom is not safe to be called concurrently. ReadAt is safe for concurrent use.
func NewDefaultStore ¶
func NewDefaultStore() Store
NewDefaultStore creates a Store with default settings. It buffers up to 1 MB in memory and if that is exceeded, up to 1 GB to a temporary file. Returned Store must be Closed if it is no longer needed.
type StoreFile ¶
type StoreFile struct {
// contains filtered or unexported fields
}
StoreFile takes data from io.Reader and provides io.ReaderAt backed by a temporary file. It implements the Store interface.
func NewStoreFile ¶
func NewStoreFile() *StoreFile
func (*StoreFile) Close ¶
Close must be called when the StoreFile is not used any more. It deletes the temporary file.
func (*StoreFile) ReadAt ¶
ReadAt reads len(b) bytes from the Store starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF. It is safe for concurrent use.
type StoreMemory ¶
type StoreMemory struct {
// contains filtered or unexported fields
}
StoreMemory takes data from io.Reader and provides io.ReaderAt backed by a memory buffer. It implements the Store interface.
func NewStoreMemory ¶
func NewStoreMemory() *StoreMemory
func (*StoreMemory) Close ¶
func (s *StoreMemory) Close() error
Close releases the memory buffer to be garbage collected.
func (*StoreMemory) ReadAt ¶
func (s *StoreMemory) ReadAt(p []byte, off int64) (n int, err error)
ReadAt reads len(b) bytes from the Store starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF. It is safe for concurrent use.
func (*StoreMemory) ReadFrom ¶
func (s *StoreMemory) ReadFrom(r io.Reader) (n int64, err error)
Read and store the contents of r to a memory buffer. Previous contents (if any) are erased.
func (*StoreMemory) Size ¶
func (s *StoreMemory) Size() int64
Size returns the amount of data (in bytes) in the Store.
Source Files
Directories