Documentation ¶
Overview ¶
Package ten provides efficient binary encoding for tensors. The format is 8 byte aligned and can be used directly for computations when transmitted, say, via RDMA. The format is supported by WebDataset with the `.ten` filename extension. It is also used by Tensorcom, Tensorcom RDMA, and can be used for fast tensor storage with LMDB and in disk files (which can be memory mapped).
Data is encoded as a series of chunks:
- magic number (int64)
- length in bytes (int64)
- bytes (multiple of 64 bytes long)
Arrays are a header chunk followed by a data chunk. Header chunks have the following structure:
- dtype (int64)
- 8 byte array name
- ndim (int64)
- dim[0]
- dim[1]
- ...
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrMagicNumberMismatch = fmt.Errorf("magic number mismatch") ErrNegativeLength = fmt.Errorf("negative length") ErrNegativeDimensions = fmt.Errorf("negative dimensions") ErrDecodingUnsupportedType = fmt.Errorf("unsupported data type") )
Decoding errors
var ( ErrTooManyDimensions = fmt.Errorf("too many dimensions") ErrInfoTooLong = fmt.Errorf("info can not exceed 8 bytes") )
Encoding errors
var MagicNumber = []byte{0x7e, 0x54, 0x65, 0x6e, 0x42, 0x69, 0x6e, 0x7e}
MagicNumber is the magic number before every chunk.
Functions ¶
This section is empty.
Types ¶
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
A Decoder reads and decodes tensor data from an input stream.
func NewDecoder ¶
NewDecoder returns a new decoder that reads from r.
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
An Encoder writes tensors to an output stream.
func NewEncoder ¶
NewEncoder returns a new encoder that writes to w.