Documentation
¶
Overview ¶
* Package commitlog implements the append-only on-disk persistency layey.
CommitLog is the on-disk persistency layer for Slait. It is a simple append-only style format aiming the best performance for the use case. The idea is borrowed from Kafka (and Jocko), but we further simpliy it to remove the index file assuming the data is cached in the main memory.
Physical layout ¶
We assume every entry has timestamp and enforce entries to be ordered by the time in ascending order. Each record is encoded as follows.
- byte 0-7: timestamp of the record in Unix epoch nano seconds - byte 8-11: the size of the payload - byte 12-: payload byte array
The byte 0-7 and 8-11 are encoded in little endian. There is no padding in between records. Since the timestamp is encoded by Unix epoch nanoseconds, the maximum value for the timestamp is somewhere around the year 2262. The timezone info will not be considered in the physical layout, and the restored time is always in UTC timezone.
Segment files ¶
A partition is split into segment files. When the newest segment is full, new record is written into a new segment file. A segment file is named after the base nanosecond from the first record in the file. When a file is trimmed, the deletion happens only at the segment level. The maximum file size of the segment files are configured by the caller.
The module does not have any concurrency protection. The caller should take the appropriate action on use of this.
Index ¶
Constants ¶
const (
LogFileSuffix = ".log"
)
Variables ¶
var ( ErrSegmentNotFound = errors.New("segment not found") Encoding = binary.LittleEndian )
Functions ¶
This section is empty.
Types ¶
type ByteSizeCleaner ¶
type ByteSizeCleaner struct {
// Options["MaxLogBytes"] should be number string accepted by strconv.Atoi()
Options CleanerOptions
// -1 to avoid any deletes
MaxLogBytes int64
}
ByteSizeCleaner deletes leading segment files based on the total byte size of segments
type Cleaner ¶
func NewCleaner ¶
func NewCleaner(options CleanerOptions) Cleaner
type CleanerOptions ¶
type CommitLog ¶
type CommitLog struct {
Options
// contains filtered or unexported fields
}
func (*CommitLog) Tell ¶
func (l *CommitLog) Tell() *position
Tell tells the current logical position.
func (*CommitLog) Trim ¶
Trim deletes segment files according to its retention policy. Returns the base nanosec time of the first segment if some files have been deleted. A zero time is returned if nothing has changed. Maximum nano sec time (= 2262-04-11 23:47:16.854775807 +0000 UTC) is returned if all the segments are deleted.
type DurationCleaner ¶
type DurationCleaner struct {
// Options["Duration"] should be one of the format accepted by time.ParseDuration
Options CleanerOptions
Duration time.Duration
}
DurationCleaner deletes leading segments based on -1 * duration from time.Now()
type Options ¶
type Options struct {
Path string
MaxSegmentBytes int64
CleanerOptions CleanerOptions
}
Source Files