inmem

package
v1.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2017 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package inmem implements a shared, in-memory index for each database.

The in-memory index is the original index implementation and provides fast access to index data. However, it also forces high memory usage for large datasets and can cause OOM errors.

Index is the shared index structure that provides most of the functionality. However, ShardIndex is a light per-shard wrapper that adapts this original shared index format to the new per-shard format.

Index

Constants

View Source
const IndexName = "inmem"

IndexName is the name of this index.

Variables

This section is empty.

Functions

func NewShardIndex

func NewShardIndex(id uint64, database, path string, opt tsdb.EngineOptions) tsdb.Index

NewShardIndex returns a new index for a shard.

Types

type FilterExprs

type FilterExprs map[uint64]influxql.Expr

FilterExprs represents a map of series IDs to filter expressions.

func (FilterExprs) DeleteBoolLiteralTrues

func (fe FilterExprs) DeleteBoolLiteralTrues()

DeleteBoolLiteralTrues deletes all elements whose filter expression is a boolean literal true.

func (FilterExprs) Len

func (fe FilterExprs) Len() int

Len returns the number of elements.

type Index

type Index struct {
	// contains filtered or unexported fields
}

Index is the in memory index of a collection of measurements, time series, and their tags. Exported functions are goroutine safe while un-exported functions assume the caller will use the appropriate locks.

func NewIndex

func NewIndex(database string) *Index

NewIndex returns a new initialized Index.

func (*Index) AssignShard

func (i *Index) AssignShard(k string, shardID uint64)

AssignShard update the index to indicate that series k exists in the given shardID.

func (*Index) Close

func (i *Index) Close() error

func (*Index) CreateMeasurementIndexIfNotExists

func (i *Index) CreateMeasurementIndexIfNotExists(name []byte) *Measurement

CreateMeasurementIndexIfNotExists creates or retrieves an in memory index object for the measurement

func (*Index) CreateSeriesIfNotExists

func (i *Index) CreateSeriesIfNotExists(shardID uint64, key, name []byte, tags models.Tags, opt *tsdb.EngineOptions, ignoreLimits bool) error

CreateSeriesIfNotExists adds the series for the given measurement to the index and sets its ID or returns the existing series object

func (*Index) DropMeasurement

func (i *Index) DropMeasurement(name []byte) error

DropMeasurement removes the measurement and all of its underlying series from the database index

func (*Index) DropSeries

func (i *Index) DropSeries(key []byte) error

DropSeries removes the series key and its tags from the index.

func (*Index) ForEachMeasurementName

func (i *Index) ForEachMeasurementName(fn func(name []byte) error) error

ForEachMeasurementName iterates over each measurement name.

func (*Index) ForEachMeasurementTagKey

func (i *Index) ForEachMeasurementTagKey(name []byte, fn func(key []byte) error) error

ForEachMeasurementTagKey iterates over all tag keys for a measurement.

func (*Index) HasTagKey

func (i *Index) HasTagKey(name, key []byte) (bool, error)

HasTagKey returns true if tag key exists.

func (*Index) HasTagValue

func (i *Index) HasTagValue(name, key, value []byte) bool

HasTagValue returns true if tag value exists.

func (*Index) Measurement

func (i *Index) Measurement(name []byte) (*Measurement, error)

Measurement returns the measurement object from the index by the name

func (*Index) MeasurementExists

func (i *Index) MeasurementExists(name []byte) (bool, error)

MeasurementExists returns true if the measurement exists.

func (*Index) MeasurementNamesByExpr

func (i *Index) MeasurementNamesByExpr(expr influxql.Expr) ([][]byte, error)

MeasurementNamesByExpr takes an expression containing only tags and returns a list of matching meaurement names.

func (*Index) MeasurementNamesByRegex

func (i *Index) MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)

MeasurementNamesByRegex returns the measurements that match the regex.

func (*Index) MeasurementSeriesKeysByExpr

func (i *Index) MeasurementSeriesKeysByExpr(name []byte, condition influxql.Expr) ([][]byte, error)

func (*Index) MeasurementTagKeyValuesByExpr added in v1.3.2

func (i *Index) MeasurementTagKeyValuesByExpr(auth query.Authorizer, name []byte, keys []string, expr influxql.Expr, keysSorted bool) ([][]string, error)

MeasurementTagKeyValuesByExpr returns a set of tag values filtered by an expression.

See tsm1.Engine.MeasurementTagKeyValuesByExpr for a fuller description of this method.

func (*Index) MeasurementTagKeysByExpr

func (i *Index) MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)

MeasurementTagKeysByExpr returns an ordered set of tag keys filtered by an expression.

func (*Index) MeasurementsByName

func (i *Index) MeasurementsByName(names [][]byte) ([]*Measurement, error)

MeasurementsByName returns a list of measurements.

func (*Index) MeasurementsSketches

func (i *Index) MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)

MeasurementsSketches returns the sketches for the measurements.

func (*Index) Open

func (i *Index) Open() (err error)

func (*Index) Rebuild added in v1.4.0

func (i *Index) Rebuild()

Rebuild recreates the measurement indexes to allow deleted series to be removed and garbage collected.

func (*Index) RemoveShard

func (i *Index) RemoveShard(shardID uint64)

RemoveShard removes all references to shardID from any series or measurements in the index. If the shard was the only owner of data for the series, the series is removed from the index.

func (*Index) Series

func (i *Index) Series(key []byte) (*Series, error)

Series returns a series by key.

func (*Index) SeriesKeys

func (i *Index) SeriesKeys() []string

func (*Index) SeriesN

func (i *Index) SeriesN() int64

SeriesN returns the number of unique non-tombstoned series in the index. Since indexes are not shared across shards, the count returned by SeriesN cannot be combined with other shards' counts.

func (*Index) SeriesPointIterator

func (i *Index) SeriesPointIterator(opt query.IteratorOptions) (query.Iterator, error)

SeriesPointIterator returns an influxql iterator over all series.

func (*Index) SeriesSketches

func (i *Index) SeriesSketches() (estimator.Sketch, estimator.Sketch, error)

SeriesSketches returns the sketches for the series.

func (*Index) SetFieldName

func (i *Index) SetFieldName(measurement []byte, name string)

SetFieldName adds a field name to a measurement.

func (*Index) SetFieldSet

func (i *Index) SetFieldSet(*tsdb.MeasurementFieldSet)

SetFieldSet sets a shared field set from the engine.

func (*Index) SnapshotTo

func (i *Index) SnapshotTo(path string) error

SnapshotTo is a no-op since this is an in-memory index.

func (*Index) TagKeyCardinality

func (i *Index) TagKeyCardinality(name, key []byte) int

TagKeyCardinality returns the number of values for a measurement/tag key.

func (*Index) TagSets

func (i *Index) TagSets(shardID uint64, name []byte, opt query.IteratorOptions) ([]*query.TagSet, error)

TagSets returns a list of tag sets.

func (*Index) TagValueN

func (i *Index) TagValueN(name, key []byte) int

TagValueN returns the cardinality of a tag value.

func (*Index) TagsForSeries

func (i *Index) TagsForSeries(key string) (models.Tags, error)

TagsForSeries returns the tag map for the passed in series

func (*Index) Type

func (i *Index) Type() string

func (*Index) UnassignShard

func (i *Index) UnassignShard(k string, shardID uint64) error

UnassignShard updates the index to indicate that series k does not exist in the given shardID.

func (*Index) WithLogger

func (i *Index) WithLogger(zap.Logger)

type Measurement

type Measurement struct {
	Name string `json:"name,omitempty"`
	// contains filtered or unexported fields
}

Measurement represents a collection of time series in a database. It also contains in memory structures for indexing tags. Exported functions are goroutine safe while un-exported functions assume the caller will use the appropriate locks.

func NewMeasurement

func NewMeasurement(database, name string) *Measurement

NewMeasurement allocates and initializes a new Measurement.

func (*Measurement) AddSeries

func (m *Measurement) AddSeries(s *Series) bool

AddSeries adds a series to the measurement's index. It returns true if the series was added successfully or false if the series was already present.

func (*Measurement) AppendSeriesKeysByID

func (m *Measurement) AppendSeriesKeysByID(dst []string, ids []uint64) []string

AppendSeriesKeysByID appends keys for a list of series ids to a buffer.

func (*Measurement) Cardinality

func (m *Measurement) Cardinality(key string) int

Cardinality returns the number of values associated with the given tag key.

func (*Measurement) CardinalityBytes

func (m *Measurement) CardinalityBytes(key []byte) int

CardinalityBytes returns the number of values associated with the given tag key.

func (*Measurement) DropSeries

func (m *Measurement) DropSeries(series *Series)

DropSeries removes a series from the measurement's index.

func (*Measurement) FieldNames

func (m *Measurement) FieldNames() []string

FieldNames returns a list of the measurement's field names, in an arbitrary order.

func (*Measurement) ForEachSeriesByExpr

func (m *Measurement) ForEachSeriesByExpr(condition influxql.Expr, fn func(tags models.Tags) error) error

ForEachSeriesByExpr iterates over all series filtered by condition.

func (*Measurement) HasField

func (m *Measurement) HasField(name string) bool

func (*Measurement) HasSeries

func (m *Measurement) HasSeries() bool

HasSeries returns true if there is at least 1 series under this measurement.

func (*Measurement) HasTagKey

func (m *Measurement) HasTagKey(k string) bool

HasTagKey returns true if at least one series in this measurement has written a value for the passed in tag key

func (*Measurement) HasTagKeyValue

func (m *Measurement) HasTagKeyValue(k, v []byte) bool

func (*Measurement) IDsForExpr

func (m *Measurement) IDsForExpr(n *influxql.BinaryExpr) SeriesIDs

IDsForExpr returns the series IDs that are candidates to match the given expression.

func (*Measurement) Rebuild added in v1.4.0

func (m *Measurement) Rebuild() *Measurement

func (*Measurement) SeriesByID

func (m *Measurement) SeriesByID(id uint64) *Series

SeriesByID returns a series by identifier.

func (*Measurement) SeriesByIDMap

func (m *Measurement) SeriesByIDMap() map[uint64]*Series

SeriesByIDMap returns the internal seriesByID map.

func (*Measurement) SeriesByIDSlice

func (m *Measurement) SeriesByIDSlice(ids []uint64) []*Series

SeriesByIDSlice returns a list of series by identifiers.

func (*Measurement) SeriesByTagKeyValue

func (m *Measurement) SeriesByTagKeyValue(key string) *TagKeyValue

SeriesByTagKeyValue returns the TagKeyValue for the provided tag key.

func (*Measurement) SeriesIDs

func (m *Measurement) SeriesIDs() SeriesIDs

func (*Measurement) SeriesIDsAllOrByExpr

func (m *Measurement) SeriesIDsAllOrByExpr(expr influxql.Expr) (SeriesIDs, error)

SeriesIDsAllOrByExpr walks an expressions for matching series IDs or, if no expressions is given, returns all series IDs for the measurement.

func (*Measurement) SeriesKeys

func (m *Measurement) SeriesKeys() [][]byte

SeriesKeys returns the keys of every series in this measurement

func (*Measurement) SeriesKeysByID

func (m *Measurement) SeriesKeysByID(ids SeriesIDs) [][]byte

SeriesKeysByID returns the a list of keys for a set of ids.

func (*Measurement) SetFieldName

func (m *Measurement) SetFieldName(name string)

SetFieldName adds the field name to the measurement.

func (*Measurement) TagKeys

func (m *Measurement) TagKeys() []string

TagKeys returns a list of the measurement's tag names, in sorted order.

func (*Measurement) TagKeysByExpr

func (m *Measurement) TagKeysByExpr(expr influxql.Expr) (map[string]struct{}, error)

tagKeysByExpr extracts the tag keys wanted by the expression.

func (*Measurement) TagSets

func (m *Measurement) TagSets(shardID uint64, opt query.IteratorOptions) ([]*query.TagSet, error)

TagSets returns the unique tag sets that exist for the given tag keys. This is used to determine what composite series will be created by a group by. i.e. "group by region" should return: {"region":"uswest"}, {"region":"useast"} or region, service returns {"region": "uswest", "service": "redis"}, {"region": "uswest", "service": "mysql"}, etc... This will also populate the TagSet objects with the series IDs that match each tagset and any influx filter expression that goes with the series TODO: this shouldn't be exported. However, until tx.go and the engine get refactored into tsdb, we need it.

func (*Measurement) TagValues

func (m *Measurement) TagValues(auth query.Authorizer, key string) []string

TagValues returns all the values for the given tag key, in an arbitrary order.

func (*Measurement) WalkTagKeys

func (m *Measurement) WalkTagKeys(fn func(k string))

WalkTagKeys calls fn for each tag key associated with m. The order of the keys is undefined.

func (*Measurement) WalkWhereForSeriesIds

func (m *Measurement) WalkWhereForSeriesIds(expr influxql.Expr) (SeriesIDs, FilterExprs, error)

WalkWhereForSeriesIds recursively walks the WHERE clause and returns an ordered set of series IDs and a map from those series IDs to filter expressions that should be used to limit points returned in the final query result.

type Measurements

type Measurements []*Measurement

Measurements represents a list of *Measurement.

func (Measurements) Intersect

func (a Measurements) Intersect(other Measurements) Measurements

func (Measurements) Len

func (a Measurements) Len() int

Len implements sort.Interface.

func (Measurements) Less

func (a Measurements) Less(i, j int) bool

Less implements sort.Interface.

func (Measurements) Swap

func (a Measurements) Swap(i, j int)

Swap implements sort.Interface.

func (Measurements) Union

func (a Measurements) Union(other Measurements) Measurements

type Series

type Series struct {
	Key string

	ID uint64
	// contains filtered or unexported fields
}

Series belong to a Measurement and represent unique time series in a database.

func NewSeries

func NewSeries(key []byte, tags models.Tags) *Series

NewSeries returns an initialized series struct

func (*Series) AssignShard

func (s *Series) AssignShard(shardID uint64)

func (*Series) Assigned

func (s *Series) Assigned(shardID uint64) bool

func (*Series) CopyTags

func (s *Series) CopyTags()

CopyTags clones the tags on the series in-place,

func (*Series) Delete added in v1.4.0

func (s *Series) Delete()

Delete marks this series as deleted. A deleted series should not be returned for queries.

func (*Series) Deleted added in v1.4.0

func (s *Series) Deleted() bool

Deleted indicates if this was previously deleted.

func (*Series) ForEachTag

func (s *Series) ForEachTag(fn func(models.Tag))

ForEachTag executes fn for every tag. Iteration occurs under lock.

func (*Series) GetTagString

func (s *Series) GetTagString(key string) string

GetTagString returns a tag value under lock.

func (*Series) Measurement

func (s *Series) Measurement() *Measurement

Measurement returns the measurement on the series.

func (*Series) SetMeasurement

func (s *Series) SetMeasurement(m *Measurement)

SetMeasurement sets the measurement on the series.

func (*Series) ShardN

func (s *Series) ShardN() int

func (*Series) Tags

func (s *Series) Tags() models.Tags

Tags returns a copy of the tags under lock.

func (*Series) UnassignShard

func (s *Series) UnassignShard(shardID uint64)

type SeriesIDs

type SeriesIDs []uint64

SeriesIDs is a convenience type for sorting, checking equality, and doing union and intersection of collections of series ids.

func (SeriesIDs) Equals

func (a SeriesIDs) Equals(other SeriesIDs) bool

Equals assumes that both are sorted.

func (SeriesIDs) Intersect

func (a SeriesIDs) Intersect(other SeriesIDs) SeriesIDs

Intersect returns a new collection of series ids in sorted order that is the intersection of the two. The two collections must already be sorted.

func (SeriesIDs) Len

func (a SeriesIDs) Len() int

Len implements sort.Interface.

func (SeriesIDs) Less

func (a SeriesIDs) Less(i, j int) bool

Less implements sort.Interface.

func (SeriesIDs) Reject

func (a SeriesIDs) Reject(other SeriesIDs) SeriesIDs

Reject returns a new collection of series ids in sorted order with the passed in set removed from the original. This is useful for the NOT operator. The two collections must already be sorted.

func (SeriesIDs) Swap

func (a SeriesIDs) Swap(i, j int)

Swap implements sort.Interface.

func (SeriesIDs) Union

func (a SeriesIDs) Union(other SeriesIDs) SeriesIDs

Union returns a new collection of series ids in sorted order that is the union of the two. The two collections must already be sorted.

type ShardIndex

type ShardIndex struct {
	*Index
	// contains filtered or unexported fields
}

ShardIndex represents a shim between the TSDB index interface and the shared in-memory index. This is required because per-shard in-memory indexes will grow the heap size too large.

func (*ShardIndex) CreateSeriesIfNotExists

func (i *ShardIndex) CreateSeriesIfNotExists(key, name []byte, tags models.Tags) error

func (*ShardIndex) CreateSeriesListIfNotExists

func (idx *ShardIndex) CreateSeriesListIfNotExists(keys, names [][]byte, tagsSlice []models.Tags) error

CreateSeriesListIfNotExists creates a list of series if they doesn't exist in bulk.

func (*ShardIndex) InitializeSeries

func (i *ShardIndex) InitializeSeries(key, name []byte, tags models.Tags) error

InitializeSeries is called during startup. This works the same as CreateSeriesIfNotExists except it ignore limit errors.

func (*ShardIndex) TagSets

func (i *ShardIndex) TagSets(name []byte, opt query.IteratorOptions) ([]*query.TagSet, error)

TagSets returns a list of tag sets based on series filtering.

type TagFilter

type TagFilter struct {
	Op    influxql.Token
	Key   string
	Value string
	Regex *regexp.Regexp
}

TagFilter represents a tag filter when looking up other tags or measurements.

type TagKeyValue added in v1.4.0

type TagKeyValue struct {
	// contains filtered or unexported fields
}

TagKeyValue provides goroutine-safe concurrent access to the set of series ids mapping to a set of tag values.

TODO(edd): This could possibly be replaced by a sync.Map once we use Go 1.9.

func NewTagKeyValue added in v1.4.0

func NewTagKeyValue() *TagKeyValue

NewTagKeyValue initialises a new TagKeyValue.

func (*TagKeyValue) Cardinality added in v1.4.0

func (t *TagKeyValue) Cardinality() int

Cardinality returns the number of values in the TagKeyValue.

func (*TagKeyValue) Contains added in v1.4.0

func (t *TagKeyValue) Contains(value string) bool

Contains returns true if the TagKeyValue contains value.

func (*TagKeyValue) Load added in v1.4.0

func (t *TagKeyValue) Load(value string) SeriesIDs

Load returns the SeriesIDs for the provided tag value.

func (*TagKeyValue) LoadByte added in v1.4.0

func (t *TagKeyValue) LoadByte(value []byte) SeriesIDs

LoadByte returns the SeriesIDs for the provided tag value. It makes use of Go's compiler optimisation for avoiding a copy when accessing maps with a []byte.

func (*TagKeyValue) Range added in v1.4.0

func (t *TagKeyValue) Range(f func(k string, a SeriesIDs) bool)

Range calls f sequentially on each key and value. A call to Range on a nil TagKeyValue is a no-op.

If f returns false then iteration over any remaining keys or values will cease.

func (*TagKeyValue) RangeAll added in v1.4.0

func (t *TagKeyValue) RangeAll(f func(k string, a SeriesIDs))

RangeAll calls f sequentially on each key and value. A call to RangeAll on a nil TagKeyValue is a no-op.

func (*TagKeyValue) Store added in v1.4.0

func (t *TagKeyValue) Store(value string, ids SeriesIDs)

Store stores ids under the value key.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL