Documentation
¶
Overview ¶
Package kvbtree
Index ¶
- type KVBTreeBucket
- func (bucket *KVBTreeBucket) Close() (err error)
- func (bucket *KVBTreeBucket) Cursor(ctx context.Context) (buckets.IBucketCursor, error)
- func (bucket *KVBTreeBucket) Delete(key string) error
- func (bucket *KVBTreeBucket) Export() map[string][]byte
- func (bucket *KVBTreeBucket) Get(key string) (val []byte, err error)
- func (bucket *KVBTreeBucket) GetMultiple(keys []string) (docs map[string][]byte, err error)
- func (bucket *KVBTreeBucket) ID() string
- func (bucket *KVBTreeBucket) Info() (info *buckets.BucketStoreInfo)
- func (bucket *KVBTreeBucket) Set(key string, doc []byte) error
- func (bucket *KVBTreeBucket) SetMultiple(docs map[string][]byte) (err error)
- type KVBTreeCursor
- func (cursor *KVBTreeCursor) BucketID() string
- func (cursor *KVBTreeCursor) Context() context.Context
- func (cursor *KVBTreeCursor) First() (key string, value []byte, valid bool)
- func (cursor *KVBTreeCursor) Last() (key string, value []byte, valid bool)
- func (cursor *KVBTreeCursor) Next() (key string, value []byte, valid bool)
- func (cursor *KVBTreeCursor) NextN(steps uint) (docs map[string][]byte, itemsRemaining bool)
- func (cursor *KVBTreeCursor) Prev() (key string, value []byte, valid bool)
- func (cursor *KVBTreeCursor) PrevN(steps uint) (docs map[string][]byte, itemsRemaining bool)
- func (cursor *KVBTreeCursor) Release()
- func (cursor *KVBTreeCursor) Seek(searchKey string) (key string, value []byte, valid bool)
- type KVBTreeStore
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type KVBTreeBucket ¶
type KVBTreeBucket struct {
BucketID string `json:"bucketID"`
ClientID string `json:"clientID"`
// contains filtered or unexported fields
}
KVBTreeBucket is an in-memory bucket for the KVBTreeBucket
func NewKVMemBucket ¶
func NewKVMemBucket(clientID, bucketID string) *KVBTreeBucket
func NewKVMemBucketFromMap ¶
func NewKVMemBucketFromMap(clientID, bucketID string, data map[string][]byte) *KVBTreeBucket
NewKVMemBucketFromMap creates a new KVBTreeBucket from a map with bucket data Intended for loading a saved store.
func (*KVBTreeBucket) Close ¶
func (bucket *KVBTreeBucket) Close() (err error)
Close the bucket and release its resources commit is not used as this store doesn't handle transactions. This decreases the refCount and detects an error if below 0
func (*KVBTreeBucket) Cursor ¶
func (bucket *KVBTreeBucket) Cursor( ctx context.Context) (buckets.IBucketCursor, error)
Cursor returns a new cursor for iterating the bucket. The cursor MUST be closed after use to release its memory.
This implementation is brute force. It generates a sorted list of key/values for use by the cursor. The cursor makes a shallow copy of the store. Store mutations are not reflected in the cursor.
This should be fast enough for many use-cases. 100K records takes around 27msec on an [email protected]
This returns a cursor with Next() and Prev() iterators
func (*KVBTreeBucket) Delete ¶
func (bucket *KVBTreeBucket) Delete(key string) error
Delete a document from the bucket Also succeeds if the document doesn't exist
func (*KVBTreeBucket) Export ¶
func (bucket *KVBTreeBucket) Export() map[string][]byte
Export returns a shallow copy of the bucket content
func (*KVBTreeBucket) Get ¶
func (bucket *KVBTreeBucket) Get(key string) (val []byte, err error)
Get an object by its ID returns an error if the key does not exist.
func (*KVBTreeBucket) GetMultiple ¶
func (bucket *KVBTreeBucket) GetMultiple(keys []string) (docs map[string][]byte, err error)
GetMultiple returns a batch of documents for the given key The document can be any text.
func (*KVBTreeBucket) ID ¶
func (bucket *KVBTreeBucket) ID() string
func (*KVBTreeBucket) Info ¶
func (bucket *KVBTreeBucket) Info() (info *buckets.BucketStoreInfo)
Info returns the bucket info
func (*KVBTreeBucket) Set ¶
func (bucket *KVBTreeBucket) Set(key string, doc []byte) error
Set writes a document to the store. If the document exists it is replaced. This will store a copy of doc
A background process periodically checks the change count. When increased: 1. Lock the store while copying the index. Unlock when done. 2. Stream the in-memory json documents to a temp file. 3. If success, move the temp file to the store file using the OS atomic move operation.
func (*KVBTreeBucket) SetMultiple ¶
func (bucket *KVBTreeBucket) SetMultiple(docs map[string][]byte) (err error)
SetMultiple writes a batch of key-values Values are copied
type KVBTreeCursor ¶
type KVBTreeCursor struct {
// contains filtered or unexported fields
}
func NewKVCursor ¶
func NewKVCursor(ctx context.Context, bucket buckets.IBucket, kvIter btree.MapIter[string, []byte]) *KVBTreeCursor
NewKVCursor create a new bucket cursor for the KV store. Cursor.Close() must be called to release the resources.
bucket is the bucket holding the data orderedKeys is a snapshot of the keys in ascending order
func NewKVCursor(bucket bucketstore.IBucket, orderedKeys []string, kvbtree btree.Map[string, []byte]) *KVBTreeCursor {
func (*KVBTreeCursor) BucketID ¶
func (cursor *KVBTreeCursor) BucketID() string
func (*KVBTreeCursor) Context ¶
func (cursor *KVBTreeCursor) Context() context.Context
Context returns the cursor application context
func (*KVBTreeCursor) First ¶
func (cursor *KVBTreeCursor) First() (key string, value []byte, valid bool)
First moves the cursor to the first item
func (*KVBTreeCursor) Last ¶
func (cursor *KVBTreeCursor) Last() (key string, value []byte, valid bool)
Last moves the cursor to the last item
func (*KVBTreeCursor) Next ¶
func (cursor *KVBTreeCursor) Next() (key string, value []byte, valid bool)
Next increases the cursor position and return the next key and value If the end is reached the returned key is empty
func (*KVBTreeCursor) NextN ¶
func (cursor *KVBTreeCursor) NextN(steps uint) (docs map[string][]byte, itemsRemaining bool)
NextN increases the cursor position N times and return the encountered key-value pairs
func (*KVBTreeCursor) Prev ¶
func (cursor *KVBTreeCursor) Prev() (key string, value []byte, valid bool)
Prev decreases the cursor position and return the previous key and value If the head is reached the returned key is empty
func (*KVBTreeCursor) PrevN ¶
func (cursor *KVBTreeCursor) PrevN(steps uint) (docs map[string][]byte, itemsRemaining bool)
PrevN decreases the cursor position N times and return the encountered key-value pairs
func (*KVBTreeCursor) Release ¶
func (cursor *KVBTreeCursor) Release()
Release the cursor capability
func (*KVBTreeCursor) Seek ¶
func (cursor *KVBTreeCursor) Seek(searchKey string) (key string, value []byte, valid bool)
Seek positions the cursor at the given searchKey. This implementation is brute force. It generates a sorted list of orderedKeys for use by the cursor. This should still be fast enough for most cases. (test shows around 500msec for 1 million orderedKeys).
BucketID to seach for. Returns and error if the bucket is not found key is the starting point. If key doesn't exist, the next closest key will be used.
This returns a cursor with Next() and Prev() iterators
type KVBTreeStore ¶
type KVBTreeStore struct {
// contains filtered or unexported fields
}
KVBTreeStore is an embedded, file backed, in-memory, lightweight, and very fast key-value bucket store. This is intended for simple use-cases with up to 100K records. Interestingly, this simple brute-force store using maps is faster than anything else I've tested and even scales up to 1M records. Pretty much all you need for basic databases.
Changes are periodically persisted to file in the background.
Limitations:
- No transaction support (basic usage, remember)
- Changes are periodically (default 3 second) written to disk
Improvements for future considerations:
- Append-only to reduce disk writes for larger databases
--- about jsonpath --- This was experimental because of the W3C WoT recommendation, and seems to work well. However this is shelved as the Hub has no use-case for it and the other stores don't support it.
Query: jsonPath: `$[?(@.properties.title.name=="title1")]`
Approx 1.8 msec with a dataset of 1K records (1 result) Approx 23 msec with a dataset of 10K records (1 result) Approx 260 msec with a dataset of 100K records (1 result)
A good overview of jsonpath implementations can be found here: > https://cburgmer.github.io/json-path-comparison/ Two good options for jsonpath queries:
> github.com/ohler55/ojg/jp > github.com/PaesslerAG/jsonpath
Note that future implementations of this service can change the storage media used while maintaining API compatibility.
func NewKVStore ¶
func NewKVStore(clientID, storePath string) (store *KVBTreeStore)
NewKVStore creates a store instance and load it with saved documents. Run Connect to start the background loop and Stop to end it.
SenderID service or user for debugging and logging storeFile path to storage file or "" for in-memory only
func (*KVBTreeStore) Close ¶
func (store *KVBTreeStore) Close() error
Close the store and stop the background update. If any changes are remaining then write to disk now.
func (*KVBTreeStore) Export ¶
func (store *KVBTreeStore) Export() map[string]map[string][]byte
export returns a map of the given bucket this copies the keys and values into a new map
func (*KVBTreeStore) GetBucket ¶
func (store *KVBTreeStore) GetBucket(bucketID string) (bucket buckets.IBucket)
GetBucket returns a bucket and creates it if it doesn't exist
func (*KVBTreeStore) Open ¶
func (store *KVBTreeStore) Open() error
Open the store and start the background loop for saving changes
func (*KVBTreeStore) SetWriteDelay ¶
func (store *KVBTreeStore) SetWriteDelay(delay time.Duration)
SetWriteDelay sets the delay for writing after a change
Source Files