Documentation
¶
Overview ¶
Package indexer provides two different methods to run real time incremental indexing for entities.
This package asks you to implement an Indexer interface, which takes care of dependency generation (for e.g. update parent entity if child entity is modified), and document regeneration and reindexing (regenerate search document for both parent and child entities, and update them in search index).
This methodology allows for real time incremental indexing systems. They can be utilized as such:
Method 1: As demonstrated in social.go, you can call indexer.Run(ctx, numRoutines) in your backend server directly, so as entities get updated via calls to store, indexer would figure out entity dependencies, regenerate all the corresponding documents and index them in the search engine. Thus, this method provides an automatic real time updating index.
Method 2: Automatic dependency generation generally isn't complete. Some indexed documents might get stale, or might never be generated if their entities were never touched. This can be fixed by running a standalone indexing server, which would continuously loop over all the entities in the store, and run regeneration of docs and reindexing for all those docs in the search index. This provides a fool proof mechanism to keep store and search data in-sync.
I recommend using both the methods. Method 1 ensures real time updates and method 2 ensures eventual consistency.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func WaitForDone ¶
Types ¶
type Indexer ¶
type Indexer interface {
// OnUpdate is called when an entity is updated due to a Commit
// on either itself, or it's direct children. Note that each
// child entity would also be called with OnUpdate. This function
// should return the Entity Ids, which need regeneration.
OnUpdate(x.Entity) []x.Entity
// Regenerate would be called on entities which need to be reprocessed
// due to a change. The workflow is:
// store.Commit -> search.OnUpdate -> Regenerate
Regenerate(x.Entity) x.Doc
}
Indexer functions are called automatically by store operations. These functions are used to determine which entities need updating, and then re-generate their corresponding documents, which then get re-indexed into search engine, overwriting past (using versioning, if available) documents.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Incremental indexing server to continously regenerate and index entities to keep store and search in-sync.
Example ¶
package main
import (
"time"
_ "github.com/aslanides/gocrud/drivers/leveldb"
_ "github.com/aslanides/gocrud/drivers/memsearch"
"github.com/aslanides/gocrud/indexer"
"github.com/aslanides/gocrud/search"
"github.com/aslanides/gocrud/store"
"github.com/aslanides/gocrud/x"
)
type SimpleIndexer struct {
}
func (si SimpleIndexer) OnUpdate(e x.Entity) (result []x.Entity) {
result = append(result, e)
return result
}
func (si SimpleIndexer) Regenerate(e x.Entity) (rdoc x.Doc) {
rdoc.Id = e.Id
rdoc.Kind = e.Kind
rdoc.NanoTs = time.Now().UnixNano()
return rdoc
}
func main() {
store.Get().Init("/tmp/ldb_" + x.UniqueString(10))
search.Get().Init("memsearch")
indexer.Register("EntityKind", SimpleIndexer{})
server := indexer.NewServer(100, 5)
server.InfiniteLoop(30 * time.Minute)
// This would never exit.
// OR, you could also just run this once, if you're
// testing your setup.
server.LoopOnce()
server.Finish() // Finish is only useful when you're looping once.
}
Output:
func NewServer ¶
NewServer returns back a server which runs continously in a loop to find and re-index entities stored. You can control the amount of memory consumed by the server via buffer of pending entities in the channel, and the rate of processing of these entities via numRoutines.
func (*Server) InfiniteLoop ¶
InfiniteLoop would infinitely cycle over all entities in the store, waiting for wait duration after each cycle.
Source Files