Documentation
¶
Index ¶
Constants ¶
const QueueSize = 48
QueueSize is the size of the buffer on the input and output channels for the pool.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Pool ¶
Pool takes in parser.Parseable objects, produces MixpanelEvents from them, and writes those events to S3.
type RequestConverter ¶
type RequestConverter struct {
// contains filtered or unexported fields
}
RequestConverter parses Parseables that come in into MixpanelEvents.
func (*RequestConverter) Listen ¶
func (p *RequestConverter) Listen()
Listen sits in a loop Processing requests until the RequestConverter is Closed.
func (*RequestConverter) Process ¶
func (p *RequestConverter) Process(r parser.Parseable) (events []parser.MixpanelEvent, err error)
Process parses the given Parseable into a list of events.
func (*RequestConverter) Wait ¶
func (p *RequestConverter) Wait()
Wait waits for the input channel to flush.
type RequestTransformer ¶
type RequestTransformer struct {
// contains filtered or unexported fields
}
RequestTransformer transforms MixpanelEvents and writes them out to a SpadeWriter.
func (*RequestTransformer) Close ¶
func (p *RequestTransformer) Close()
Close stops the transformer's Listen() method.
func (*RequestTransformer) Listen ¶
func (p *RequestTransformer) Listen(w writer.SpadeWriter)
Listen listens for incoming events, transforms them, and writes them to the SpadeWriter.
func (*RequestTransformer) Process ¶
func (p *RequestTransformer) Process(e *parser.MixpanelEvent) (request *writer.WriteRequest)
Process transforms the given event into a WriteRequest.
type SpadeProcessorPool ¶
type SpadeProcessorPool struct {
// contains filtered or unexported fields
}
SpadeProcessorPool is pool of RequestConverters and RequestTransformers.
func BuildProcessorPool ¶
func BuildProcessorPool(schemaConfigs transformer.SchemaConfigLoader, eventMetadataConfigs transformer.EventMetadataConfigLoader, rep reporter.Reporter, writer writer.SpadeWriter, stats reporter.StatsLogger) *SpadeProcessorPool
BuildProcessorPool builds a new SpadeProcessorPool.
func (*SpadeProcessorPool) Close ¶
func (p *SpadeProcessorPool) Close()
Close closes all converters and trnasformers in the pool. Important: Ensure pool is drained before calling close.
func (*SpadeProcessorPool) Process ¶
func (p *SpadeProcessorPool) Process(request parser.Parseable)
Process submits the given Parseable to the pool for converting/transforming.
func (*SpadeProcessorPool) StartListeners ¶
func (p *SpadeProcessorPool) StartListeners()
StartListeners starts up goroutines for the converters and transformers.
Source Files