Documentation
¶
Overview ¶
Package reduce reads messages from isb and is responsible for the first part of reduce subsystem. It is responsible for feeding in the data to the second part of the reduce subsystem called processAndForward and processes the read data from the ISB, writes to the outbound channel called PBQ so the message can be asynchronously processed by `processAndForward`, and then closes the partition if possible based on watermark progression. To write to the outbound channel (PBQ), it has to partition the message first. Hence, also contains partitioning logic. Partitioner identifies a set of elements with a common key and time, and buckets them in to a common window. A partition is uniquely identified using a tuple {window, key}. Type of window does not matter. Partitioner is responsible for managing the persistence and processing of each partition. It uses PBQ for durable persistence of elements that belong to a partition and orchestrates the processing of elements using processAndForward function. Partitioner tracks active partitions, closes the partitions based on watermark progression and co-ordinates the materialization and forwarding the results to the next vertex in the pipeline.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DataForward ¶
type DataForward struct {
// contains filtered or unexported fields
}
DataForward is responsible for reading and forwarding the message from ISB to PBQ.
func NewDataForward ¶
func NewDataForward(ctx context.Context, vertexInstance *dfv1.VertexInstance, fromBuffer isb.BufferReader, toBuffers map[string][]isb.BufferWriter, pbqManager *pbq.Manager, storeManager wal.Manager, whereToDecider forwarder.ToWhichStepDecider, fw fetch.Fetcher, watermarkPublishers map[string]publish.Publisher, windowingStrategy window.TimedWindower, idleManager wmb.IdleManager, of *pnf.ProcessAndForward, opts ...Option) (*DataForward, error)
NewDataForward creates a new DataForward
func (*DataForward) ReplayPersistedMessages ¶ added in v0.8.1
func (df *DataForward) ReplayPersistedMessages(ctx context.Context) error
ReplayPersistedMessages replays persisted messages, because during boot up, it has to replay the WAL from the store, before it can start reading from ISB. ReplayPersistedMessages will return only after the replay has been completed.
func (*DataForward) ShutDown ¶ added in v0.8.1
func (df *DataForward) ShutDown(ctx context.Context)
ShutDown shutdowns the read-loop.
type Option ¶
func WithAllowedLateness ¶ added in v0.8.0
WithAllowedLateness sets allowedLateness
func WithReadBatchSize ¶
WithReadBatchSize sets the read batch size
type Options ¶
type Options struct {
// contains filtered or unexported fields
}
Options for forwarding the message
func DefaultOptions ¶
func DefaultOptions() *Options
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
partition
Package partition is a tuple containing (start, end) time and an optional slot.
|
Package partition is a tuple containing (start, end) time and an optional slot. |
|
wal/aligned/fs
Package fs implements write-ahead-log on FileSystem.
|
Package fs implements write-ahead-log on FileSystem. |
|
wal/unaligned/fs
Package fs implements write-ahead-log on FileSystem.
|
Package fs implements write-ahead-log on FileSystem. |