gears

type Dataflow ¶

type Dataflow interface {
	fmt.Stringer

	Graph() graph.Graph
	Run(context.Context) error
}

Dataflow manages a DAG (Directed Acyclic Graph) of primitive Transforms, and the Dataset that the Transforms consume and produce. Each Dataflow is self-contained and isolated from any other Dataflow. The Dataflow owns the Datasets and Transforms, and they can be used by that Dataflow only.

func NewDataflow(o Option) Dataflow

type Dataset ¶

type Dataset interface {
	Edge() graph.Edge
}

Dataset is a collection of Element. A Dataset can contain either a bounded or unbounded number of elements. Bounded and unbounded Dataset are produced as the output of Transform, and can be passed as the inputs of other Transform. Some root Transforms produce bounded Dataset and others produce unbounded ones.

Each element in a Dataset has an associated timestamp. Sources assign timestamps to elements when they create Dataset, and other Transforms propagate these timestamps from their input to their output implicitly or explicitly.

func NewDataset(e graph.Edge) Dataset

func Parallel(df Dataflow, o parallel.Option, f DoFunction, inputs []Dataset, args ...any) []Dataset

type DoFunction ¶

type DoFunction func(inputs, outputs []source.Source, args ...any)

type Option ¶

type Option interface {
	fmt.Stringer

	Name() string

	WithName(name string) Option
}

func NewOption() Option