Documentation
¶
Index ¶
- Variables
- func Build(ctx context.Context, db *sql.DB, dockerClient *docker.Client, ...) (map[string]components.BuildMetadata, error)
- func CalculateStages(specification FlowSpecification) ([][]string, error)
- func Execute(ctx context.Context, db *sql.DB, dockerClient *docker.Client, flowID string) (map[string]components.ExecutionMetadata, error)
- func InsertFlow(db *sql.DB, component FlowMetadata) error
- func ReadMountConfiguration(reader io.Reader) (map[string][]components.MountConfiguration, error)
- type FlowMetadata
- type FlowSpecification
Constants ¶
This section is empty.
Variables ¶
var ErrCyclicDependency = errors.New("Cyclic dependency detected in given flow")
ErrCyclicDependency is returned when flow dependency resolution fails because there was a cycle in the dependency graph.
var ErrEmptyID = errors.New("ID must be a non-empty string")
ErrEmptyID signifies that a caller attempted to create component metadata in which the ID string was the empty string
var ErrEmptySpecificationPath = errors.New("SpecificationPath must be a non-empty string")
ErrEmptySpecificationPath signifies that a caller attempted to create component metadata in which the SpecificationPath string was the empty string
var ErrFlowNotFound = errors.New("Could not find the specified flow")
ErrFlowNotFound - signifies that a single row lookup against a state database returned no rows
Functions ¶
func Build ¶
func Build(ctx context.Context, db *sql.DB, dockerClient *docker.Client, outstream io.Writer, flowID string) (map[string]components.BuildMetadata, error)
Build - Builds images for each component of a given flow
func CalculateStages ¶
func CalculateStages(specification FlowSpecification) ([][]string, error)
CalculateStages calculates stages for the execution of the flow with the given specification. Each stage is an array of flow steps which can be executed concurrently (although they do not have to be)
func Execute ¶
func Execute( ctx context.Context, db *sql.DB, dockerClient *docker.Client, flowID string, ) (map[string]components.ExecutionMetadata, error)
Execute - Executes the given builds of each step in a workflow in an order which respects the dependencies between steps
func InsertFlow ¶
func InsertFlow(db *sql.DB, component FlowMetadata) error
InsertFlow creates a new row in the components table with the given component information.
func ReadMountConfiguration ¶
func ReadMountConfiguration(reader io.Reader) (map[string][]components.MountConfiguration, error)
ReadMountConfiguration reads mount configurations for each step of a shnorky flow. The flow mount configuration is expected to be a JSON object, the keys of which are steps in the flow, and the values of which are mount configuration arrays for the corresponding components.
Types ¶
type FlowMetadata ¶
type FlowMetadata struct {
ID string `json:"id"`
SpecificationPath string `json:"specification_path"`
CreatedAt time.Time `json:"created_at"`
}
FlowMetadata - the metadata about a flow that gets stored in the state database
func AddFlow ¶
func AddFlow(db *sql.DB, id, specificationPath string) (FlowMetadata, error)
AddFlow registers a flow (by metadata) against a shnorky state database. It validates the specification at the given path first. This is the handler for `shnorky flows add`
func GenerateFlowMetadata ¶
func GenerateFlowMetadata(id, specificationPath string) (FlowMetadata, error)
GenerateFlowMetadata creates a FlowMetadata instance from the specified parameters, applying defaults as required and reasonable.
func SelectFlowByID ¶
func SelectFlowByID(db *sql.DB, id string) (FlowMetadata, error)
SelectFlowByID gets flow metadata from the given state database using the given ID. If no flow with the given ID is found, returns ErrFlowNotFound in the error position.
type FlowSpecification ¶
type FlowSpecification struct {
// Steps indexes each step in the flow and maps step names to component IDs
Steps map[string]string `json:"steps"`
// Dependencies has step names as its keys and the corresponding value are the names of steps
// that the key step depends on. Steps which have no dependencies need not be included in this
// map
Dependencies map[string][]string `json:"dependencies"`
// Stages denotes the sequence in which steps will execute. Steps appearing in the same stage
// can be run in parallel.
Stages [][]string `json:"stages,omitempty"`
// Mounts maps each step (by name) to mount configurations for its corresponding component
Mounts map[string][]components.MountConfiguration `json:"mounts"`
// Env maps each step (by name) to environment variable mappings (key-value mappings of variable
// name to variable value) for that step. The environment variable values get materialized
// following the same rules as values in a component runtime specification.
Env map[string]map[string]string `json:"env,omitempty"`
}
FlowSpecification - struct specifying a shnorky data processing flow
func MaterializeFlowSpecification ¶
func MaterializeFlowSpecification(rawSpecification FlowSpecification) (FlowSpecification, error)
MaterializeFlowSpecification takes a raw FlowSpecification struct and returns a materialized one in which the members of the raw specification have been validated and special values have been rendered.
func ReadSingleSpecification ¶
func ReadSingleSpecification(reader io.Reader) (FlowSpecification, error)
ReadSingleSpecification reads a single ComponentSpecification JSON document and returns the corresponding ComponentSpecification struct. It returns an error if there was an issue parsing the specification into the struct.
Source Files