driver

package
v2.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 15, 2024 License: Apache-2.0 Imports: 35 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func PreprocessingTaskNamespace added in v2.9.0

func PreprocessingTaskNamespace(pipelineInfo *pps.PipelineInfo) string

func ProcessingTaskNamespace added in v2.9.0

func ProcessingTaskNamespace(pipelineInfo *pps.PipelineInfo) string

Types

type Driver

type Driver interface {
	Jobs() col.PostgresCollection
	Pipelines() col.PostgresCollection

	NewTaskSource() task.Source
	NewPreprocessingTaskDoer(string, task.Cache) task.Doer
	NewProcessingTaskDoer(string, task.Cache) task.Doer

	// Returns the PipelineInfo for the pipeline that this worker belongs to
	PipelineInfo() *pps.PipelineInfo

	// Returns the kubernetes namespace that the worker is deployed in
	Namespace() string

	// Returns the path that will contain the input filesets for the job
	InputDir() string

	// Returns the pachd API client for the driver
	PachClient() *client.APIClient

	// Returns the number of workers to be used
	ExpectedNumWorkers() (int64, error)

	// WithContext clones the current driver and applies the context to its
	// pachClient. The pachClient context will be used for other blocking
	// operations as well.
	WithContext(context.Context) Driver

	// WithActiveData swaps the given scratch directory into the 'active' input
	// directory used when running user code. This also locks a mutex so that no
	// two datums can be active concurrently.
	WithActiveData([]*common.Input, string, func() error) error

	// UserCodeEnv returns the set of environment variables to construct when
	// launching the configured user process.
	UserCodeEnv(string, *pfs.Commit, []*common.Input, string) []string

	RunUserCode(context.Context, logs.TaggedLogger, []string) error

	RunUserErrorHandlingCode(context.Context, logs.TaggedLogger, []string) error

	// TODO: provide a more generic interface for modifying jobs, and
	// some quality-of-life functions for common operations.
	DeleteJob(context.Context, *pachsql.Tx, *pps.JobInfo) error
	UpdateJobState(*pps.Job, pps.JobState, string) error

	GetJobInfo(job *pps.Job) (*pps.JobInfo, error)

	// TODO: figure out how to not expose this - currently only used for a few
	// operations in the map spawner
	NewSQLTx(func(context.Context, *pachsql.Tx) error) error

	// Returns the image ID associated with a container running in the worker pod
	GetContainerImageID(context.Context, string) (string, error)
}

Driver provides an interface for common functions needed by worker code, and captures the relevant objects necessary to provide these functions so that users do not need to keep track of as many variables. In addition, this interface can be used to mock out external calls to make unit-testing simpler.

func NewDriver

func NewDriver(
	ctx context.Context,
	env serviceenv.ServiceEnv,
	pachClient *client.APIClient,
	pipelineInfo *pps.PipelineInfo,
	rootPath string,
) (Driver, error)

NewDriver constructs a Driver object using the given clients and pipeline settings. It makes blocking calls to determine the user/group to use with the user code on the current worker node, as well as determining if enterprise features are activated (for exporting stats).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL