Documentation ¶
Overview ¶
Package plan is the primary interface to the query planner.
The New and NewSplit functions can be used to create query plans from AST expressions. The NewSplit function creates a query plan that is intended for parallel evaluation on multiple independent machines.
Index ¶
- Variables
- func AddTransportDecoder(typ string, fn func() TransportDecoder)
- func EncodeTransport(t Transport, st *ion.Symtab, buf *ion.Buffer) error
- func Exec(ep *ExecParams) error
- func Graphviz(t *Tree, dst io.Writer) error
- func Serve(rw io.ReadWriteCloser, run Runner) error
- type CachedTable
- type Client
- type CountStar
- type Descriptor
- type Distinct
- type DummyOutput
- type Env
- type ExecParams
- type ExecStats
- type Explain
- type FSRunner
- type Filter
- type Geometry
- type HashAggregate
- type HashOrder
- type Hints
- type IfMatcher
- type Index
- type Indexer
- type Input
- func (in *Input) Append(other *Input)
- func (in *Input) Blocks() int
- func (in *Input) CanPartition(part string) bool
- func (in *Input) CompressedSize() (n int64)
- func (in *Input) Empty() bool
- func (in *Input) Encode(dst *ion.Buffer, st *ion.Symtab)
- func (in *Input) Filter(e expr.Node) *Input
- func (in *Input) HashSplit(n int) []*Input
- func (in *Input) Partition(parts []string) (*InputGroups, bool)
- func (in *Input) Size() (n int64)
- type InputGroups
- type Leaf
- type Limit
- type LocalTransport
- type NoOutput
- type Node
- type Nonterminal
- type Op
- type OrderBy
- type OutputIndex
- type OutputPart
- type PartGroups
- type Project
- type Result
- type ResultSet
- type Runner
- type Schemer
- type Server
- type SimpleAggregate
- type SplitEnv
- type Substitute
- type Table
- type TableLister
- type Transport
- type TransportDecoder
- type Tree
- type UnionMap
- type UnionPartition
- type Unnest
- type Unpivot
- type UnpivotAtDistinct
- type UploadEnv
- type UploadFS
Constants ¶
This section is empty.
Variables ¶
var (
ErrNotSupported = errors.New("plan: query not supported")
)
Functions ¶
func AddTransportDecoder ¶
func AddTransportDecoder(typ string, fn func() TransportDecoder)
AddTransportDecoder adds function which returns a new TransportDecoder for the given type name.
func EncodeTransport ¶
EncodeTransport attempts to encode t to buf. If t cannot be serialized, this returns an error.
func Exec ¶
func Exec(ep *ExecParams) error
Exec executes a query plan using the parameters provided in [ep].
Types ¶
type CachedTable ¶
CachedTable is an interface optionally implemented by a vm.Table. If a vm.Table returned by TableHandle.Open implements CachedTable, then the returned Hits and Misses statistics will be added to ExecStats.CacheHits and ExecStats.CacheMisses, respectively.
type Client ¶
type Client struct { // Pipe is the connection to the // remote query environment. Pipe io.ReadWriteCloser // contains filtered or unexported fields }
Client represents a connection to a "remote" query-processing environment.
A Client can be constructed simply by declaring a zero-value Client and then assigning the Pipe field to the desired connection.
func (*Client) Exec ¶
func (c *Client) Exec(ep *ExecParams) error
Exec executes a query across the client connection. Exec implements Transport.Exec.
Exec is *not* safe to call from multiple goroutines simultaneously.
type CountStar ¶
type CountStar struct { Nonterminal As string // output count name NonEmpty bool // don't output count=0 }
CountStar implements COUNT(*)
type Descriptor ¶
type Descriptor struct { // Descriptor is the input object descriptor. // // The query planner uses the contents of Descriptor.Trailer // for the purposes of query planning, but otherwise leaves // the semantics of Descriptor.{Path,ETag,...} up to the // [Runner] used to execute the query. blockfmt.Descriptor // Blocks indicates the list of blocks within // the object that are actually referenced. Blocks ints.Intervals }
A Descriptor describes a single input object.
func (*Descriptor) CompressedSize ¶
func (d *Descriptor) CompressedSize() (n int64)
CompressedSize returns the number of compressed bytes that comprise all of the input blocks.
func (*Descriptor) Decode ¶
func (d *Descriptor) Decode(td *blockfmt.TrailerDecoder, v ion.Datum) error
func (*Descriptor) Empty ¶
func (d *Descriptor) Empty() bool
Empty is equivalent to
len(d.Blocks) == 0
func (*Descriptor) Size ¶
func (d *Descriptor) Size() (n int64)
Size returns the decompressed size of all the data referenced by [d].
type DummyOutput ¶
type DummyOutput struct{}
func (DummyOutput) String ¶
func (n DummyOutput) String() string
type Env ¶
type Env interface { // Stat returns a *Input // associated with the given PartiQL expression. // The Hints provided in the second // argument can be used to constrain the set of // rows and columns that are present in the returned *Input. // The information provided by the *Input // is used by the query planner to make query-splitting // decisions. Stat(tbl expr.Node, h *Hints) (*Input, error) }
Env represents the global binding environment at the time that the query was compiled
type ExecParams ¶
type ExecParams struct { // Plan is the query plan being executed. Plan *Tree // Output is the destination of the query output. Output io.Writer // Stats are stats that are collected // during query execution. Stats ExecStats // Parallel determines the (local) parallelism // of plan execution. If Parallel is unset, then // runtime.GOMAXPROCS(0) is used instead. Parallel int // Rewriter is a rewrite that should be applied // to each expression in the query plan before // the query begins execution. Rewriter expr.Rewriter // Context indicates the cancellation scope // of the query. Transports are expected to // stop processing queries after Context is canceled. Context context.Context // Runner is the local execution environment // for the query. If Runner is nil, then query // execution will fail. Runner Runner // FS is the file system to read inputs from. // This may implement UploadFS, which is // required to enable support for SELECT INTO. FS fs.FS // contains filtered or unexported fields }
ExecParams is a collection of all the runtime parameters for a query.
func (*ExecParams) AddRewrite ¶
func (ep *ExecParams) AddRewrite(r expr.Rewriter)
AddRewrite adds a rewrite to ep.Rewriter. Each rewrite added via AddRewrite is executed on the results produced by rewriters added from preceding calls to AddRewrite.
func (*ExecParams) PopRewrite ¶
func (ep *ExecParams) PopRewrite()
PopRewrite removes the most-recently-added Rewriter added via ep.AddRewrite.
type ExecStats ¶
type ExecStats struct {
// CacheHits and CacheMisses
// are the sum of the results
// of CachedTable.Hits() and CachedTable.Misses(),
// respectively.
//
// NOTE: see tenant/dcache.Stats
// for a detailed description of how
// we do bookkeeping for cache statistics.
CacheHits, CacheMisses int64
// BytesScanned is the number
// of bytes scanned.
BytesScanned int64
}
ExecStats is a collection of statistics that are aggregated during the execution of a query.
func (*ExecStats) Marshal ¶
Marshal is identical to Encode except that it uses the same symbol table that UnmarshalBinary expects will be used.
func (*ExecStats) UnmarshalBinary ¶
UnmarshalBinary implements encoding.BinaryUnmarshaler
type Explain ¶
type Explain struct { Format expr.ExplainFormat Query *expr.Query Tree *Tree }
Explain is leaf executor for explaining queries
type Filter ¶
type Filter struct { Nonterminal Expr expr.Node }
Filter is a plan that filters the input rows on some set of criteria
type Geometry ¶
type Geometry struct {
Peers []Transport
}
Geometry represents the shape of a distributed query in terms of the peers that are available for dispatching partial queries.
type HashAggregate ¶
type HashAggregate struct { Nonterminal Agg vm.Aggregation By vm.Selection Windows vm.Aggregation Limit int OrderBy []HashOrder NonEmpty bool }
func (*HashAggregate) String ¶
func (h *HashAggregate) String() string
type HashOrder ¶
type HashOrder struct { Column int Ordering vm.SortOrdering }
type Hints ¶
type Hints struct { // Filter, if non-nil, is a predicate // that is applied to every row of the table. // (Env.Stat may use this to produce a *Input // that produces fewer rows than it otherwise would // due to the presence of some secondary indexing information.) Filter expr.Node // Fields is a list of top-level record fields explicitly // referenced by the query. The list of fields will always // be in lexicographical order. Fields []string // AllFields is set to true if all of the fields // are implicitly referenced in the query (i.e. via "*"); // otherwise it is set to false. AllFields bool }
Hints describes a set of hints passed to Env.Stat that can be used to optimize the access to a table.
type IfMatcher ¶
type IfMatcher interface { // IfMatch is called to specify that the file // should be checked against the given ETag // and return an error if it doesn't match. // // If the ETag does not match, the error may // be returned at the time IfMatch is called, // on read, or at close time. IfMatch(etag string) error }
IfMatcher can be implemented by a file that supports ETag matching using semantics compatible with the HTTP "If-Match" header.
type Index ¶
An Index may be returned by Indexer.Index to provide additional table metadata that may be used during optimization.
type Indexer ¶
type Indexer interface { // Index returns the index for the given table // expression. This may return (nil, nil) if // the index for the table is not available. Index(expr.Node) (Index, error) }
Indexer may optionally be implemented by Env to provide an index for a table.
type Input ¶
type Input struct { // Descs is the list of referenced descriptors // in the input. Descs []Descriptor // Fields are the fields in the input that are // needed by the query. If nil, all fields are // assumed to be needed. To specify zero // fields, use a non-nil slice of length 0. Fields []string // contains filtered or unexported fields }
Input represents a collection of input objects. The zero value of Input represents an empty table.
func (*Input) CanPartition ¶
CanPartition indicates whether a call to Partition including the given part would be successful.
func (*Input) CompressedSize ¶
CompressedSize returns the number of compressed bytes that comprise all of the input blocks.
func (*Input) Filter ¶
Filter returns an equivalent of [in] which contains only the blocks for which [e] may evaluate to TRUE. This will return a distinct object if [e] would exclude any of the blocks referenced by [in], but may simply return [in] if the filtered result would be identical. This method will not mutate [in].
func (*Input) HashSplit ¶
HashSplit splits the input [in] into [n] groups deterministically based on the ETags within [in.Descs].
The resulting slice may contain nil pointers if no blocks were assigned to that slot.
type InputGroups ¶
type InputGroups struct {
// contains filtered or unexported fields
}
InputGroups is a set of [Input]s partitioned along a particular axis. InputGroups can be queried efficiently for a particular group with [Get] or it can be used to iterate all the groups individually with [Each].
func (*InputGroups) Each ¶
func (in *InputGroups) Each(fn func(parts []ion.Datum, i *Input))
Each calls [fn] on each unique group. The [parts] slice indicates the constants that are associated with [i.Fields] for each Input [in].
func (*InputGroups) Fields ¶
func (in *InputGroups) Fields() []string
Fields returns the list of fields over which the input has been grouped.
func (*InputGroups) Get ¶
func (in *InputGroups) Get(equal []ion.Datum) *Input
Get returns the Input associated with the given partition. Get will return nil if there is no data associated with the given partition constraints.
func (*InputGroups) Groups ¶
func (in *InputGroups) Groups() int
Groups returns the number of unique groups in [in].
type Leaf ¶
type Leaf struct { // Orig is the original table expression; // this exists mostly for presentation purposes. Orig *expr.Table // Filter is pushed down before exec if the // parent node supports filter pushdown. Filter expr.Node // OnEqual is a filtering operation that applies // specifically to partitions. The table will // only be iterated for partitions where // // OnEqual[i] = PARTITION_VALUE(i) // // Since OnEqual depends on PARTITION_VALUE(i) being rewritten, // OnEqual is only present when Leaf is part of a sub-query // for a partitioned query. OnEqual []string EqualExpr []expr.Node }
Leaf is the leaf of a plan tree, and just contains the table from which the data will be processed.
type Limit ¶
type Limit struct { Nonterminal Num int64 }
type LocalTransport ¶
type LocalTransport struct { // Threads is the number of threads // used for query evaluation. // If Threads is <= 0, then runtime.GOMAXPROCS // is used. Threads int }
LocalTransport is a Transport that executes queries locally.
func (*LocalTransport) Exec ¶
func (l *LocalTransport) Exec(ep *ExecParams) error
Exec implements Transport.Exec
type Node ¶
type Node struct { // OutputType is the type of // the output column(s) of the // sub-query produced by this tree. // Note that we cannot always infer // the output types of every query. // For example, 'SELECT * ...' on data // without a schema does not produce // a known ResultSet. OutputType ResultSet // Input is the original input associated with this Op. Input int // Op is the first element of a linked list // of query execution steps. The linked list // is encoded in reverse-execution-order, so // Op is the last step in execution order, // and the terminal element of the list // is the first in execution order. Op Op }
A Node is one node of a query plan tree and contains the operation sequence for one step of the plan, as well as links to subtrees this step of the plan depends on.
Simple operations like filtering, aggregation, extended projection, etc. are grouped into sequences, and relational operations and sub-queries are handled by connecting their constituent subsequences together into a Node.
(One motivating analogy might be that of basic blocks within a control flow graph, except that we restrict the vertices to form a tree rather than any directed graph.)
type Nonterminal ¶
type Nonterminal struct {
From Op
}
Nonterminal is embedded in every Op that has an input Op.
type Op ¶
type Op interface { fmt.Stringer ion.FieldSetter // contains filtered or unexported methods }
Op represents a single node in the query plan tree. The root of the plan tree is the final output node, and the leaves are the tables from which the data will be queried.
type OrderBy ¶
type OrderBy struct { Nonterminal Columns []vm.SortColumn Limit int Offset int }
OrderBy implements ORDER BY clause (without GROUP BY).
type OutputIndex ¶
type OutputIndex struct { Nonterminal DB, Table string Basename string Key *blockfmt.Key }
OutputIndex is a nonterminal plan node that accepts rows from OutputPart and collects them into an Index object. OutputIndex writes one output row containing the autogenerated table name.
func (*OutputIndex) String ¶
func (o *OutputIndex) String() string
type OutputPart ¶
type OutputPart struct { Nonterminal Basename string }
OutputPart is a nonterminal plan node that produces one blockfmt.Descriptor row for each thread of execution that points to an uploaded file containing all the data written into this node.
func (*OutputPart) String ¶
func (o *OutputPart) String() string
type PartGroups ¶
type PartGroups[T any] struct { // contains filtered or unexported fields }
PartGroups contains a list of values that are grouped by sets of ion datum constants. PartGroups is constructed by the Partition function.
func Partition ¶
func Partition[T any](lst []T, parts []string, getconst func(T, string) (ion.Datum, bool)) (*PartGroups[T], bool)
Partition groups a list of items by associated constants.
func (*PartGroups[T]) Each ¶
func (p *PartGroups[T]) Each(fn func(parts []ion.Datum, group []T))
Each iterates all the groups within p and calls fn for each group. Each parts[i] corresponds to fields[i] returned from [p.Fields].
func (*PartGroups[T]) Fields ¶
func (p *PartGroups[T]) Fields() []string
Fields returns the ordered list of named fields used to group the associated values.
func (*PartGroups[T]) Get ¶
func (p *PartGroups[T]) Get(equal []ion.Datum) []T
Get returns the values associated with the partition for which parts[i]==equal[i] for each part label given by p.Fields. The result of Get is unspecified if len(equal) is not equal to len(p.Fields())
func (*PartGroups[T]) Groups ¶
func (p *PartGroups[T]) Groups() int
Groups returns the number of distinct groups in the set.
type Project ¶
type Project struct { Nonterminal Using []expr.Binding }
Project is a plan Op that projects and re-names fields
type Result ¶
Result is a (field, type) tuple that indicates the possible output encoding of a particular field
type Runner ¶
type Runner interface { // Run should write the contents of src into dst // and update ep.Stats as it does so. Run(dst vm.QuerySink, src *Input, ep *ExecParams) error }
Runner is the caller-provided interface through which table data is actually written into a vm.QuerySink. This interface exists in order to allow callers to have fine-grained control over data access patterns during query plan execution (for e.g. caching, etc.)
See also ExecParams.
type Schemer ¶
type Schemer interface { // Schema returns type hints associated // with a particular table expression. // In the event that there is no available // type information, Schema may return nil. Schema(expr.Node) expr.Hint }
Schemer may optionally be implemented by Env to provide type hints for a table.
type Server ¶
type Server struct { // Runner is the local execution environment // for the query. If Runner is nil, then query // execution will fail. Runner Runner // InitFS is used to initialize a file system // which will be used by the Runner to access // input objects. The datum passed to InitFS // is provided by the client to pass // appropriate information necessary to access // file system (e.g., credentials). InitFS func(ion.Datum) (fs.FS, error) }
A Server can be used to serve queries.
type SimpleAggregate ¶
type SimpleAggregate struct { Nonterminal Outputs vm.Aggregation NonEmpty bool }
SimpleAggregate computes aggregates for a list of fields
func (*SimpleAggregate) String ¶
func (s *SimpleAggregate) String() string
type SplitEnv ¶
SplitEnv is an Env that can be used for planning distributed queries by supplying a Geometry.
type Substitute ¶
type Substitute struct { Nonterminal // Inner is the list of sub-queries that need // their results substituted into the input // of this Substitute Op. The order of Inner elements // is important, as each Inner node i is used to substitute // results into the *REPLACEMENT(i) expressions. Inner []*Node }
Substitute is an Op that substitutes the result of executing a list of Nodes into its input Op.
type TableLister ¶
type TableLister interface { // ListTables returns the names of tables in // the given db. Callers must not modify the // returned list. ListTables(db string) ([]string, error) }
TableLister is an interface an Env or Index can optionally implement to support TABLE_GLOB and TABLE_PATTERN expressions.
type Transport ¶
type Transport interface { // Exec executes the provided query plan, // streaming the output of the query to ep.Output // (ep.Output may not be nil). // Each call to ep.Output.Write should contain exactly // one "chunk" of ion-encoded data, which will // begin with an ion BVM and be followed by zero // or more ion structures. // // The ep.Rewrite provided via ExecParams, if non-nil, // determines how table expressions are re-written // before they are provided to Transport. Exec(ep *ExecParams) error }
Transport models the exection environment of a query plan.
See LocalTransport for executing queries locally. See Client for executing queries remotely.
type TransportDecoder ¶
type TransportDecoder interface { ion.FieldSetter Transport }
TransportDecoder is a transport that can be decoded from an ion struct.
type Tree ¶
type Tree struct { // ID is an opaque ID assigned to this query by the caller. ID string // Inputs is the global list of inputs for the tree. // Each [Node.Input] references an element of this array. // // (These are stored globally so that the same table // referenced multiple times does not consume extra space.) Inputs []*Input // Data is arbitrary data that can be included // along with the tree during serialization. Data ion.Datum // Root is the root node of the plan tree. Root Node Results []expr.Binding ResultTypes []expr.TypeSet }
A Tree is the root an executable query plan tree as well as the inputs for the plan.
A Tree can be constructed with New or NewSplit and it can be executed with Exec or Transport.Exec.
func Decode ¶
Decode decodes an ion-encoded tree from the provided symbol table and buffer. During decoding, each Leaf op in the Tree will have its TableHandle populated with env.Stat. See also: Plan.Encode, Plan.EncodePart.
func (*Tree) MaxScanned ¶
MaxScanned returns the maximum number of scanned bytes for this query plan by traversing the plan tree and adding TableHandle.Size bytes for each table reference.
type UnionMap ¶
type UnionMap struct { Nonterminal // Geometry determines how table handle inputs // are distributed onto the constituent partials Geometry *Geometry }
UnionMap is an op that gathers a collection of sub-operations and yields their results (in any order, and without deduplication)
type UnionPartition ¶
type UnionPartition struct { Nonterminal By []string }
func (*UnionPartition) String ¶
func (u *UnionPartition) String() string
type Unnest ¶
type Unnest struct { Nonterminal // source op Expr expr.Node Result string }
Unnest joins a row on a list-like field within that row and computes a projection plus an optional conditional clause
type Unpivot ¶
type Unpivot struct { Nonterminal As *string At *string }
type UnpivotAtDistinct ¶
type UnpivotAtDistinct struct { Nonterminal At string }
func (*UnpivotAtDistinct) String ¶
func (u *UnpivotAtDistinct) String() string