syntax

package
v0.0.0-...-3a5c849 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 20, 2020 License: BSD-3-Clause Imports: 10 Imported by: 0

Documentation

Overview

Package syntax provides a Starlark parser and abstract syntax tree.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Walk

func Walk(n Node, f func(Node) bool)

Walk traverses a syntax tree in depth-first order. It starts by calling f(n); n must not be nil. If f returns true, Walk calls itself recursively for each non-nil child of n. Walk then calls f(nil).

Example

ExampleWalk demonstrates the use of Walk to enumerate the identifiers in a Starlark source file containing a nonsense program with varied grammar.

package main

import (
	"fmt"
	"log"
	"strings"

	"github.com/k14s/starlark-go/syntax"
)

func main() {
	const src = `
load("library", "a")

def b(c, *, d=e):
    f += {g: h}
    i = -(j)
    return k.l[m + n]

for o in [p for q, r in s if t]:
    u(lambda: v, w[x:y:z])
`
	f, err := syntax.Parse("hello.star", src, 0)
	if err != nil {
		log.Fatal(err)
	}

	var idents []string
	syntax.Walk(f, func(n syntax.Node) bool {
		if id, ok := n.(*syntax.Ident); ok {
			idents = append(idents, id.Name)
		}
		return true
	})
	fmt.Println(strings.Join(idents, " "))

	// The identifer 'a' appears in both LoadStmt.From[0] and LoadStmt.To[0].

}
Output:

a a b c d e f g h i j k l m n o p q r s t u v w x y z

Types

type AssignStmt

type AssignStmt struct {
	OpPos Position
	Op    Token // = EQ | {PLUS,MINUS,STAR,PERCENT}_EQ
	LHS   Expr
	RHS   Expr
	// contains filtered or unexported fields
}

An AssignStmt represents an assignment:

x = 0
x, y = y, x
x += 1

func (*AssignStmt) AllocComments

func (cr *AssignStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (AssignStmt) Comments

func (cr AssignStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*AssignStmt) Span

func (x *AssignStmt) Span() (start, end Position)

type BinaryExpr

type BinaryExpr struct {
	X     Expr
	OpPos Position
	Op    Token
	Y     Expr
	// contains filtered or unexported fields
}

A BinaryExpr represents a binary expression: X Op Y.

As a special case, BinaryExpr{Op:EQ} may also represent a named argument in a call f(k=v) or a named parameter in a function declaration def f(param=default).

func (*BinaryExpr) AllocComments

func (cr *BinaryExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (BinaryExpr) Comments

func (cr BinaryExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*BinaryExpr) Span

func (x *BinaryExpr) Span() (start, end Position)

type BranchStmt

type BranchStmt struct {
	Token    Token // = BREAK | CONTINUE | PASS
	TokenPos Position
	// contains filtered or unexported fields
}

A BranchStmt changes the flow of control: break, continue, pass.

func (*BranchStmt) AllocComments

func (cr *BranchStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (BranchStmt) Comments

func (cr BranchStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*BranchStmt) Span

func (x *BranchStmt) Span() (start, end Position)

type CallExpr

type CallExpr struct {
	Fn     Expr
	Lparen Position
	Args   []Expr // arg = expr | ident=expr | *expr | **expr
	Rparen Position
	// contains filtered or unexported fields
}

A CallExpr represents a function call expression: Fn(Args).

func (*CallExpr) AllocComments

func (cr *CallExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (CallExpr) Comments

func (cr CallExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*CallExpr) Span

func (x *CallExpr) Span() (start, end Position)

type Comment

type Comment struct {
	Start Position
	Text  string // without trailing newline
}

A Comment represents a single # comment.

type Comments

type Comments struct {
	Before []Comment // whole-line comments before this expression
	Suffix []Comment // end-of-line comments after this expression (up to 1)

	// For top-level expressions only, After lists whole-line
	// comments following the expression.
	After []Comment
}

Comments collects the comments associated with an expression.

type Comprehension

type Comprehension struct {
	Curly   bool // {x:y for ...} or {x for ...}, not [x for ...]
	Lbrack  Position
	Body    Expr
	Clauses []Node // = *ForClause | *IfClause
	Rbrack  Position
	// contains filtered or unexported fields
}

A Comprehension represents a list or dict comprehension: [Body for ... if ...] or {Body for ... if ...}

func (*Comprehension) AllocComments

func (cr *Comprehension) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (Comprehension) Comments

func (cr Comprehension) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*Comprehension) Span

func (x *Comprehension) Span() (start, end Position)

type CondExpr

type CondExpr struct {
	If      Position
	Cond    Expr
	True    Expr
	ElsePos Position
	False   Expr
	// contains filtered or unexported fields
}

CondExpr represents the conditional: X if COND else ELSE.

func (*CondExpr) AllocComments

func (cr *CondExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (CondExpr) Comments

func (cr CondExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*CondExpr) Span

func (x *CondExpr) Span() (start, end Position)

type DefStmt

type DefStmt struct {
	Def    Position
	Name   *Ident
	Params []Expr // param = ident | ident=expr | * | *ident | **ident
	Body   []Stmt

	Function interface{} // a *resolve.Function, set by resolver
	// contains filtered or unexported fields
}

A DefStmt represents a function definition.

func (*DefStmt) AllocComments

func (cr *DefStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (DefStmt) Comments

func (cr DefStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*DefStmt) Span

func (x *DefStmt) Span() (start, end Position)

type DictEntry

type DictEntry struct {
	Key   Expr
	Colon Position
	Value Expr
	// contains filtered or unexported fields
}

A DictEntry represents a dictionary entry: Key: Value. Used only within a DictExpr.

func (*DictEntry) AllocComments

func (cr *DictEntry) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (DictEntry) Comments

func (cr DictEntry) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*DictEntry) Span

func (x *DictEntry) Span() (start, end Position)

type DictExpr

type DictExpr struct {
	Lbrace Position
	List   []Expr // all *DictEntrys
	Rbrace Position
	// contains filtered or unexported fields
}

A DictExpr represents a dictionary literal: { List }.

func (*DictExpr) AllocComments

func (cr *DictExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (DictExpr) Comments

func (cr DictExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*DictExpr) Span

func (x *DictExpr) Span() (start, end Position)

type DotExpr

type DotExpr struct {
	X       Expr
	Dot     Position
	NamePos Position
	Name    *Ident
	// contains filtered or unexported fields
}

A DotExpr represents a field or method selector: X.Name.

func (*DotExpr) AllocComments

func (cr *DotExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (DotExpr) Comments

func (cr DotExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*DotExpr) Span

func (x *DotExpr) Span() (start, end Position)

type Error

type Error struct {
	Pos Position
	Msg string
}

An Error describes the nature and position of a scanner or parser error.

func (Error) Error

func (e Error) Error() string

type Expr

type Expr interface {
	Node
	// contains filtered or unexported methods
}

An Expr is a Starlark expression.

func ParseExpr

func ParseExpr(filename string, src interface{}, mode Mode) (expr Expr, err error)

ParseExpr parses a Starlark expression. A comma-separated list of expressions is parsed as a tuple. See Parse for explanation of parameters.

type ExprStmt

type ExprStmt struct {
	X Expr
	// contains filtered or unexported fields
}

An ExprStmt is an expression evaluated for side effects.

func (*ExprStmt) AllocComments

func (cr *ExprStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (ExprStmt) Comments

func (cr ExprStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*ExprStmt) Span

func (x *ExprStmt) Span() (start, end Position)

type File

type File struct {
	Path  string
	Stmts []Stmt

	Module interface{} // a *resolve.Module, set by resolver
	// contains filtered or unexported fields
}

A File represents a Starlark file.

func Parse

func Parse(filename string, src interface{}, mode Mode) (f *File, err error)

Parse parses the input data and returns the corresponding parse tree.

If src != nil, ParseFile parses the source from src and the filename is only used when recording position information. The type of the argument for the src parameter must be string, []byte, or io.Reader. If src == nil, ParseFile parses the file specified by filename.

func ParseCompoundStmt

func ParseCompoundStmt(filename string, readline func() ([]byte, error)) (f *File, err error)

ParseCompoundStmt parses a single compound statement: a blank line, a def, for, while, or if statement, or a semicolon-separated list of simple statements followed by a newline. These are the units on which the REPL operates. ParseCompoundStmt does not consume any following input. The parser calls the readline function each time it needs a new line of input.

func (*File) AllocComments

func (cr *File) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (File) Comments

func (cr File) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*File) Span

func (x *File) Span() (start, end Position)

type ForClause

type ForClause struct {
	For  Position
	Vars Expr // name, or tuple of names
	In   Position
	X    Expr
	// contains filtered or unexported fields
}

A ForClause represents a for clause in a list comprehension: for Vars in X.

func (*ForClause) AllocComments

func (cr *ForClause) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (ForClause) Comments

func (cr ForClause) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*ForClause) Span

func (x *ForClause) Span() (start, end Position)

type ForStmt

type ForStmt struct {
	For  Position
	Vars Expr // name, or tuple of names
	X    Expr
	Body []Stmt
	// contains filtered or unexported fields
}

A ForStmt represents a loop: for Vars in X: Body.

func (*ForStmt) AllocComments

func (cr *ForStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (ForStmt) Comments

func (cr ForStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*ForStmt) Span

func (x *ForStmt) Span() (start, end Position)

type Ident

type Ident struct {
	NamePos Position
	Name    string

	Binding interface{} // a *resolver.Binding, set by resolver
	// contains filtered or unexported fields
}

An Ident represents an identifier.

func (*Ident) AllocComments

func (cr *Ident) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (Ident) Comments

func (cr Ident) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*Ident) Span

func (x *Ident) Span() (start, end Position)

type IfClause

type IfClause struct {
	If   Position
	Cond Expr
	// contains filtered or unexported fields
}

An IfClause represents an if clause in a list comprehension: if Cond.

func (*IfClause) AllocComments

func (cr *IfClause) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (IfClause) Comments

func (cr IfClause) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*IfClause) Span

func (x *IfClause) Span() (start, end Position)

type IfStmt

type IfStmt struct {
	If      Position // IF or ELIF
	Cond    Expr
	True    []Stmt
	ElsePos Position // ELSE or ELIF
	False   []Stmt   // optional
	// contains filtered or unexported fields
}

An IfStmt is a conditional: If Cond: True; else: False. 'elseif' is desugared into a chain of IfStmts.

func (*IfStmt) AllocComments

func (cr *IfStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (IfStmt) Comments

func (cr IfStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*IfStmt) Span

func (x *IfStmt) Span() (start, end Position)

type IndexExpr

type IndexExpr struct {
	X      Expr
	Lbrack Position
	Y      Expr
	Rbrack Position
	// contains filtered or unexported fields
}

An IndexExpr represents an index expression: X[Y].

func (*IndexExpr) AllocComments

func (cr *IndexExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (IndexExpr) Comments

func (cr IndexExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*IndexExpr) Span

func (x *IndexExpr) Span() (start, end Position)

type LambdaExpr

type LambdaExpr struct {
	Lambda Position
	Params []Expr // param = ident | ident=expr | * | *ident | **ident
	Body   Expr

	Function interface{} // a *resolve.Function, set by resolver
	// contains filtered or unexported fields
}

A LambdaExpr represents an inline function abstraction.

Although they may be added in future, lambda expressions are not currently part of the Starlark spec, so their use is controlled by the resolver.AllowLambda flag.

func (*LambdaExpr) AllocComments

func (cr *LambdaExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (LambdaExpr) Comments

func (cr LambdaExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*LambdaExpr) Span

func (x *LambdaExpr) Span() (start, end Position)

type ListExpr

type ListExpr struct {
	Lbrack Position
	List   []Expr
	Rbrack Position
	// contains filtered or unexported fields
}

A ListExpr represents a list literal: [ List ].

func (*ListExpr) AllocComments

func (cr *ListExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (ListExpr) Comments

func (cr ListExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*ListExpr) Span

func (x *ListExpr) Span() (start, end Position)

type Literal

type Literal struct {
	Token    Token // = STRING | INT
	TokenPos Position
	Raw      string      // uninterpreted text
	Value    interface{} // = string | int64 | *big.Int
	// contains filtered or unexported fields
}

A Literal represents a literal string or number.

func (*Literal) AllocComments

func (cr *Literal) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (Literal) Comments

func (cr Literal) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*Literal) Span

func (x *Literal) Span() (start, end Position)

type LoadStmt

type LoadStmt struct {
	Load   Position
	Module *Literal // a string
	From   []*Ident // name defined in loading module
	To     []*Ident // name in loaded module
	Rparen Position
	// contains filtered or unexported fields
}

A LoadStmt loads another module and binds names from it: load(Module, "x", y="foo").

The AST is slightly unfaithful to the concrete syntax here because Starlark's load statement, so that it can be implemented in Python, binds some names (like y above) with an identifier and some (like x) without. For consistency we create fake identifiers for all the strings.

func (*LoadStmt) AllocComments

func (cr *LoadStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (LoadStmt) Comments

func (cr LoadStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*LoadStmt) ModuleName

func (x *LoadStmt) ModuleName() string

ModuleName returns the name of the module loaded by this statement.

func (*LoadStmt) Span

func (x *LoadStmt) Span() (start, end Position)

type Mode

type Mode uint

A Mode value is a set of flags (or 0) that controls optional parser functionality.

const (
	RetainComments Mode = 1 << iota // retain comments in AST; see Node.Comments
	BlockScanner   Mode = 1 << iota // use if/end syntax instead of indent
)

type Node

type Node interface {
	// Span returns the start and end position of the expression.
	Span() (start, end Position)

	// Comments returns the comments associated with this node.
	// It returns nil if RetainComments was not specified during parsing,
	// or if AllocComments was not called.
	Comments() *Comments

	// AllocComments allocates a new Comments node if there was none.
	// This makes possible to add new comments using Comments() method.
	AllocComments()
}

A Node is a node in a Starlark syntax tree.

type ParenExpr

type ParenExpr struct {
	Lparen Position
	X      Expr
	Rparen Position
	// contains filtered or unexported fields
}

A ParenExpr represents a parenthesized expression: (X).

func (*ParenExpr) AllocComments

func (cr *ParenExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (ParenExpr) Comments

func (cr ParenExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*ParenExpr) Span

func (x *ParenExpr) Span() (start, end Position)

type Position

type Position struct {
	Line int32 // 1-based line number; 0 if line unknown
	Col  int32 // 1-based column (rune) number; 0 if column unknown
	// contains filtered or unexported fields
}

A Position describes the location of a rune of input.

func End

func End(n Node) Position

End returns the end position of the expression.

func MakePosition

func MakePosition(file *string, line, col int32) Position

MakePosition returns position with the specified components.

func Start

func Start(n Node) Position

Start returns the start position of the expression.

func (Position) Filename

func (p Position) Filename() string

Filename returns the name of the file containing this position.

func (Position) IsValid

func (p Position) IsValid() bool

IsValid reports whether the position is valid.

func (Position) String

func (p Position) String() string

type ReturnStmt

type ReturnStmt struct {
	Return Position
	Result Expr // may be nil
	// contains filtered or unexported fields
}

A ReturnStmt returns from a function.

func (*ReturnStmt) AllocComments

func (cr *ReturnStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (ReturnStmt) Comments

func (cr ReturnStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*ReturnStmt) Span

func (x *ReturnStmt) Span() (start, end Position)

type SliceExpr

type SliceExpr struct {
	X            Expr
	Lbrack       Position
	Lo, Hi, Step Expr // all optional
	Rbrack       Position
	// contains filtered or unexported fields
}

A SliceExpr represents a slice or substring expression: X[Lo:Hi:Step].

func (*SliceExpr) AllocComments

func (cr *SliceExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (SliceExpr) Comments

func (cr SliceExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*SliceExpr) Span

func (x *SliceExpr) Span() (start, end Position)

type Stmt

type Stmt interface {
	Node
	// contains filtered or unexported methods
}

A Stmt is a Starlark statement.

type Token

type Token int8

A Token represents a Starlark lexical token.

const (
	ILLEGAL Token = iota
	EOF

	NEWLINE
	INDENT
	OUTDENT

	// Tokens with values
	IDENT  // x
	INT    // 123
	FLOAT  // 1.23e45
	STRING // "foo" or 'foo' or ”'foo”' or r'foo' or r"foo"

	// Punctuation
	PLUS          // +
	MINUS         // -
	STAR          // *
	SLASH         // /
	SLASHSLASH    // //
	PERCENT       // %
	AMP           // &
	PIPE          // |
	CIRCUMFLEX    // ^
	LTLT          // <<
	GTGT          // >>
	TILDE         // ~
	DOT           // .
	COMMA         // ,
	EQ            // =
	SEMI          // ;
	COLON         // :
	LPAREN        // (
	RPAREN        // )
	LBRACK        // [
	RBRACK        // ]
	LBRACE        // {
	RBRACE        // }
	LT            // <
	GT            // >
	GE            // >=
	LE            // <=
	EQL           // ==
	NEQ           // !=
	PLUS_EQ       // +=    (keep order consistent with PLUS..GTGT)
	MINUS_EQ      // -=
	STAR_EQ       // *=
	SLASH_EQ      // /=
	SLASHSLASH_EQ // //=
	PERCENT_EQ    // %=
	AMP_EQ        // &=
	PIPE_EQ       // |=
	CIRCUMFLEX_EQ // ^=
	LTLT_EQ       // <<=
	GTGT_EQ       // >>=
	STARSTAR      // **

	// Keywords
	AND
	BREAK
	CONTINUE
	DEF
	ELIF
	ELSE
	FOR
	IF
	IN
	LAMBDA
	LOAD
	NOT
	NOT_IN // synthesized by parser from NOT IN
	OR
	PASS
	RETURN
	WHILE
)

func (Token) GoString

func (tok Token) GoString() string

GoString is like String but quotes punctuation tokens. Use Sprintf("%#v", tok) when constructing error messages.

func (Token) String

func (tok Token) String() string

type TupleExpr

type TupleExpr struct {
	Lparen Position // optional (e.g. in x, y = 0, 1), but required if List is empty
	List   []Expr
	Rparen Position
	// contains filtered or unexported fields
}

A TupleExpr represents a tuple literal: (List).

func (*TupleExpr) AllocComments

func (cr *TupleExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (TupleExpr) Comments

func (cr TupleExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*TupleExpr) Span

func (x *TupleExpr) Span() (start, end Position)

type UnaryExpr

type UnaryExpr struct {
	OpPos Position
	Op    Token
	X     Expr // may be nil if Op==STAR
	// contains filtered or unexported fields
}

A UnaryExpr represents a unary expression: Op X.

As a special case, UnaryOp{Op:Star} may also represent the star parameter in def f(*args) or def f(*, x).

func (*UnaryExpr) AllocComments

func (cr *UnaryExpr) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (UnaryExpr) Comments

func (cr UnaryExpr) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*UnaryExpr) Span

func (x *UnaryExpr) Span() (start, end Position)

type WhileStmt

type WhileStmt struct {
	While Position
	Cond  Expr
	Body  []Stmt
	// contains filtered or unexported fields
}

A WhileStmt represents a while loop: while X: Body.

func (*WhileStmt) AllocComments

func (cr *WhileStmt) AllocComments()

AllocComments enables comments to be associated with a syntax node.

func (WhileStmt) Comments

func (cr WhileStmt) Comments() *Comments

Comments returns the comments associated with a syntax node, or nil if AllocComments has not yet been called.

func (*WhileStmt) Span

func (x *WhileStmt) Span() (start, end Position)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL