lexer

type Cursor ¶

type Cursor interface {
	// NextRune is called to extend the cursor by reading the next rune from
	// input.  Also updates the pending string except when skipping spaces, and
	// returns the updated cursor and the rune that was read.
	NextRune(input io.RuneReader) (Cursor, rune)
	// Similar to NextRune() but will read from the pending buffer if nonempty,
	// and read from input, populating pending, if pending was empty.  Implicitly
	// ignores leading spaces if needing to read from input.
	FirstRune(input io.RuneReader) (Cursor, rune)

	// Consumes all characters in the pending rune list, updating pos to match.
	ConsumeAll() (Cursor, string)
	// Same as ConsumeAll() except the last rune is left in the pending buffer.
	ConsumeExceptFinal() (Cursor, string)

	// Resets the TokenPos for this cursor to (0, 0, UNKNOWN).
	ResetPos() Cursor

	// The current position of the next Token that would be produced by consuming
	// the contents of this Cursor, whether or not anything is in pending buffer.
	Pos() TokenPos

	// Returns true if there is nothing pending in the cursor.
	IsEmpty() bool

	// Returns true if the last ReadRune call returned an error.
	HasError() bool
	// Returns `true` if the embedded error is io.EOF.
	IsEOF() bool
	// Returns the error (or nil) from the most recent read of input.  If an error
	// is encountered, it will persist through update methods and prohibit reads.
	//
	// Intentionally not extending `error` interface by naming this ErrorValue.
	ErrorValue() error
}

The Cursor represents a few properties of the lexer's state that are invariably coupled to each other -- token position, the runes ready to be integrated into the next token, and whether there is a pending rune waiting to be processed. The token's next position should always be the current position plus the size of the pending rune, if there is a pending rune, but that relies on whether scanning can be done in LL(1) or (in some cases) LL(0) as with `(` and `)`. It also smelled bad to be updating only part of the lexer state and another part that depended on it, while not doing so atomically.

This, and its backing struct, are a solution to the above problems while also aiding the readability of the token-specific lexer code. The coupled updates are done within the Advance and Consume methods, there is no redundant next pos or ambiguity about the contents of the pending image. In addition to that, the cursor is copy-on-write, all updates are conveyed by the return value of the updating method, and the implementing methods use by-value receivers so downcast-and-update has limited adverse effect.

However, it assumes that it is the only reader on the provided input, and that its scan position is consistent between calls to Advance. If there is need of multiple concurrent cursors on the same reader source, use new readers for each cursor or tee the source RunReader. Rather than further complicate this code with management of byte offsets and seeks at each read, especially while this task of tokenizing byte streams is inherently single- threaded. Calling code is expected to manage it, typically via lexerState.

func NewCursor() Cursor

type ExprEndToken ¶

type ExprEndToken struct{ SymbolToken }

Token EXPR_END = ")"

var EXPR_END ExprEndToken

func (tok ExprEndToken) Image() string

func (tok ExprEndToken) TypeString() string

type ExprStartToken ¶

type ExprStartToken struct{ SymbolToken }

Token EXPR_START = "("

var EXPR_START ExprStartToken

func (tok ExprStartToken) Image() string

func (tok ExprStartToken) TypeString() string

type KeywordToken ¶

type KeywordToken struct {
	// contains filtered or unexported fields
}

All keywords are given the KEYWORD token type.

func (tok KeywordToken) At(pos TokenPos) Token

func (tok KeywordToken) Image() string

func (tok KeywordToken) TypeString() string

type LDArrowToken ¶

type LDArrowToken struct{ SymbolToken }

Token ARROW_LD = "<="

var ARROW_LD LDArrowToken

func (tok LDArrowToken) Image() string

func (tok LDArrowToken) TypeString() string

type QMarkToken ¶

type QMarkToken struct{ SymbolToken }

Token QUE_MARK = "?"

var QUE_MARK QMarkToken

func (tok QMarkToken) Image() string

func (tok QMarkToken) TypeString() string

type SymbolToken ¶

type SymbolToken struct{}

Symbol tokens always have the same image, they can share a common instance.

type Token ¶

type Token struct {
	TokenPos
	TokenType
}

Represents a Token instance by its position in the source and its type. The TokenType is an embedded interface (see above) and may be initialized with state/context or reuse a shared instance for the many tokens that are universally identical within their type (e.g. keywords, operator symbols). TokenPos is a 32-bit uint composite value defined in [token_pos.go].

func ExpressionEnd(pos TokenPos) Token

func ExpressionStart(pos TokenPos) Token

func Identifier(name string, pos TokenPos) Token

func Integer(image string, pos TokenPos) Token

func KeywordAt(image string, pos TokenPos) Token

func LeftDoubleArrow(pos TokenPos) Token

func LineComment(image string, pos TokenPos) Token

func QuestionMark(pos TokenPos) Token

func UnexpectedToken(image string, pos TokenPos) Token

func (data Token) String() string

type TokenPos ¶

type TokenPos uint32

TokenPos encoded as 32-bit uint:

.LLLLLLLLLLLLLLLLLLLLCCCCCCCCCCFF. :[++++++++++++++++++] : : 20 bits LINE : : [++++++++] : : 10 bits COLUMN : : []: : 2 bits FLAGS: : : `10987654321098765432109876543210'

Use Line(), Column() and Next*() methods to read and update values.

func NewTokenPos(line, col uint) TokenPos

func (pos TokenPos) Column() uint

func (pos TokenPos) InComment() TokenPos

func (pos TokenPos) InMetaBlock() TokenPos

func (pos TokenPos) InSentence() TokenPos

func (pos TokenPos) Line() uint

func (pos TokenPos) NextAt(lines, cols uint) TokenPos

func (pos TokenPos) NextCol() TokenPos

func (pos TokenPos) NextLine() TokenPos

func (pos TokenPos) ResetFlag() TokenPos

func (data TokenPos) String() string

type TokenReader ¶

type TokenReader interface {
	// Reads the next token, sending it to output, returning error or nil.  If an
	// io.EOF error was encountered it is returned here as well.
	NextToken() error

	// Read/Receive-only channel for Token values sent as being read from the input.
	// Calling NextToken() or ReadAll() will produce tokens on this channel and one
	// of those methods will close the channel when it encounters EOF. An EOF token
	// is also produced as the last token on the channel, so consumers can listen
	// for it specifically or listen until channel close using `for ... := range`.
	TokenReceiver() <-chan Token
}

Public interface for reading a stream of tokens, sending them to a channel. See also ReadAll(reader) which provides a simpler interface for full reads.

func NewTokenReader(input io.RuneReader, output chan Token) TokenReader

type TokenType ¶

type TokenType interface {
	// Returns a string representation of the type of this token.
	TypeString() string
	// Returns a string representation of this token, its syntactic image.
	Image() string
}

TokenType intrinsically defines the subtype of a Token and provides identifying methods.