scanner

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 31, 2020 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package scanner is a subpackage of ptk that contains various implementations of the Scanner, along with related support types and code such as Char and Location. A scanner is an object that can be queried for the next character in a character stream, such as a file or a list of arguments; its return value is a Char, which packages together a Unicode rune and a location that may be used for error reporting.

Index

Constants

View Source
const DefaultTabStop = 8

DefaultTabStop is the default tab stop for the scanner.

View Source
const (
	EOF rune = unicode.MaxRune + iota + 1 // Signals end-of-file

)

Special runes used to signal end-of-file or other conditions in the scanners.

Variables

View Source
var (
	ErrSplitLocation = errors.New("Attempt to range file location through an incompatible location")
	ErrBadEncoding   = errors.New("Invalid UTF-8 encoding")
)

Simple errors that may be generated within the package.

View Source
var DOSLineStyle = &dosLineStyle{}

DOSLineStyle is a style for handling the use of carriage return and newline line endings, also known as DOS line endings.

View Source
var MacLineStyle = &macLineStyle{}

MacLineStyle is a style for handling the use of bare carriage return line endings, also known as Mac classic line endings.

View Source
var NoLineStyle = &noLineStyle{}

NoLineStyle is a style for handling the case when no newlines in the source should be recognized. All carriage returns and newlines will be substituted with spaces.

View Source
var UNIXLineStyle = &unixLineStyle{}

UNIXLineStyle is a style for handling the use of bare newline line endings, also known as UNIX line endings.

View Source
var UnknownLineStyle = &unknownLineStyle{}

UnknownLineStyle is a style for handling files where the line ending style is not yet known. It guesses the line ending style from the first line ending encountered, then switches to the appropriate line ending style.

Functions

func LocationError

func LocationError(loc Location, err error) error

LocationError wraps an error and includes a location.

Types

type ArgJoiner

type ArgJoiner string

ArgJoiner is an argument option that specifies the string that should logically be expected between each argument. By default, this is a single space (" ").

type ArgLocation

type ArgLocation struct {
	B ArgPos // Beginning of the range
	E ArgPos // End of the range
}

ArgLocation is an implementation of Location that identifies the location of an element within a list of strings, typically command line arguments. It represents a full range, but tab stops and newlines are not treated specially.

func (ArgLocation) Incr

func (l ArgLocation) Incr(c rune, tabstop int) Location

Incr increments the location by one character. It is passed the character (a rune) and the tabstop size (for handling tabs). It should return a new Location.

func (ArgLocation) String

func (l ArgLocation) String() string

String constructs a string representation of the location.

func (ArgLocation) Thru

func (l ArgLocation) Thru(other Location) (Location, error)

Thru creates a new Location that ranges from the beginning of this location to the beginning of another Location.

func (ArgLocation) ThruEnd

func (l ArgLocation) ThruEnd(other Location) (Location, error)

ThruEnd is similar to Thru, except that it creates a new Location that ranges from the beginning of this location to the ending of another location.

type ArgOption

type ArgOption interface {
	// contains filtered or unexported methods
}

ArgOption is an option that may be passed to the NewArgumentScanner function.

type ArgPos

type ArgPos struct {
	I int // The index of the argument in its list
	C int // The index of the character within the argument
}

ArgPos specifies a particular character location within an argument list. It is a component of the ArgLocation type.

type ChainingScanner

type ChainingScanner struct {
	// contains filtered or unexported fields
}

ChainingScanner is a scanner that chains together several scanners. When one scanner returns EOF, the ChainingScanner skips it and proceeds to the next one. Note that if any call to Next returns an error, that error is returned immediately.

func NewChainingScanner

func NewChainingScanner(streams []Scanner) *ChainingScanner

NewChainingScanner constructs and returns a Scanner implementation that returns characters from each of the provided scanners in turn. When one scanner returns EOF, that EOF is skipped and the chaining scanner begins drawing characters from the next one. If any scanner returns an error, that error is returned immediately.

func (*ChainingScanner) Next

func (ccs *ChainingScanner) Next() (Char, error)

Next returns the next character from the stream as a Char, which will include the character's location. If an error was encountered, that will also be returned.

type Char

type Char struct {
	Rune rune     // The rune read from the source
	Loc  Location // The location of the rune within the stream
}

Char represents a character retrieved from the source input stream. It bundles together a rune and a location.

type EncodingErrorHandler

type EncodingErrorHandler interface {
	// Handle handles the reported encoding error.  If it returns
	// non-nil, the scanner will report an error.
	Handle(e error) error
}

EncodingErrorHandler is an interface for an encoding error handler. A scanner will call the Handle method of the error handler, which must return either nil or an error (which may be the same error).

type EncodingErrorOption

type EncodingErrorOption struct {
	// contains filtered or unexported fields
}

EncodingErrorOption is the type that stores the encoding error handler that the file scanner should use.

func EncodingError

func EncodingError(enc EncodingErrorHandler) EncodingErrorOption

EncodingError is an option that may be passed to either NewFileScanner or NewArgumentScanner. It is used to specify an EncodingErrorHandler to use to handle encoding errors.

type FileLocation

type FileLocation struct {
	File string  // Name of the file
	B    FilePos // The beginning of the range
	E    FilePos // The end of the range
}

FileLocation is an implementation of Location that identifies the location of an element within a file. It represents a full range, and has some additional utilities to simplify handling advancement within the file, including of tab stops.

func (FileLocation) Incr

func (l FileLocation) Incr(c rune, tabstop int) Location

Incr increments the location by one character. It is passed the character (a rune) and the tabstop size (for handling tabs). It should return a new Location.

func (FileLocation) String

func (l FileLocation) String() string

String constructs a string representation of the location.

func (FileLocation) Thru

func (l FileLocation) Thru(other Location) (Location, error)

Thru creates a new Location that ranges from the beginning of this location to the beginning of another Location.

func (FileLocation) ThruEnd

func (l FileLocation) ThruEnd(other Location) (Location, error)

ThruEnd is similar to Thru, except that it creates a new Location that ranges from the beginning of this location to the ending of another location.

type FileOption

type FileOption interface {
	// contains filtered or unexported methods
}

FileOption is an option that may be passed to the NewFileScanner function.

func LineEndings

func LineEndings(ls LineStyle) FileOption

LineEndings is a file scanner option that may be used to set the preferred line ending style. A line ending style is an instance of LineStyle that controls how the scanner recognizes newlines. The scanner always converts line endings into single newlines.

type FilePos

type FilePos struct {
	L int // The line number of the position (1-indexed)
	C int // The column number of the position (1-indexed)
}

FilePos specifies a particular character location within a file. It is a component of the FileLocation type.

type FileScanner

type FileScanner struct {
	// contains filtered or unexported fields
}

FileScanner is an implementation of Scanner that wraps io.Reader objects. FileScanner also contains logic to handle diverse line ending styles through the use of LineStyle objects; line endings will be converted into single newlines if this support is enabled.

func NewFileScanner

func NewFileScanner(r io.Reader, loc Location, options ...FileOption) *FileScanner

NewFileScanner constructs a new instance of the FileScanner.

func (*FileScanner) Next

func (s *FileScanner) Next() (Char, error)

Next returns the next character from the stream as a Char, which will include the character's location. If an error was encountered, that will also be returned.

type LineDis

type LineDis int

LineDis is a type for the return value of LineStyle.Handle.

const (
	LineDisNewline     LineDis = iota // Newline sequence recognized
	LineDisNewlineSave                // Newline followed by newline
	LineDisSpace                      // Substitute a space
	LineDisMore                       // Need another character
)

Possible return value codes for LineStyle.Handle.

type LineStyle

type LineStyle interface {
	// Handle checks to see if a line ending sequence has been
	// encountered.  It returns a LineDis value, which indicates
	// the disposition of the character; and a LineStyle object to
	// use next time around.
	Handle(chs []rune) (LineDis, LineStyle)
}

LineStyle represents a line ending style. This controls how lines are handled when read from the source input stream, and is used for maintaining the location that is then attached to runes.

type ListScanner

type ListScanner struct {
	// contains filtered or unexported fields
}

ListScanner is a scanner that returns characters from a simple list. It is intended for testing in cases where MockScanner is not a good fit.

func NewListScanner

func NewListScanner(chars []Char, err error) *ListScanner

NewListScanner constructs and returns a Scanner implementation that returns characters from a list of characters. The last character should be a EOF; this character will be returned with the error passed in. This scanner is intended for testing.

func (*ListScanner) Next

func (lcs *ListScanner) Next() (Char, error)

Next returns the next character from the stream as a Char, which will include the character's location. If an error was encountered, that will also be returned.

type Location

type Location interface {
	// String constructs a string representation of the location.
	String() string

	// Thru creates a new Location that ranges from the beginning
	// of this location to the beginning of another Location.
	Thru(other Location) (Location, error)

	// ThruEnd is similar to Thru, except that it creates a new
	// Location that ranges from the beginning of this location to
	// the ending of another location.
	ThruEnd(other Location) (Location, error)

	// Incr increments the location by one character.  It is
	// passed the character (a rune) and the tabstop size (for
	// handling tabs).  It should return a new Location.
	Incr(c rune, tabstop int) Location
}

Location is an interface for location data. Each token and node should have attached location data that reports its location. This aids in finding the location of errors.

func LocationOf

func LocationOf(err error) Location

LocationOf attempts to retrieve the location of an error. If the location is not available, it returns nil.

type MemoizingScanner

type MemoizingScanner struct {
	// contains filtered or unexported fields
}

MemoizingScanner is a scanner that wraps another scanner and records the characters that it returns. Once the wrapped scanner returns an EOF, the MemoizingScanner will loop back around and simply return the same characters over and over.

func NewMemoizingScanner

func NewMemoizingScanner(src Scanner) *MemoizingScanner

NewMemoizingScanner constructs and returns a Scanner implementation that wraps another and simply returns all the characters it returns. After the EOF character is returned--which is passed through unchanged--the scanner is replayed over and over. If the source scanner returns an error, that error is reported immediately.

func (*MemoizingScanner) Next

func (mcs *MemoizingScanner) Next() (Char, error)

Next returns the next character from the stream as a Char, which will include the character's location. If an error was encountered, that will also be returned.

type Scanner

type Scanner interface {
	// Next returns the next character from the stream as a Char,
	// which will include the character's location.  If an error
	// was encountered, that will also be returned.
	Next() (Char, error)
}

Scanner presents a stream of characters. The basic scanner does not provide backtracking or character push-back. The FileScanner is one implementation of Scanner.

func NewArgumentScanner

func NewArgumentScanner(args []string, options ...ArgOption) Scanner

NewArgumentScanner constructs and returns a Scanner implementation that returns characters drawn from a provided list of argument strings. This is intended for use with arguments taken from the command line, but could be useful in other contexts as well. The strings are logically joined by spaces; to use a different joiner, pass that as an option.

type TabStop

type TabStop int

TabStop is a file scanner option that specifies the tab stop to apply. The default tab stop is 8.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL