parser

Overview ¶

Package parser implements the parser for Hydra. The parser takes a given source file and generates from it an abstract syntax tree.

The parser is split into several parts. The work begins with the scanner, which reads in a file of a given encoding and converts it to UTF-8. The scanner performs its scan as efficiently as possible, emitting a sequence of augmented characters. Augmented characters, defined in /parser/common/classes.go, package together the character, a character class, the location of the character in the file, and any semantic value. The scanner also has unlimited push-back, enabling the lexer to reprocess characters as needed.

The lexer is responsible for consuming the continuous stream of characters produced by the scanner and breaking it up into tokens. A token represents a "word" in a programming language, e.g., something like a keyword like "if", or punctuation with meaning, such as "{". The Hydra lexer uses a concept of a "recognizer" to recognize particular classes of tokens.

Finally, the parser itself takes the tokens generated by the lexer and assembles them into an abstract syntax tree. The AST contains a full description of an entire Hydra source file, and can be used to construct the underlying machine code of a compiled Hydra module.

Besides the three main pieces described above, the parser also contains a "common" package, containing common bits that are shared by the various components of the parser. There is also an "ast" package, which contains the definitions of the abstract syntax tree structures used for representing the AST generated by the parser.