Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Tokenizer ¶
type Tokenizer interface { // Next processes the next token and returns true // if there are more tokens to process. Next() bool // Token returns the current token. // It must be called after Next. Token() string }
Tokenizer is an interface for tokenizers.
func Chain ¶
Chain returns a Tokenizer that combines the given tokenizers into a chain.
The Next method of chain gets tokens serially from the chained tokenizers until the last one returns false. For example, the following code:
ch := tokenizer.Chain(tokenizer.Words("hello world"), tokenizer.Words("this is me")) for ch.Next() { fmt.Printf("%s, ", ch.Token()) }
will print:
hello, world, this, is, me,
Click to show internal directories.
Click to hide internal directories.