Versions in this module Expand all Collapse all v0 v0.7.0 Dec 8, 2016 Changes in this version + type Tokenizer struct + func NewTokenizer(s []byte) (*Tokenizer, error) + func (t *Tokenizer) Destroy() + func (t *Tokenizer) Next() []byte + func (t *Tokenizer) Tokens() []string