Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security Policy
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

datok

package module

Go to main page

Versions in this module

v0

v0.2.2

Sep 6, 2023

v0.2.1

Sep 5, 2023

v0.2.0

Sep 5, 2023

v0.1.5

Mar 27, 2022

v0.1.4

Mar 27, 2022

v0.1.3

Mar 9, 2022

v0.1.1

Nov 28, 2021

v0.1.0

Nov 28, 2021

Changes in this version

+ const DAMAGIC

+ const DEBUG

+ const EOT

+ const FIRSTBIT

+ const MAMAGIC

+ const NONE

+ const PROPS

+ const RESTBIT

+ const SECONDBIT

+ const SIGMA

+ const STATES

+ const VERSION

+ type Automaton struct

+ func LoadFomaFile(file string) *Automaton

+ func ParseFoma(ior io.Reader) *Automaton

+ func (auto *Automaton) ToDoubleArray() *DaTokenizer

+ func (auto *Automaton) ToMatrix() *MatrixTokenizer

+ type Bits uint8

+ const NEWLINE_AFTER_EOT

+ const SENTENCES

+ const SENTENCE_POS

+ const SIMPLE

+ const TOKENS

+ const TOKEN_POS

+ type DaTokenizer struct

+ func LoadDatokFile(file string) *DaTokenizer

+ func ParseDatok(ior io.Reader) *DaTokenizer

+ func (DaTokenizer) Type() string

+ func (dat *DaTokenizer) GetSize() int

+ func (dat *DaTokenizer) LoadFactor() float64

+ func (dat *DaTokenizer) Save(file string) (n int64, err error)

+ func (dat *DaTokenizer) TransCount() int

+ func (dat *DaTokenizer) Transduce(r io.Reader, w io.Writer) bool

+ func (dat *DaTokenizer) TransduceTokenWriter(r io.Reader, w *TokenWriter) bool

+ func (dat *DaTokenizer) WriteTo(w io.Writer) (n int64, err error)

+ type MatrixTokenizer struct

+ func LoadMatrixFile(file string) *MatrixTokenizer

+ func ParseMatrix(ior io.Reader) *MatrixTokenizer

+ func (MatrixTokenizer) Type() string

+ func (mat *MatrixTokenizer) Save(file string) (n int64, err error)

+ func (mat *MatrixTokenizer) Transduce(r io.Reader, w io.Writer) bool

+ func (mat *MatrixTokenizer) TransduceTokenWriter(r io.Reader, w *TokenWriter) bool

+ func (mat *MatrixTokenizer) WriteTo(w io.Writer) (n int64, err error)

+ type TokenWriter struct

+ Flush func() error

+ SentenceEnd func(int)

+ TextEnd func(int)

+ Token func(int, []rune)

+ func NewTokenWriter(w io.Writer, flags Bits) *TokenWriter

+ type Tokenizer interface

+ Transduce func(r io.Reader, w io.Writer) bool

+ TransduceTokenWriter func(r io.Reader, w *TokenWriter) bool

+ Type func() string

+ func LoadTokenizerFile(file string) Tokenizer