alasdairforsythe/tokenmonster

@alasdairforsythe

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

↗ Open on GitHub

Stars

627

Forks

23

Language

Go

License

MIT

Last push

2 months ago

Go tokenizer tokenization vocabulary text-tokenization tokenisation tokenize tokenizing vocabulary-builder vocabulary-generator

No related intel yet

This repo has not appeared in any of the sources the radar tracks. The collector runs on a schedule — check back once it covers this repo.