marcelroed/gigatoken

Language model tokenization at GB/s

↗ Open on GitHub

Stars

4,075

Forks

219

Language

Rust

License

MIT

Last push

5 days ago

Rust llm nlp tokenizer tokenization

No related intel yet

This repo has not appeared in any of the sources the radar tracks. The collector runs on a schedule — check back once it covers this repo.