tokenization
Tracked open-source repos tagged tokenization, sorted by stars.
Related topics
Topics that frequently appear alongside tokenization on the same repo.
Recent risers
Repos created in the last 90 days, tagged tokenization.
No new repos tagged with this topic in the last 90 days.
- #1★ 33,861+23Star change over the last 7 days
- #2
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
★ 25,275+55Star change over the last 7 days - #3★ 4,067+18Star change over the last 7 days
- #4★ 2,005+0Star change over the last 7 days
- #5
Secure Vault for Customer PII/PHI/PCI/KYC Records
★ 1,483+1Star change over the last 7 days - #6
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
★ 1,469+0Star change over the last 7 days - #7★ 1,119+2Star change over the last 7 days
- #8
👑 spaCy building blocks and visualizers for Streamlit apps
★ 860+0Star change over the last 7 days - #9
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
★ 798+2Star change over the last 7 days - #10
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
★ 673-1Star change over the last 7 days - #11
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
★ 627+0Star change over the last 7 days - #12
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
★ 562+0Star change over the last 7 days - #13
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
★ 534+0Star change over the last 7 days
