#Tokenizer
Showing 49 of 49 repositories tagged #tokenizer, ranked by stars
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
Parser Building Toolkit for JavaScript
Online playground for OpenAPI tokenizers
Solves basic Russian NLP tasks, API for lower level Natasha projects
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
Self-contained Japanese Morphological Analyzer written in pure Go
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构建基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注需要依赖开源的数字底座进行人员岗位管控。
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
A multilingual morphological analysis library.
[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding
A multilingual command line sentence tokenizer in Golang
Lex machinary for go.
JS tokenizer for LLaMA 1 and 2
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
Text2Text Language Modeling Toolkit
Full-stack open-source AI engine for building language models — tokenizer training, transformer architecture, cognitive reasoning and chat pipeline.
Detailed python notes & code for lectures and exercises of Andrej Karpathy's course "Neural Networks: Zero to Hero." The course is focused on building neural networks from scratch.
Rust re-implementation of OpenFST - library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs). A Python binding is also available.
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
CLI tool – estimates LLM tokens/costs and runs provider-aware load tests for OpenAI, Anthropic, OpenRouter, or custom endpoints.
This repository contains the collection of explorative notebooks pure in python and in the language that we, humans can read. Have tried to compile all lectures from the Andrej Karpathy's 💎 playlist on Neural Networks - which we will end up with building GPT.
Collection of developer toolkits
Kanade is a single-layer disentangled speech tokenizer that extracts compact tokens suitable for both generative and discriminative modeling.
Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.
OpenShield is a new generation security layer for AI models
Swift BPE Encoder/Decoder for OpenAI GPT Models. A programmatic interface for tokenizing text for OpenAI ChatGPT API.
Chinese tokenizer for tantivy, based on jieba-rs
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
[ICML 2025 Tokshop] One-D-Piece: Image Tokenizer Meets Quality-Controllable Compression
Sylber: Syllabic Embedding Representation of Speech from Raw Audio
Essential NLP & ML, short & fast pure Python code
Ruby Object Inspection for IRB
LangChain-lite for ABAP: Enterprise LLM orchestration framework with lazy execution, powerful templating, caching, and token prediction. Build complex AI workflows in SAP ERP, SAP S/4.
On Device Stable Diffusion In Mobile Devices
a unix-like du command line tool to count token usage per files and directories
Open Source Generative AI Models for Automatic Rhythm Game Beatmap Generation (for acoustic people)
Tokenize English sentences using neural networks.
Thot toolkit for statistical machine translation
Get identifiers, paths, URLs and words from the previous command output and use them for the next command in @xonsh.
ZON → 35-70% cheaper LLM prompts than JSON/TOON. Zero overhead.
NeuroRVQ: Multi-Scale Biosignal Tokenization for Generative Foundation Models
A small godot project with a tokenizer written in GDScript.
AI21's Jamba models tokenizers
C language lexer & parser & virtual interpreter from scratch
Smart code context extractor for AI assistants
SageMaker Ployglot based RAG opensearch
3DPass mobile light wallet for Android and iOS. 3D object recognition and wallet functions.
The ultimate tool for analyzing, visualizing, and optimizing your LLM prompts