Open source repos that use tokenizers
10 public repos that SeeRepo read use tokenizers. It's used most with torch, transformers, and pillow.
LinkedInUsed most with
- torch3 repos
pillow3 repos
- FastAPI3 repos
- sentencepiece2 repos
- transformers3 repos
- Pydantic3 repos
OpenAI3 repos- SQLite2 repos
The repos, by stars
- 1Comfy-Org/
ComfyUI The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.
Python+8144.9kAI 0.3% - 2vllm-project/
vllm A high-throughput and memory-efficient inference and serving engine for LLMs
Python+4396.3kAI 12.8% - 4opendatalab/
MinerU Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
Python+1483.1kAI 12.6% - 5unslothai/
unsloth Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
Python+9579.1kAI 1% - 6headroomlabs-ai/
headroom Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python+2178.9kAI 28.1%
- 7rohitg00/
ai-engineering-from-scratch Learn AI Engineering! Learn it. Build it. Ship it for others.
Python+1876.9kAI 0.8%
- 8comet-ml/
opik Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Python+8923.2kAI 19.7%
- 9microsoft/
unilm Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
Python+2822.2k
- 10screenpipe/
screenpipe YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust
Frequently asked questions
What is the most popular open source repo that uses tokenizers?
The proof: Comfy-Org/ComfyUI on GitHub
What other technologies is tokenizers used with?
What languages are the repos that use tokenizers written in?
How many repos that use tokenizers have commits signed by AI?
The proof: the commits each tool signed in screenpipe/screenpipe
See inside your code, today.
Paste your repo and see it drawn, part by part: a medium one in under a minute.
Free, no account.
