Source: https://seerepo.com/uses/tokenizers
Markdown source: https://seerepo.com/uses/tokenizers.md
Description: 10 public repos that SeeRepo read use tokenizers. It's used most with torch, transformers, and pillow.

# Open source repos that use tokenizers

10 public repos that SeeRepo read use tokenizers. It's used most with torch, transformers, and pillow.

## Used most with

- torch: 3 repos
- transformers: 3 repos
- pillow: 3 repos
- Pydantic: 3 repos
- FastAPI: 3 repos
- OpenAI: 3 repos
- sentencepiece: 2 repos
- SQLite: 2 repos

## The repos, by stars

1. [Comfy-Org/ComfyUI](https://seerepo.com/r/Comfy-Org/ComfyUI): Python · 144,948 stars · 0.3% signed by AI — The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.
2. [vllm-project/vllm](https://seerepo.com/r/vllm-project/vllm): Python · 96,302 stars · 12.8% signed by AI — A high-throughput and memory-efficient inference and serving engine for LLMs
3. [datawhalechina/hello-agents](https://seerepo.com/r/datawhalechina/hello-agents): Python · 86,039 stars · 2.4% signed by AI — 📚 《从零开始构建智能体》——从零开始的智能体原理与实践教程
4. [opendatalab/MinerU](https://seerepo.com/r/opendatalab/MinerU): Python · 83,126 stars · 12.6% signed by AI — Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
5. [unslothai/unsloth](https://seerepo.com/r/unslothai/unsloth): Python · 79,050 stars · 1% signed by AI — Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
6. [headroomlabs-ai/headroom](https://seerepo.com/r/headroomlabs-ai/headroom): Python · 78,851 stars · 28.1% signed by AI — Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
7. [rohitg00/ai-engineering-from-scratch](https://seerepo.com/r/rohitg00/ai-engineering-from-scratch): Python · 76,859 stars · 0.8% signed by AI — Learn AI Engineering! Learn it. Build it. Ship it for others.
8. [comet-ml/opik](https://seerepo.com/r/comet-ml/opik): Python · 23,166 stars · 19.7% signed by AI — Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
9. [microsoft/unilm](https://seerepo.com/r/microsoft/unilm): Python · 22,240 stars — Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
10. [screenpipe/screenpipe](https://seerepo.com/r/screenpipe/screenpipe): Rust · 22,239 stars · 36% signed by AI — YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

## Frequently asked questions

### What is the most popular open source repo that uses tokenizers?

Comfy-Org/ComfyUI, with 144,948 stars on GitHub.

The proof: [Comfy-Org/ComfyUI on GitHub](https://github.com/Comfy-Org/ComfyUI)

### What other technologies is tokenizers used with?

With torch (3), transformers (3), pillow (3), Pydantic (3), and FastAPI (3), out of the 10 repos that use it.

### What languages are the repos that use tokenizers written in?

Python (9) and Rust (1).

### How many repos that use tokenizers have commits signed by AI?

9 of the 10; the most, screenpipe/screenpipe, with 36% of its commits.

The proof: [the commits each tool signed in screenpipe/screenpipe](https://seerepo.com/r/screenpipe/screenpipe/ai)

## What is your repo built with?

Any public repo, with no account: https://seerepo.com/r/owner/repo/built, with the owner and name it has on GitHub.