Repos open source que usan tokenizers
10 repos públicos que SeeRepo leyó usan tokenizers. Se usa más con torch, transformers y pillow.
Se usa más con
- torch3 repos
pillow3 repos
- FastAPI3 repos
- sentencepiece2 repos
- transformers3 repos
- Pydantic3 repos
OpenAI3 repos- SQLite2 repos
Los repos, por estrellas
- 1Comfy-Org/
ComfyUI The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.
Python+8144.9 kIA 0.3% - 2vllm-project/
vllm A high-throughput and memory-efficient inference and serving engine for LLMs
Python+4396.3 kIA 12.8% - 4opendatalab/
MinerU Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
Python+1483.1 kIA 12.6% - 5unslothai/
unsloth Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.
Python
+9579.1 kIA 1% - 6headroomlabs-ai/
headroom Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python+2178.9 kIA 28.1%
- 7rohitg00/
ai-engineering-from-scratch Learn AI Engineering! Learn it. Build it. Ship it for others.
Python
+1876.9 kIA 0.8%
- 8comet-ml/
opik Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Python+8923.2 kIA 19.7%
- 9microsoft/
unilm Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
Python+2822.2 k
- 10screenpipe/
screenpipe YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust+9422.2 kIA 36%
Preguntas frecuentes
¿Cuál es el repo open source más popular que usa tokenizers?
La prueba: Comfy-Org/ComfyUI en GitHub
¿Con qué otras tecnologías se usa tokenizers?
¿En qué lenguajes están escritos los repos que usan tokenizers?
¿Cuántos repos que usan tokenizers tienen commits firmados por IA?
La prueba: los commits que firmó cada herramienta en screenpipe/screenpipe
Mira tu código por dentro, hoy.
Pega tu repo y lo ves dibujado, parte por parte: uno mediano, en menos de un minuto.
Gratis y sin cuenta.

