What is TensorRT-LLM built with?

Q: Which repos have a stack like TensorRT-LLM's?

The ones that share most of its stack: ray-project/ray, sgl-project/sglang, and vllm-project/vllm.

TensorRT-LLM is a Python repo built with FastAPI, anyio, uvicorn, and starlette. It stores its data in pyzmq, uses AI through accelerate, apache-tvm-ffi, cuda-python, flash-linear-attention, flashinfer-python, and 32 more, makes its content with jsonlines, pillow, jinja2, and mako, validates its data with Pydantic, msgspec, and jsonschema, is tested with pytest-cov, pytest-env, pytest-forked, pytest-rerunfailures, pytest-timeout, and 6 more, deploys with opentelemetry-api, opentelemetry-semantic-conventions-ai, psutil, prometheus-client, boto3, and 3 more, and uses blobfile and h5py. It has 2,884,778 written lines, with 2,703 test files.

Read on November 8, 2026, at commit b433473. Read it as markdown

Frontend

Interface 3

Rich12 files
matplotlib1 file
Plotly1 file

Backend

The base 1

FastAPI15 files

Data 3

h5py
pyzmq13 files
blobfile

AI 36

torch1,611 files
diffusers35 files
safetensors17 files
torchvision10 files
xgrammar4 files
librosa2 files
mcp2 files
partial-json-parser2 files
evaluate1 file
lm-eval1 file
peft1 file
accelerate

24 more

apache-tvm-ffi
flash-linear-attention
flashinfer-python
NVIDIA
nvidia-cudnn-frontend
openai-codex
optimum
rouge-score
torchao
triton218 files
transformers168 files
einops20 files
datasets7 files
flash-attn7 files
mistral-common2 files
tritonclient2 files
llguidance1 file
cuda-python
openai-harmony
quack-kernels
rouge
sentencepiece
tiktoken
torch-c-dlpack-ext

Services 1

OpenAI3 files · tensorrt_llm/scaffolding/task.py, line 22

Support

The base 3

uvicorn6 files
starlette2 files
anyio7 files

Content 4

pillow34 files
jinja24 files
mako2 files
jsonlines

Validation 3

Pydantic34 files
msgspec4 files
jsonschema2 files

Testing 11

Pytest2,232 files
pytest-asyncio4 files
pytest-mock1 file
pytest-cov
pytest-env
pytest-forked
pytest-rerunfailures
pytest-timeout
pytest-xdist
parameterized30 files
pytest-split1 file

Deploy 8

psutil7 files
prometheus-client6 files
OpenTelemetry1 file
opentelemetry-sdk1 file
opentelemetry-api
boto31 file
prometheus-fastapi-instrumentator1 file
opentelemetry-semantic-conventions-ai

Tools 27

click21 files
Setuptools1 file
bandit
build
meson
ninja
pre-commit
pybind11
Ruff
Sphinx
typing-extensions20 files
packaging15 files

15 more

cuda-tile4 files
docstring-parser1 file
autodoc-pydantic
click-option-group
fire
graphviz
grpcio-tools
mypy
patchelf
sphinx-argparse
sphinx-click
sphinx-copybutton
sphinx-togglebutton
sphinxcontrib-mermaid
typeguard

Other 35

NumPy108 files
pyyaml56 files
msgpack5 files
pandas4 files
filelock3 files
pynvvideocodec1 file
rapidfuzz1 file
colored
lazy-loader
nvidia-ml-py
omegaconf
pydantic-settings

23 more

strenum12 files
nvtx4 files
blake33 files
soundfile3 files
uvloop3 files
regex2 files
requests2 files
cloudpickle1 file
distro1 file
numexpr1 file
ordered-set1 file
tabulate1 file
aenum
backoff
ftfy
grpcio
grpcio-health-checking
hf-transfer
jieba
lark
mpmath
protobuf
python-multipart

Other repos with a similar stack

Ordered by how much of its stack they share with TensorRT-LLM, then by stars.

ray-projectray
44.2k
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
35 of 135 in commonCompare →
sgl-projectsglang
38.5k
SGLang is a high-performance serving framework for large language models and multimodal models.
29 of 135 in commonCompare →
vllm-projectvllm
96.3k
A high-throughput and memory-efficient inference and serving engine for LLMs
28 of 135 in commonCompare →
onyx-dot-apponyx
33.3k
Open Source AI Platform - AI Chat with advanced features that works with every LLM
27 of 135 in commonCompare →
run-llamallama_index
52.8k
LlamaIndex is the document processing platform for AI
25 of 135 in commonCompare →
mudlerLocalAI
50.2k
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.
24 of 135 in common · GoCompare →

Frequently asked questions

What language is TensorRT-LLM written in?

TensorRT-LLM is written in Python, with 2,884,778 written lines.

What framework does TensorRT-LLM use?

TensorRT-LLM is built with FastAPI, anyio, uvicorn, and starlette.

What database does TensorRT-LLM use?

TensorRT-LLM stores its data in pyzmq.

Does TensorRT-LLM use AI?

Yes. TensorRT-LLM uses AI through accelerate, apache-tvm-ffi, cuda-python, flash-linear-attention, flashinfer-python, NVIDIA, nvidia-cudnn-frontend, openai-codex, openai-harmony, optimum, quack-kernels, rouge, rouge-score, sentencepiece, tiktoken, torch-c-dlpack-ext, torchao, torch, triton, transformers, diffusers, einops, safetensors, torchvision, datasets, flash-attn, xgrammar, librosa, mcp, mistral-common, partial-json-parser, tritonclient, evaluate, llguidance, lm-eval, peft, and OpenAI.

What services does TensorRT-LLM use?

TensorRT-LLM uses blobfile and h5py.

Does TensorRT-LLM have tests?

Yes. TensorRT-LLM has 2,703 test files and is tested with pytest-cov, pytest-env, pytest-forked, pytest-rerunfailures, pytest-timeout, pytest-xdist, Pytest, parameterized, pytest-asyncio, pytest-mock, and pytest-split.

Which repos have a stack like TensorRT-LLM's?

The ones that share most of its stack:ray-project/ray sgl-project/sglang vllm-project/vllm