Decision comparison
SGLang vs TensorRT-LLM vs vLLM
All three are Apache-2.0, all three batch continuously, and all three answer an OpenAI-shaped request, so the licence and the interface will not decide this. Three other things will. The first is hardware: TensorRT-LLM runs on NVIDIA and nothing else, while vLLM and SGLang run on almost anything, and that is a procurement question before it is an engineering one. The second is prompt shape: if your traffic shares long prefixes, SGLang's cache reuse is a mechanism the other two do not have in the same form. The third is whether an ahead-of-time build step is acceptable, because TensorRT-LLM's performance comes from work done before serving, and that work is repeated every time the model changes. Benchmark all three on your own model and your own traffic; none of the published numbers will settle it.
Direct comparison. These are reviewed substitutes bought for the same job, so the differences below are the ones that decide between them.
All 3 are model runtimes.
Quick Comparison
| Decision factor | SGLang | TensorRT-LLM | vLLM |
|---|---|---|---|
| Best for | Workloads where prompts repeat. A long system prompt, a retrieval template, a few-shot block or a multi-turn conversation all mean the engine is asked to process the same tokens again and again, and SGLang is built to stop doing that. Hosted under the LMSYS non-profit, which removes the risk of a vendor moving features behind a commercial edition later. | A committed NVIDIA fleet at high utilisation, where tokens per GPU-hour is the number that decides the budget. It is written by the company that designed the accelerators, and low-precision formats land here first for that reason. | The default when you want production serving without committing to one vendor's silicon or to a build pipeline. It began at UC Berkeley in February 2023 and is governed as a project rather than a company product, so there is no vendor to negotiate with and no relicensing risk. |
| The technique it is built around | RadixAttention, which keeps a radix tree of cached prefixes and starts generation from the longest match rather than recomputing the whole prompt. Around it sit prefix caching generally, a zero-overhead batch scheduler that interleaves arriving requests into the running batch, and grammar-constrained decoding that produces valid JSON at the decoding level rather than through a validate-and-retry loop. | Ahead-of-time optimisation. The library does work before serving that the others do at request time, compiling for a specific model and hardware target. In-flight batching and paged KV cache sit on top, with speculative decoding via EAGLE, MTP and NGram, and FP8 on H100 and later and FP4 on B200 — which NVIDIA documents as able to double performance and halve memory against 16-bit floating point. | PagedAttention, which manages the KV cache in fixed-size blocks the way an operating system pages memory, removing the fragmentation that otherwise forces conservative per-request memory reservation. Continuous batching interleaves arrivals, and chunked prefill, prefix caching and CUDA/HIP graphs sit around it. |
| Hardware it runs on | NVIDIA, AMD, Intel Xeon, Google TPU, Ascend NPU and Moore Threads MUSA. Broad enough that owned non-NVIDIA capacity is usable capacity. | NVIDIA GPUs only, tuned for H100, H200, B200 and Blackwell. That is the product rather than a limitation to work around, and it is a procurement commitment as much as a technical one. | NVIDIA, AMD, x86, ARM and PowerPC CPUs, Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend and Apple Silicon. The widest coverage of the three, which can decide a procurement question rather than an engineering one. |
| What you do before serving | Install and configure. There is no compilation step: pip, source or a container image, then `launch_server` and requests are answered. | Build an engine, per model and per hardware target. That pipeline is yours to own, and it is the cost teams underestimate: a stable model amortises it, a model you change weekly does not. | Install and configure, with no compilation step. Quantisation coverage is unusually wide — FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt and TorchAO — so a format decision does not mean an engine decision. |
| Licence and cost | Apache 2.0 with no paid tier, published by a non-profit. The cost is the GPU capacity and the engineering to tune parallelism and cache sizing. | Apache 2.0 with no paid tier. The repository LICENSE is a composite that GitHub reports as unresolved, so an automated open-source check cannot confirm it without a human reading the file. Support is sold separately through NVIDIA AI Enterprise. | Apache 2.0 with no paid tier and no vendor. The cost is GPU capacity plus the engineering to configure parallelism and keep the service healthy. |
| When it is the wrong choice | When your prompts share no prefix. The reason to prefer it over the alternatives then largely evaporates, and you are choosing on general throughput, which needs measuring rather than assuming. Also wrong as a developer's local runner. | On any accelerator that is not NVIDIA's, and on a model roster that changes often enough that the build step is paid repeatedly. | As a developer's local model runner, where Ollama or LocalAI is answering in a fraction of the setup time. And on a fully committed NVIDIA fleet at high utilisation, where an engine written for that silicon is worth the build step. |
SGLang
- Best for:
- Workloads where prompts repeat. A long system prompt, a retrieval template, a few-shot block or a multi-turn conversation all mean the engine is asked to process the same tokens again and again, and SGLang is built to stop doing that. Hosted under the LMSYS non-profit, which removes the risk of a vendor moving features behind a commercial edition later.
- The technique it is built around:
- RadixAttention, which keeps a radix tree of cached prefixes and starts generation from the longest match rather than recomputing the whole prompt. Around it sit prefix caching generally, a zero-overhead batch scheduler that interleaves arriving requests into the running batch, and grammar-constrained decoding that produces valid JSON at the decoding level rather than through a validate-and-retry loop.
- Hardware it runs on:
- NVIDIA, AMD, Intel Xeon, Google TPU, Ascend NPU and Moore Threads MUSA. Broad enough that owned non-NVIDIA capacity is usable capacity.
- What you do before serving:
- Install and configure. There is no compilation step: pip, source or a container image, then `launch_server` and requests are answered.
- Licence and cost:
- Apache 2.0 with no paid tier, published by a non-profit. The cost is the GPU capacity and the engineering to tune parallelism and cache sizing.
- When it is the wrong choice:
- When your prompts share no prefix. The reason to prefer it over the alternatives then largely evaporates, and you are choosing on general throughput, which needs measuring rather than assuming. Also wrong as a developer's local runner.
TensorRT-LLM
- Best for:
- A committed NVIDIA fleet at high utilisation, where tokens per GPU-hour is the number that decides the budget. It is written by the company that designed the accelerators, and low-precision formats land here first for that reason.
- The technique it is built around:
- Ahead-of-time optimisation. The library does work before serving that the others do at request time, compiling for a specific model and hardware target. In-flight batching and paged KV cache sit on top, with speculative decoding via EAGLE, MTP and NGram, and FP8 on H100 and later and FP4 on B200 — which NVIDIA documents as able to double performance and halve memory against 16-bit floating point.
- Hardware it runs on:
- NVIDIA GPUs only, tuned for H100, H200, B200 and Blackwell. That is the product rather than a limitation to work around, and it is a procurement commitment as much as a technical one.
- What you do before serving:
- Build an engine, per model and per hardware target. That pipeline is yours to own, and it is the cost teams underestimate: a stable model amortises it, a model you change weekly does not.
- Licence and cost:
- Apache 2.0 with no paid tier. The repository LICENSE is a composite that GitHub reports as unresolved, so an automated open-source check cannot confirm it without a human reading the file. Support is sold separately through NVIDIA AI Enterprise.
- When it is the wrong choice:
- On any accelerator that is not NVIDIA's, and on a model roster that changes often enough that the build step is paid repeatedly.
vLLM
- Best for:
- The default when you want production serving without committing to one vendor's silicon or to a build pipeline. It began at UC Berkeley in February 2023 and is governed as a project rather than a company product, so there is no vendor to negotiate with and no relicensing risk.
- The technique it is built around:
- PagedAttention, which manages the KV cache in fixed-size blocks the way an operating system pages memory, removing the fragmentation that otherwise forces conservative per-request memory reservation. Continuous batching interleaves arrivals, and chunked prefill, prefix caching and CUDA/HIP graphs sit around it.
- Hardware it runs on:
- NVIDIA, AMD, x86, ARM and PowerPC CPUs, Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend and Apple Silicon. The widest coverage of the three, which can decide a procurement question rather than an engineering one.
- What you do before serving:
- Install and configure, with no compilation step. Quantisation coverage is unusually wide — FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt and TorchAO — so a format decision does not mean an engine decision.
- Licence and cost:
- Apache 2.0 with no paid tier and no vendor. The cost is GPU capacity plus the engineering to configure parallelism and keep the service healthy.
- When it is the wrong choice:
- As a developer's local model runner, where Ollama or LocalAI is answering in a fraction of the setup time. And on a fully committed NVIDIA fleet at high utilisation, where an engine written for that silicon is worth the build step.
Public signals
Verified factual signals only. Bars appear only for like-for-like metrics with five weekly assessments for every tool; missing evidence stays explicit. These signals do not establish enterprise adoption, product quality, or total cost.
| Metric | SGLang | TensorRT-LLM | vLLM |
|---|---|---|---|
| Docker Hub pulls(Product adoption) | 13.5M | Not available | Not available |
| GitHub commits, 90d(Product adoption) | 4.2k | 2.5k | 3.8k |
| GitHub stars(Product adoption) | 36,000+ | 14,000+ | 92,000+ |
| PyPI weekly downloads(Product adoption) | 574.0k | 6.5k | 408.7k |
As of September 21, 2026 — updated weekly.
Health & risk evidence
Observed public-source checks for mapped package versions and repositories.
SGLang
September 21, 2026Package vulnerabilities
PyPI · sglang@0.5.20
0 vulnerabilities
across 1 package
Repository security score
Not available
TensorRT-LLM
September 21, 2026Package vulnerabilities
PyPI · tensorrt-llm@1.2.1
0 vulnerabilities
across 1 package
Repository security score
Not available
vLLM
September 21, 2026Package vulnerabilities
PyPI · vllm@0.29.0
0 vulnerabilities
across 1 package
Repository security score
Not available
Feature Comparison
| Feature | SGLang | TensorRT-LLM | vLLM |
|---|---|---|---|
| Serving architecture | |||
| Batching | Zero-overhead batch scheduler interleaving arrivals into the running batch | In-flight batching managing request execution dynamically | Continuous batching interleaving arrivals rather than waiting for a window |
| KV cache management | Radix tree of cached prefixes, reused across requests that share one | Paged KV cache with block reuse | PagedAttention: fixed-size blocks, paged like operating-system memory |
| Ahead-of-time build step | Install and launch: pip, source or a container image | Required, per model and per hardware target | Install and launch, with no compilation stage |
| Speculative decoding | Full support | EAGLE, MTP and NGram | Full support |
| Structured-output decoding | Grammar and JSON-schema constrained at the decoding level | Guided decoding, combinable with speculative decoding | Supported via structured-output backends |
| Hardware | |||
| NVIDIA GPUs | Full support | The only target, tuned for H100, H200, B200 and Blackwell | Full support |
| AMD GPUs | Full support | NVIDIA GPUs are the documented target throughout | Full support |
| Google TPU | Full support | NVIDIA GPUs are the documented target throughout | Full support |
| CPUs | Intel Xeon | NVIDIA GPUs are the documented target throughout | x86, ARM and PowerPC |
| Other accelerators | Ascend NPU and Moore Threads MUSA | NVIDIA GPUs are the documented target throughout | Intel Gaudi, IBM Spyre, Huawei Ascend and Apple Silicon |
| Scale-out and interface | |||
| Multi-GPU parallelism | Multi-GPU parallelism from a single card to distributed clusters | Tensor, pipeline and expert parallelism across GPUs and nodes | Tensor, pipeline, data, expert and context parallelism, plus disaggregated prefill and decode |
| Serving interface | OpenAI-compatible API, Hugging Face compatible | `trtllm-serve` for online serving and a Python LLM API for offline work | OpenAI-compatible server, Anthropic Messages API and gRPC |
| Measurement tooling | Published benchmarks and a public development meeting | `trtllm-bench` and `trtllm-eval` ship with the library | Published benchmarks; the June 2023 launch figures are three years old and need re-measuring |
| Ecosystem integrations | Integrated with major reinforcement-learning frameworks | NVIDIA Dynamo and Triton Inference Server, first-party | 200+ model architectures including MoE, Mamba hybrids and multi-modal |
| Licence and governance | |||
| Licence | Apache 2.0 | Apache 2.0, in a composite LICENSE that GitHub reports as unresolved | Apache 2.0 |
| Paid tier of the engine | Apache 2.0 throughout, published by the LMSYS non-profit | Apache 2.0; NVIDIA AI Enterprise is a separate portfolio subscription | Apache 2.0 throughout, governed as a project rather than a company product |
| Who governs it | The LMSYS non-profit | NVIDIA, which also sells the hardware | A project rather than a company product, begun at UC Berkeley |
Serving architecture
Batching
KV cache management
Ahead-of-time build step
Speculative decoding
Structured-output decoding
Hardware
NVIDIA GPUs
AMD GPUs
Google TPU
CPUs
Other accelerators
Scale-out and interface
Multi-GPU parallelism
Serving interface
Measurement tooling
Ecosystem integrations
Licence and governance
Licence
Paid tier of the engine
Who governs it
Which to choose
All three are Apache-2.0, all three batch continuously, and all three answer an OpenAI-shaped request, so the licence and the interface will not decide this. Three other things will. The first is hardware: TensorRT-LLM runs on NVIDIA and nothing else, while vLLM and SGLang run on almost anything, and that is a procurement question before it is an engineering one. The second is prompt shape: if your traffic shares long prefixes, SGLang's cache reuse is a mechanism the other two do not have in the same form. The third is whether an ahead-of-time build step is acceptable, because TensorRT-LLM's performance comes from work done before serving, and that work is repeated every time the model changes. Benchmark all three on your own model and your own traffic; none of the published numbers will settle it.
Best-fit scenarios
Choose vLLM if:
Choose vLLM as the default, and depart from it for a reason. It has the widest hardware coverage of the three — NVIDIA, AMD, TPU, Gaudi, Ascend, Apple Silicon and several CPU architectures — which means owned capacity is usable capacity rather than a procurement constraint. Its quantisation list is long enough that choosing a format does not mean choosing a different engine. There is no build step, so swapping models is loading weights. It is governed as a project rather than a company product, so there is no vendor to negotiate with and no path to a source-available relicensing. Accept in exchange that you are not getting silicon-specific tuning from the hardware vendor, and that the project's published throughput figures date from June 2023 against a baseline that has since been archived — measure on your own model rather than reading them as current.
Choose SGLang if:
Choose SGLang when your traffic repeats itself. A long system prompt, a retrieval template, a shared few-shot block or multi-turn chat all mean the same tokens arrive again and again, and RadixAttention reuses the KV cache across those requests instead of recomputing. That is a specific mechanism for a specific workload property, not a general speed claim, and it is the honest reason to pick this engine. Grammar-constrained decoding is the second reason: valid JSON at the decoding level costs less than a retry loop at high request rates. Hardware coverage spans NVIDIA, AMD, Xeon, TPU, Ascend and MUSA, and the LMSYS non-profit governs it, so no vendor has an incentive to introduce a paid tier later. Accept that if your prompts share no prefix, the main reason to prefer it does not apply, and that you will not know until you measure.
Choose TensorRT-LLM if:
Choose TensorRT-LLM when the fleet is NVIDIA, the models are stable, and GPU-hour cost is the constraint you are optimising. FP8 on H100 and later and FP4 on B200 arrive here first because the hardware and the library ship from the same company, and NVIDIA documents them as doubling performance and halving memory against 16-bit floating point — figures worth measuring on your model rather than accepting or dismissing. If you already run Triton Inference Server or NVIDIA Dynamo, this is the engine those products are built to drive. Accept two costs. The first is hardware lock-in, which is the product rather than a limitation: no AMD, no TPU, no Gaudi. The second is the build step, a pipeline you own that runs per model and per hardware target, and that a team iterating weekly pays every time while the other two never charge it.
These scenarios reflect the available product evidence. Your requirements, existing stack, and team expertise should guide the final decision.
Frequently Asked Questions
Which of the three is fastest?
That question does not have a portable answer, which is why this page does not give one. Throughput depends on the model, the quantisation format, the accelerator and — for SGLang in particular — how much your prompts repeat. All three ship benchmarking tools, and TensorRT-LLM includes `trtllm-bench` specifically. Run a week of your own traffic through each before deciding; every published figure was measured on somebody else's workload.
Do I have to pay for any of them?
No. All three are Apache 2.0 with no paid tier of the engine. The costs are GPU capacity and engineering time. NVIDIA sells support through NVIDIA AI Enterprise, which is a separate portfolio subscription rather than a tier of TensorRT-LLM, and is the answer when procurement requires a contract behind an open-source component.
What is the build step, and why does only one of them have it?
TensorRT-LLM does optimisation work before serving, compiling for a specific model and hardware target, which is where its performance comes from. vLLM and SGLang interpret at request time instead. The trade is that the build must be repeated per model and per hardware target: a stable roster amortises it and a changing one pays it repeatedly.
Can any of them run on AMD or TPU hardware?
vLLM and SGLang can. vLLM covers NVIDIA, AMD, TPU, Gaudi, Ascend, Apple Silicon and x86, ARM and PowerPC CPUs; SGLang covers NVIDIA, AMD, Intel Xeon, TPU, Ascend and MUSA. TensorRT-LLM targets NVIDIA GPUs only, and that is deliberate rather than a gap awaiting a release.
When should I use a managed provider instead of any of these?
When you do not want to own the operations. Baseten bills dedicated GPUs per minute of active compute with scale to zero; Fireworks AI and Together AI bill per million tokens. Below a utilisation crossover those are cheaper than owning hardware once engineering time is counted, and above it they are not. Several of them run engines like these underneath, so the question is who operates the GPU rather than which software does the serving.