vLLM: product and architecture
This vLLM review looks at an inference engine, not a convenience runtime — the difference decides whether it is the right layer of your stack. vLLM is an Apache-2.0 Python server that loads open-weight models and serves them to many simultaneous callers behind an OpenAI-compatible API, and its entire design goal is keeping expensive accelerators busy. It is free with no paid tier, so the cost is GPU capacity. We evaluated it against its documentation, its published throughput benchmarks, and the GitHub repository that is the product, with attention to what it asks of you operationally.
Overview
vLLM began at UC Berkeley in February 2023 and now sits at roughly 93,000+ GitHub stars, making it one of the most-adopted open-source inference projects. It is Apache-2.0 licensed and governed as a project rather than a company product, which matters commercially: there is no vendor to negotiate with, no seat count, and no risk of a licence change to a source-available model.
Its market position is the production tier of self-hosted inference. Where a runtime like Ollama optimises for one developer getting a model running in a minute, vLLM optimises for throughput per GPU-hour across concurrent requests. The project published launch benchmarks in June 2023 on LLaMA-7B on an A10G and LLaMA-13B on a 40GB A100, reporting multiples of 14x to 24x against a Hugging Face Transformers baseline. We would not present those as current numbers: they are three years old, and one of the baselines used, Hugging Face Text Generation Inference, is now an archived project. Treat them as a statement of design intent and measure on your own model and hardware.
The audience is platform and ML infrastructure teams with real concurrency and their own accelerators, not individual developers.
Key Features and Architecture
The central technique is PagedAttention, which manages the KV cache in fixed-size blocks the way an operating system pages memory. That removes the fragmentation that otherwise forces conservative memory reservation per request, and it is what allows many sequences to share a GPU. Around it sit continuous batching, which interleaves arriving requests into the running batch rather than waiting for a fixed window, plus chunked prefill and prefix caching. Model execution uses piecewise and full CUDA/HIP graphs to cut per-step launch overhead.
Scale-out is first-class. vLLM supports tensor, pipeline, data, expert, and context parallelism, plus disaggregated prefill and decode, which lets you split a model across accelerators or split the two phases of inference onto differently-sized hardware. Quantisation coverage is unusually wide: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, and TorchAO.
Hardware support is the quiet differentiator. Beyond NVIDIA it runs on AMD GPUs, x86, ARM, and PowerPC CPUs, Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, and Apple Silicon. If your capacity is not NVIDIA, this is one of very few engines that will use it.
The API surface is an OpenAI-compatible server, an Anthropic Messages API, and gRPC, across 200+ model architectures including Mixture-of-Expert models such as DeepSeek-V3, hybrid Mamba architectures, and multi-modal models like LLaVA and Qwen-VL.
Ideal Use Cases
vLLM earns its complexity in four situations. The clearest is serving an open model to many concurrent users behind a product feature — the case where a runtime's single-request ceiling becomes the outage. The second is cost crossover: once monthly token spend on a hosted API exceeds the cost of GPU capacity plus the engineering to run it, self-hosting on vLLM is straightforwardly cheaper, and continuous batching is what makes that arithmetic work rather than leaving a GPU idle between requests.
Third is non-NVIDIA hardware. If you hold AMD, TPU, or Gaudi capacity, vLLM is one of the few engines that will actually use it, and that can decide a procurement question. Fourth is a model too large for one accelerator, where tensor and pipeline parallelism are the mechanism.
Do not choose vLLM as a developer's local model runner. It is a Python server expecting a discrete GPU and deliberate configuration, and Ollama or LocalAI will have you running in a fraction of the time with none of the tuning. It is also the wrong answer if you have no one to operate it: unlike a managed API, capacity planning, upgrades, and incident response are yours.
Pricing and Licensing
vLLM is free and Apache-2.0 licensed, with no paid tier at all. There is no licence fee, no per-seat charge, and no per-token bill, and because it is a project rather than a company there is no vendor relationship to manage. The Apache-2.0 licence is also permissive enough to satisfy a stack that must be open source end to end, including commercial redistribution — a bar no managed model API meets.
The real cost is infrastructure, and it is not small. Production inference means GPU capacity, whether rented or owned, plus the engineering time to configure parallelism, choose a quantisation format, and keep the service healthy. That is the honest comparison against a hosted provider: Together AI charges from $0.04 per million input tokens to $15.00 per million output tokens with zero operational burden, and Modal offers a free Starter tier with a paid Team plan. vLLM's cost curve starts higher and flattens, so it wins at volume and loses at low usage.
We recommend estimating the crossover before committing. Take your current monthly token bill, price the GPU capacity that would serve the same load with continuous batching, and add a realistic engineering allowance. Below that crossover, a metered API is the cheaper answer even though vLLM's software is free.
Strengths & Trade-offs
Pros
- Apache-2.0 with no paid tier, so it satisfies an open-source licence requirement and carries no risk of a relicensing surprise.
- PagedAttention and continuous batching deliver the throughput the project was built for, with published benchmarks against named baselines and hardware.
- Genuinely broad hardware support — AMD, TPU, Gaudi, Ascend, Apple Silicon, and CPUs — which few competing engines match.
- Distributed serving via tensor, pipeline, data, expert, and context parallelism, so a model larger than one accelerator is a configuration rather than a blocker.
- OpenAI-compatible API means adopting it is usually a base URL change for callers.
Cons
- Operationally demanding: a Python server to deploy, tune, and keep healthy, versus a single binary or an API key.
- Wants a discrete GPU for the workloads it is good at, so the entry cost is hardware rather than software.
- The published throughput figures date from June 2023 and one baseline, TGI, is now an archived project, so they need re-measuring on your own model and hardware rather than being taken as current.
- No vendor means no support contract, which some procurement processes will not accept.