Baseten: product and architecture
This Baseten review looks at a managed inference platform and the question it exists to answer: who runs the GPUs. Baseten serves open-source, custom, and fine-tuned models on infrastructure it operates, bills dedicated deployments per minute of active compute, and scales endpoints to zero so idle time is not charged. It is proprietary, and the open-source part of the story — Truss, its model-packaging format — is a component rather than the product. We evaluated it against its documentation, its published pricing, and the adoption signals we collect weekly.
Overview
Baseten is a commercial platform. Its public evidence is unusual and worth being explicit about: there is no repository that is the product, so it has no core adoption signal of the kind an open-source engine has. What we can measure is the vendor's own open-source component, Truss, which our 14 September 2026 snapshot recorded at about 1,200 GitHub stars and roughly 67,000 PyPI downloads a week, alongside 101 votes on its Product Hunt launch. Those are developer-adoption numbers for a packaging library, and we do not present them as product adoption.
Its market position is the high-performance end of managed inference. The vendor's framing is "the fastest model runtimes, cross-cloud high availability", and the named customers on its site — Notion, Cursor, Descript, HubSpot, Abridge, OpenEvidence among them — are consistent with a product bought for latency-sensitive production traffic rather than for experimentation.
The audience is product teams shipping an AI feature on an open or fine-tuned model who do not want to own a GPU fleet.
Key Features and Architecture
The platform has three surfaces. Dedicated deployments serve your model on infrastructure Baseten operates, sized and autoscaled per endpoint. Model APIs are pre-optimised endpoints for current open models — GLM-5.3, DeepSeek V4, Kimi K3 and others — billed per token, for prototyping or for workloads that do not need a dedicated deployment. Training runs on the same stack via a Loops SDK, so a model trained there deploys to inference without changing infrastructure.
Truss is the connective tissue and the part you can inspect. It is an MIT-licensed open-source format for packaging a model with its dependencies and serving code, and because it is portable, the same artefact moves between your environment and Baseten's. That is the practical answer to the lock-in question: your packaging is not proprietary even though the platform is.
Two operational properties are the reason teams pay. Cold starts are the number that decides whether scale-to-zero is usable: an endpoint that scales to zero but takes a minute to wake is not serving production traffic, and Baseten sells fast cold starts as a first-class feature. Cross-cloud deployment means workloads can run in Baseten's cloud or in yours, across regions, with a published 99.99% uptime target.
The performance story is a stack rather than a setting: custom kernels, current decoding techniques, and caching built into what the vendor calls the Baseten Inference Stack. That is a claim we cannot independently verify, and it is the kind of claim worth testing against your own model during a trial rather than accepting.
Ideal Use Cases
Four situations favour Baseten. The first is a latency-sensitive product feature on an open or fine-tuned model — the case where a metered frontier API is either too slow, too expensive, or does not host the model you need, and where owning GPUs is not something your team wants to do.
The second is variable traffic. Scale-to-zero with fast cold starts is the specific combination that makes bursty workloads affordable; without the cold-start work, scale-to-zero is a billing feature you cannot actually use.
The third is a model you trained or fine-tuned yourself. Serverless token APIs host a catalogue; a dedicated deployment hosts your weights, and Truss is how you hand them over without a bespoke integration.
The fourth is a data-residency or cloud requirement that rules out a single-region SaaS, which is what the cross-cloud, any-region option addresses.
Do not choose Baseten if you are serving a frontier model that its vendor already hosts — you would be paying for infrastructure to run something you can call directly. It is also the wrong answer if an open-source licence requirement covers your serving layer, because the platform is proprietary and only Truss is MIT. And if your traffic is steady and large enough, owning or renting GPUs and running vLLM or SGLang yourself will be cheaper; the platform's value is in the operations it removes, and at high steady utilisation that value shrinks against the bill.
Pricing and Licensing
Baseten is usage-based. The Basic plan is $0/month and pay-as-you-go, with Pro and Enterprise negotiating volume discounts against the same underlying rates rather than charging a platform fee.
Dedicated deployments bill per minute of active compute, which is the detail that makes the arithmetic work: T4 at $0.01052/min, L4 at $0.01414/min, A10G at $0.02012/min, H100 MIG at $0.0625/min, A100 80GB at $0.06667/min, H100 80GB at $0.10833/min, and B200 at $0.16633/min. CPU instances start at $0.00058/min. Endpoints scale to zero and idle time is not charged, so the meaningful number is not the hourly rate but the hourly rate multiplied by your actual utilisation.
Model APIs are billed per million tokens instead, which is the right shape for prototyping and for spiky low-volume use. New accounts receive complimentary credits. Training uses the same GPU rates as inference, so there is no separate training price list to reconcile.
We recommend modelling both shapes before committing. For a steady workload, multiply the per-minute rate by the hours you will genuinely be serving and compare against renting the same GPU elsewhere plus the engineering to run vLLM or SGLang on it. For a bursty workload, the scale-to-zero arithmetic usually favours Baseten by a wide enough margin that the comparison is not close.
Strengths & Trade-offs
Pros
- Per-minute billing with scale-to-zero, which prices bursty traffic honestly rather than charging for an idle GPU.
- A broad GPU range from T4 to B200, so you can match the accelerator to the model instead of over-provisioning.
- Truss is MIT-licensed and portable, so model packaging is not locked to the platform even though the platform is proprietary.
- Cross-cloud and any-region deployment, including into your own cloud, which answers most data-residency objections.
- Training and inference on one stack, so a fine-tuned model does not need a second deployment story.
Cons
- Proprietary. There is no repository that is the product, so it cannot satisfy an open-source-only requirement and it has no public adoption signal of its own.
- Performance claims about the inference stack are the vendor's and are not independently verifiable — test them on your own model during evaluation.
- At high steady utilisation the managed premium stops paying for itself against renting GPUs and running an open engine.
- Pro and Enterprise discounts are negotiated rather than published, so the price at volume is not knowable in advance.
- Per-minute rates reward accurate autoscaling configuration, so a badly tuned endpoint is a bill rather than an outage.