LiteLLM: product and architecture
This LiteLLM review looks at the layer most AI stacks add second and wish they had added first: a gateway. LiteLLM puts one OpenAI-shaped API in front of 100-plus model providers, and its self-hosted proxy adds the things a platform team actually needs — virtual keys, budgets, rate limits, and per-team spend tracking. The core is free to self-host under the MIT licence; the enterprise build is quoted. We evaluated it against its documentation, its GitHub repository, and the adoption signals we collect weekly.
Overview
LiteLLM is published by BerriAI. Our 14 September 2026 snapshot recorded about 59,100 GitHub stars and roughly 21.8 million PyPI downloads a week, with the v1.101.0 release in mid-September 2026. The repository describes a Rust core with a Python SDK, and the repository's own LICENSE splits the project: everything outside the enterprise/ directory is MIT, and that directory is separately licensed.
Its market position is the platform team's control point. The product is sold to the people who have to answer "who is spending what on which model", not to the developer writing the prompt. That framing shows up in the named references — NVIDIA, Netflix, Okta, Lemonade, AT&T all appear as customers on the vendor's own site — and in the feature list, which is mostly governance.
The audience is organisations with more than one model provider, more than one team calling them, and somebody accountable for the bill.
Key Features and Architecture
LiteLLM ships in two shapes and the distinction matters. The SDK is a Python library: you call completion() and it translates to whichever provider the model string names. The proxy is a server you deploy, and it is where the platform features live. A team that installs the SDK and concludes LiteLLM is a thin translation layer has looked at half the product.
The translation itself covers 100-plus providers in OpenAI format, with native formats available where you need them. That is the feature that makes a model swap a configuration change rather than a code change, and it is the one customers describe as saving procurement and security review cycles rather than engineering time.
The governance surface is the substance of the proxy. Virtual keys issue per-team or per-user credentials that map onto your real provider keys, so a revoked internal key does not mean rotating a provider secret. Budgets cap spend before it runs rather than reporting it afterwards. Rate limits are enforced per key as RPM and TPM. Spend tracking attributes cost by key, user, team, and organisation. Audit logs, SSO with SCIM, OIDC and JWT auth, and RBAC are enterprise-tier additions.
Routing is the cost lever. Load balancing spreads requests across providers, regions, and keys; lowest-cost routing sends a request to the cheapest deployment that can serve it; auto-routing sends simple prompts to cheaper models and hard ones to capable models. LiteLLM also acts as an MCP gateway, which extends the same key and policy model to tool servers.
Observability integrates outward rather than inward: Langfuse, Arize Phoenix, LangSmith, and plain OpenTelemetry are all supported logging destinations. LiteLLM does not try to be your tracing backend.
Ideal Use Cases
Four situations make LiteLLM close to mandatory. The first is multi-provider by necessity — you use one vendor for reasoning, another for cheap classification, a third for embeddings — and you do not want that choice compiled into application code. The second is multi-team access, where the real problem is not the API call but issuing and revoking credentials without a procurement cycle each time.
The third is cost control with teeth. A budget that caps spend is a different product from a dashboard that reports it, and lowest-cost and auto-routing are the mechanisms that reduce the bill rather than just describe it. The fourth is regulated or air-gapped deployment, where the gateway has to run inside your network and the enterprise build's air-gapped option is the reason to pay.
Do not deploy the LiteLLM proxy if you have one provider, one team, and no budget question — the SDK alone, or the provider's own client, is less to run. It is also not an observability product: it sends logs to Langfuse or Phoenix and expects one of those to exist. And it is a hop in your request path, so it is one more service to keep available; a single-region deployment in front of a critical feature is a new failure mode, which is why the enterprise tier sells a multi-region control plane.
Pricing and Licensing
The gateway is free to self-host. Everything outside the repository's enterprise/ directory is MIT-licensed, and the free build is not a crippled demo: 140-plus provider integrations, virtual keys, budgets and teams, load balancing, RPM and TPM limits, LLM guardrails, and logging to Langfuse, Arize Phoenix, LangSmith and OTEL are all in it, with no credit card.
LiteLLM Enterprise is quoted per deployment rather than listed. It adds SSO and SCIM, OIDC and JWT auth, RBAC, audit logs, secret-manager integration, a multi-region control plane, self-hosted and air-gapped deployment, dedicated onboarding, and 24/7 support with SLAs. There is a 30-day trial at no cost and no credit card. We would rather quote a number here, and the vendor does not publish one — the honest reading is that the free tier covers the technical problem and the paid tier covers the compliance and support problem.
The cost that is not LiteLLM's is the important one: the gateway takes no cut of tokens. You keep paying each model provider directly at their own rates, which is the structural difference from a hosted gateway such as OpenRouter, which resells access and earns on credit purchases. With LiteLLM the commercial relationship with the model vendors stays yours, and so does the negotiating position.
Against the alternatives, this is the cheapest gateway to start with and the one with the most operational burden. Budget for infrastructure and an on-call rotation rather than a licence.
Strengths & Trade-offs
Pros
- The free self-hosted build includes the governance features — virtual keys, budgets, rate limits, spend tracking — that most vendors reserve for a paid tier.
- No cut of token spend. Provider relationships and rates stay yours, which also keeps the exit cheap.
- 100-plus providers behind one OpenAI-shaped API, so changing model is configuration rather than code, procurement, and a security review.
- Routing that reduces cost rather than only reporting it: load balancing, lowest-cost routing, and auto-routing by prompt difficulty.
- Logs outward to Langfuse, Arize Phoenix, LangSmith, or plain OpenTelemetry instead of locking tracing to itself.
Cons
- The proxy is a service you run, and it sits in the request path. That is a new availability dependency in front of everything.
- Enterprise pricing is not published, so the cost of SSO, audit logs, and support cannot be estimated without a sales conversation.
- The repository's LICENSE is a split rather than a single identifier, and automated licence checks read it as unresolved — you have to read the file to know that the core is MIT.
- Breadth across 100-plus providers means edge-case behaviour differs by provider, and a feature that works on one may translate imperfectly on another.
- The SDK and the proxy are often confused, and teams that adopt only the SDK miss most of what the product is for.