Ollama: product and architecture
This Ollama review covers a model runtime rather than a model provider — a distinction that decides whether it belongs in your stack at all. Ollama is an MIT-licensed Go program that downloads open-weight models and serves them on your own machine behind an OpenAI-compatible HTTP API. You point a coding agent, an application, or a notebook at localhost:11434 instead of a vendor endpoint, and no prompt leaves your hardware. We evaluated it against its public documentation, its pricing page, and the GitHub repository that is the product, with attention to where self-hosting stops paying off.
Overview
Ollama started in June 2023 and is now among the most-adopted open-source AI projects on GitHub, at roughly 180,000 stars — an adoption signal that matters here because a runtime's value depends on how quickly it supports new model releases, and that tracks contributor attention. The project is written in Go, ships as a single installable for macOS, Windows, Linux, and Docker, and has official Python and JavaScript SDKs.
Its market position has shifted. Early Ollama was a convenience wrapper for running Llama locally; the current product leads with coding agents, positioning itself as the way to "use open models with your coding agents so you can spend less while keeping your data private." Ollama also now sells a hosted tier for models too large for consumer hardware, served from the US, Europe, and Singapore. That makes it two things at once: a free local runtime, and a metered inference vendor. Buyers should be clear which one they are adopting, because the economics are opposite.
The target audience is developers and small teams who already have capable hardware — an Apple Silicon machine or a GPU workstation — and who value data locality or predictable cost over frontier model quality.
Key Features and Architecture
The core is a local HTTP server on port 11434 with a CLI in front of it. ollama run gemma4 pulls a quantised model, loads it, and opens an interactive session; ollama launch claude wires it into Claude Code. Models come from a curated library spanning chat, coding, vision, embedding, and reasoning families, so switching models is a pull rather than an integration project.
Two compatibility layers do most of the integration work. Ollama exposes an OpenAI-compatible API and an Anthropic-compatible API, which means most SDKs and agent frameworks work by changing a base URL and nothing else. In practice this is the feature that makes Ollama a drop-in: LangChain, LlamaIndex, and anything speaking the OpenAI REST shape connect without a custom adapter. The documented capability set covers streaming, tool calling, structured outputs, thinking, vision, and embeddings, plus a documented web search integration.
Named integrations include Claude Code, Codex, OpenCode, VS Code, and n8n. Embeddings are documented for semantic search and RAG, which lets a retrieval stack run end-to-end locally when paired with a self-hosted vector database such as Qdrant.
The architecture's limits follow from where inference happens. Model choice is bounded by VRAM, so quantisation is not optional at consumer scale, and quantised weights cost accuracy. Throughput is single-machine: the free tier allows one concurrent request, and Ollama does not do the continuous batching or paged attention that a production serving engine uses to keep a GPU saturated across many callers.
Ideal Use Cases
Ollama fits best in four situations. For an individual developer or a two-to-five person team running coding agents daily, replacing a metered API with local inference removes the per-token bill entirely, and the hardware is usually already bought. For regulated or privacy-sensitive work — legal, health, internal HR documents — the fact that prompts never leave the machine is a compliance argument rather than a preference, and it is far easier to make than a data processing agreement.
Local RAG and embedding workloads are a strong fit for the same reason: embedding a document corpus is exactly the kind of high-volume, low-value-per-call work that is expensive to meter and cheap to run locally. And for prototyping, developing against Ollama's OpenAI-compatible endpoint means the production switch to a hosted provider is a base URL change.
Do not use Ollama as your production inference tier for a multi-user application. One concurrent request on the free tier is the honest signal: this is a single-operator runtime, and serving many simultaneous users needs an engine built for it. It is also the wrong choice when you need frontier-model quality, because the open weights it runs still trail the best closed models on hard reasoning.
Strengths & Trade-offs
Pros
- MIT-licensed runtime, which makes it usable in stacks that require an open-source licence end to end — a bar every managed model API fails by definition.
- OpenAI- and Anthropic-compatible APIs turn integration into a base URL change rather than an adapter.
- Single-binary install across macOS, Windows, Linux, and Docker; no Python environment or CUDA setup to manage first.
- Curated model library covering chat, coding, vision, embeddings, and reasoning, so model swaps are a pull command.
- Local execution removes both the per-token bill and the data processing agreement.
Cons
- Not a production serving tier: one concurrent request free, ten at $500/mo, and no continuous batching means GPU utilisation stays low under real concurrency.
- Model quality is capped by what fits in your VRAM after quantisation, and quantisation costs accuracy on hard tasks.
- The paid cloud's three-to-one credit ratio means it is worth arithmetic against per-token providers rather than assumed cheaper.
- Two products under one name — a free local runtime and a metered vendor — which makes "we use Ollama" ambiguous about cost and data locality.