Braintrust: product and architecture
This Braintrust review covers a platform built in the opposite order from most of its category. Where an observability tool starts by recording production and adds scoring later, Braintrust starts from the eval — define what good looks like, score it in CI, block a bad release — and adds production logging as the thing that feeds the next eval set. If that ordering matches how your team wants to work, it is the strongest argument for choosing it. We evaluated it against its documentation, its published pricing, and the adoption signals we collect weekly.
Overview
Braintrust is a proprietary hosted platform. Its public evidence is limited by design and we should be plain about that: there is no repository that is the product, so it has no core adoption signal. What we can measure is the vendor's open-source work — braintrustdata/autoevals, an MIT-licensed scoring library, at about 1,000 GitHub stars — and the client SDKs, at roughly 1.5 million weekly PyPI downloads and 1.6 million weekly npm downloads. Those are developer-adoption numbers for libraries usable without the platform, and we record them in that lane rather than presenting them as product usage.
Its market position is evaluation-first AI engineering. The vendor's framing is "discover patterns in production, turn them into evals, and improve quality with every release", and the compliance surface — SOC 2 Type II, HIPAA, GDPR, granular per-project permissions — signals a product sold into organisations with a procurement process.
The audience is teams whose problem is not "what happened" but "how do we stop it happening again", and who want that answer enforced at release time.
Key Features and Architecture
Three pillars, and the order matters.
Evals are the centre. You define datasets, run experiments against them, compare prompts and models side by side, and score outputs with an LLM judge, with code, or with a human. Datasets are versioned, which is the detail that makes a regression claim meaningful — scoring against a moving test set tells you nothing.
Observability records what actually happened: every agent trace and tool call, searchable across millions of logs, with latency, cost, and quality tracked live. The design intent is that a production failure becomes a dataset row, which is the loop the whole product is arranged around.
CI enforcement is the part that differentiates it. Scored evals can block a release before it reaches production. That turns quality from a dashboard somebody checks into a gate that fails a build, and it is a different organisational fact.
Around the pillars sit a prompt playground for fast iteration over your own datasets, custom charts, environments, and annotation. Autoevals is the vendor's open-source scoring library, MIT-licensed and usable standalone — a reasonable way to try the scoring approach before buying the platform.
Deployment is hosted, with a hybrid option: the Brainstore data plane can run on your own infrastructure while the control plane stays with the vendor. That is a genuine answer to data residency, though not the same answer as a fully self-hosted product.
Ideal Use Cases
Four situations favour Braintrust. The first is establishing evaluation as a discipline. If your team ships prompt changes on judgement and finds out later, the value here is the workflow, not the feature list, and no amount of tracing supplies it.
The second is release gating. Blocking a build on a quality score is the mechanism that stops silent regression, and it is genuinely hard to assemble from parts.
The third is a regulated buyer. SOC 2 Type II, HIPAA, GDPR, granular permissions, and a hybrid data plane together clear procurement bars that a self-hosted open-source project usually has to answer one question at a time.
The fourth is cross-functional quality work. The vendor positions the platform for engineering and product together, and annotation plus custom views is what makes a non-engineer's judgement land in the same place as a test result.
Do not choose Braintrust if an open-source licence requirement covers this layer — the platform is proprietary and only Autoevals is MIT. It is also the wrong pick if you need to run everything inside your own network: the hybrid data plane is not a full self-hosted build, and Langfuse is the alternative that is. And if your immediate problem is simply seeing requests and costs, this is more platform than the question needs; Helicone is a smaller step.
Pricing and Licensing
Starter is $0/month and is unusually complete for a free tier: unlimited users, projects, datasets, playgrounds, and experiments, with $10/month of model credits, 1 GB of processed data, 10,000 scores, and 14-day retention. Beyond those allowances it is $4/GB of processed data and $2.50 per 1,000 scores.
Pro is $249/month with $100 of model credits, 5 GB of processed data, 50,000 scores, and 30-day retention, then $3/GB and $1.50 per 1,000 scores; keeping data longer than 30 days is $0.50/GB/month. Pro adds custom charts, environments, RBAC, the loop agent, and priority support. Qualifying startups get 6 to 12 months of Pro free.
Enterprise is quoted, with custom retention and export, premium support, and on-prem or hosted deployment.
The structure is worth reading carefully, because three meters run at once: processed data, scores, and model credits. Scores are the one that surprises, since an eval suite run on every commit multiplies quickly — 50,000 scores sounds generous until a twenty-case suite runs on fifty commits a day against five scorers. The model credits are a convenience rather than a saving; you can bring your own keys.
We recommend modelling a month of your intended CI cadence against the score allowance before choosing a tier, because that is the meter your workflow controls and the one the pricing is built around.
Strengths & Trade-offs
Pros
- Evaluation is the centre of the product rather than a stage of it, which is the right shape for teams trying to establish the discipline.
- Scored evals can block a release, turning quality into a build gate instead of a dashboard.
- A genuinely usable free tier: unlimited users, projects, and experiments, with real allowances attached.
- Versioned datasets, without which a regression claim does not mean anything.
- SOC 2 Type II, HIPAA, GDPR, granular permissions, and a hybrid data plane — a procurement story most open-source alternatives cannot match.
Cons
- Proprietary, with no repository that is the product. It cannot satisfy an open-source-only requirement and has no public adoption signal of its own.
- The hybrid data plane is not a full self-hosted build, so a strict "nothing leaves our network" policy is not satisfied.
- Three simultaneous meters — data, scores, credits — make the bill harder to predict than a single per-trace price.
- The score meter scales with CI cadence, so the pricing rewards running fewer evals, which is in tension with the product's own argument.
- Heavier than the problem if all you need today is request logging and cost tracking.