Auditi: product and architecture
Our verdict: this Auditi review finds a focused open-source option for teams that need AI-agent tracing and evaluation in the same workflow, without adopting a closed-source platform or assembling separate logging and evaluation systems. Auditi’s strongest fit is a technical team that can self-host and wants automatic capture of model calls, automated judging, human review, cost visibility, and dataset generation in one interface. It is not a proven broad enterprise observability standard: the public GitHub repository has 4 stars, and our research does not establish customer scale, benchmark results, hosted-service availability, or a mature integration ecosystem.
Overview
Auditi is an AI-agent observability tool intended to continuously monitor and update AI-agent performance and behaviour. Its core proposition is unusually direct: combine tracing and evaluation rather than treating them as separate products or leaving evaluation to custom scripts. For data engineers and analytics engineers operating LLM-enabled services, that is a meaningful design decision because production traces become the starting point for quality assessment rather than an isolated debugging artifact.
The project describes itself as an open-source tool with tracing, evaluation, human annotation workflows, real-time cost tracking, and the ability to turn production traces into fine-tuning datasets. It supports automatic instrumentation for OpenAI, Anthropic, and Google API calls, which makes it relevant to teams using more than one model provider. The stated setup path is self-hosting with docker compose up, a practical advantage for teams that need to run the tool in their own environment.
Auditi’s public repository is licensed under MIT, uses JavaScript as its primary language, and had 4 GitHub stars in the repository data. The repository was last pushed on 2026-04-22T12:56:56Z. Those are public activity and adoption signals, not proof of enterprise adoption, production reliability, or support capacity.
We recommend Auditi for engineering-led teams that value inspectable open-source software and are prepared to own deployment and operational responsibility. Choose a more established commercial platform instead if procurement requires defined support commitments, published enterprise references, or a documented hosted operating model. Auditi’s appeal is consolidation and control; its trade-off is that buyers must validate maturity for themselves.
Key Features and Architecture
Auditi’s technical center is automatic trace capture. The tool states that 2-line auto-instrumentation captures all OpenAI, Anthropic, and Google API calls, reducing the amount of manual logging required to observe model-provider interactions. That is especially useful when an application routes work across providers, because the same observability layer can capture calls from all three named ecosystems.
The second major capability is built-in evaluation. Auditi states that 7+ LLM-as-Judge evaluators run automatically on traces, tying quality assessment directly to the execution records collected from production systems. This distinguishes the proposed workflow from logging-only tools, where teams may have to export traces, define a separate evaluator pipeline, and reconcile results afterward.
Key product capabilities described for Auditi include:
- Two-line auto-instrumentation: captures OpenAI, Anthropic, and Google API calls without requiring teams to hand-build tracing around every request.
- Trace-linked automated evaluation: runs 7+ LLM-as-Judge evaluators automatically on captured traces, creating a direct path from observed execution to quality signals.
- Human annotation workflows: provides a review path for cases where AI judges are not sufficient, which matters for ambiguous outputs and domain-specific acceptance criteria.
- Real-time cost tracking: tracks costs as part of the monitoring workflow, allowing model usage and spending to be examined alongside behaviour and evaluation outcomes.
- Fine-tuning dataset creation: turns production traces into datasets for fine-tuning, connecting observed production interactions with model-improvement work.
- Self-hosted deployment: the project says it can be self-hosted with
docker compose up, placing deployment control with the user’s team. - Technical stack disclosure: Auditi lists a Python SDK, FastAPI, and React, while the GitHub repository identifies JavaScript as its primary language.
This architecture favors a feedback loop: capture calls, evaluate traces, involve humans where automated judges are inadequate, then use production evidence to build fine-tuning datasets. The value is not merely that every component exists; it is that Auditi positions them as one workflow. The cost is operational ownership: self-hosting means the team must assess its own security controls, deployment practices, upgrades, availability requirements, and internal support model.
Auditi's Capterra listing also lists documentation and live online training, online support, English language support, and organizational coverage spanning small businesses, mid-size businesses, enterprises, freelancers, nonprofits, governments, and startups. However, that material is mixed with descriptions of audit-evidence management features such as compliance management, audit planning, mobile access, and multi-year planning. We would not use those audit-workflow claims to evaluate the AI-agent observability product; the product identity and feature descriptions conflict with Auditi’s GitHub repository and technical description.
Ideal Use Cases
Auditi is best suited to a small platform or data team running an LLM-enabled application that calls OpenAI, Anthropic, Google, or a combination of those providers. A team of 3 to 10 engineers can use the stated 2-line instrumentation approach to establish a shared trace record before it has the time or desire to build a custom observability layer. The automatic 7+ LLM-as-Judge evaluators then give that team an initial evaluation mechanism that can be inspected and supplemented with human annotations.
A second strong use case is an analytics engineering or AI quality team managing production feedback for a customer-facing agent. The team can use trace-linked evaluation to investigate quality behavior while real-time cost tracking provides a separate operational lens on model usage. When stakeholders need a human decision for edge cases, the stated annotation workflows offer a route that does not force teams to pretend automated judges are authoritative in every situation.
A third use case is a self-hosting organization that wants to convert actual production interactions into fine-tuning data. For example, a 10-to-25-person data and AI function supporting a regulated internal workflow may prefer docker compose up deployment and an MIT-licensed codebase over sending traces to a closed-source service. Auditi’s Python SDK, FastAPI, and React components are useful context for teams evaluating whether the implementation matches their internal engineering capabilities.
Do not use Auditi if your decision depends on demonstrated enterprise adoption, published performance benchmarks, confirmed managed hosting, or named service-level commitments. Auditi's public documentation establishes the project’s functionality and MIT license, but it does not establish those operational assurances. Avoid it as a general governance, compliance, or audit-evidence platform as well: its Capterra listing contains similarly named audit-management descriptions, but those claims do not cleanly describe the AI-agent observability tool reviewed here.
We recommend Auditi for teams that need an integrated, self-hosted feedback loop around AI-agent behavior and are comfortable validating it in a limited production pilot. Start with one agent, instrument the named model-provider calls, review automated and human evaluation outputs, and decide whether the workflow gives your team actionable quality evidence. This is a tool for technically capable owners, not a substitute for an AI governance program or a turnkey enterprise operating model.
Pricing and Licensing
Auditi’s stated pricing model is Open Source, and the pricing page says it is free and open source. Its GitHub repository uses the MIT license, which is a permissive open-source license. That combination means the software license itself is not presented as a per-seat subscription or a usage-priced commercial service in Auditi's public documentation.
For an AI-agent observability tool, “free and open source” should not be read as “zero cost.” Self-hosting with docker compose up shifts much of the cost model from license fees to the organization operating the service. Teams should account for the engineering time required to deploy, secure, update, monitor, back up, and support the system, as well as the infrastructure that stores traces and serves the application.
Model usage is another material cost category, particularly because Auditi runs 7+ LLM-as-Judge evaluators automatically on traces. Auditi's public documentation does not state which models power those evaluators, how frequently they execute, or their cost. Data leaders should therefore assess the evaluation policy carefully: the volume of traced calls, the number of evaluators enabled, data retention, and the number of human annotations can all influence total operating cost even where the software license is free.
The most important commercial comparison factors in this category are whether pricing is per seat, per traced event, per model call, per retained record, or tied to managed infrastructure; whether data retention or export carries a charge; and whether support is included. Auditi does not provide dollar prices, paid tiers, or published usage limits in Auditi's public documentation, so we do not assign a dollar figure to the tool. Typical commercial pricing ranges for this category are not established by the public documentation and should not be inferred from Auditi’s free license. Check the official Auditi website and repository for current pricing, deployment requirements, licensing terms, and any future commercial offerings before making a purchasing decision.
Strengths & Trade-offs
Auditi has a coherent technical proposition, but the positive case is specific rather than universal. Its benefits come from combining tracing, automated evaluation, annotation, cost tracking, and dataset creation; its risks come from limited public evidence of adoption and an owner-operated deployment model. The following assessment separates documented strengths from decision-relevant limitations.
Pros
- Captures calls across three named model ecosystems: the stated 2-line auto-instrumentation covers OpenAI, Anthropic, and Google API calls, which is useful for teams avoiding provider-specific observability silos.
- Runs 7+ automated evaluators on traces: LLM-as-Judge evaluation is built into the trace workflow rather than described as a separate export-and-process task.
- Keeps human review in the loop: Auditi includes human annotation workflows for outputs where automated evaluation is inadequate, a practical requirement for nuanced quality decisions.
- Connects observability to model improvement: it can turn production traces into fine-tuning datasets, reducing the gap between monitoring an agent and collecting material for iteration.
- Offers cost visibility in the product workflow: real-time cost tracking makes model spending part of operational analysis rather than a separate accounting exercise.
- Uses an MIT license and supports self-hosting: teams can evaluate the codebase and operate it through the stated Docker Compose deployment route without a stated license fee.
Cons
- Public adoption evidence is limited: GitHub data lists 4 stars, which is a weak standalone signal for a buyer seeking broad community validation or a large contributor base.
- The tool requires self-hosting ownership:
docker compose upmakes deployment accessible, but it does not remove responsibility for infrastructure, upgrades, security, resilience, or internal support. - Automated evaluation can add operational cost and review burden: Auditi states that 7+ LLM-as-Judge evaluators run automatically, while providing no published configuration, model, or cost details for those evaluators.
- Enterprise-operating evidence is missing: Auditi's public documentation does not document a managed service, SLA, customer references, benchmark performance, retention limits, or security certifications.
- Its Capterra listing is identity-conflicted: it mixes the AI-agent tool with an audit-evidence platform and features such as audit planning and compliance management, so its reviews cannot be read as reviews of this Auditi product.
Capterra says its reviews are verified and that it has verified more than 2.5 million reviews overall, but that does not resolve the product-description mismatch in the public documentation. We therefore do not treat the included audit-management text as evidence that users of this AI observability Auditi endorse those features. That limitation matters: buyers should run a technical evaluation rather than rely on the external listing as a reliable proxy for AI-agent users’ experience.