350+ Tools CoveredSource Data Updated Weeklydates

Decision comparison

AgentVault vs Auditi

AgentVault and Auditi watch different things. AgentVault watches what an OpenClaw agent does to the machine it runs on: files, shell commands and network calls. It stops risky actions until a person approves them. Auditi watches what an LLM application says and what each call costs, then grades the output with LLM judges and human reviewers. Both are free MIT projects that you host yourself, so price does not decide this choice. Pick by risk. If the danger is an agent stealing keys or running rm -rf, start with AgentVault. If the problem is answer quality and model spend, start with Auditi. A team that runs agents with system access in production will want both, because neither one covers the other's job.

Cross-category comparison
Last Updated:

Architecture choice. These take different approaches to the same problem. Read the table as a fit question rather than a feature race.

These are different kinds of product — AI Security and Agent Observability.

Quick Comparison

AgentVault

Best For:
A developer or small team that runs OpenClaw agents with real access to a laptop or server. AgentVault watches what the agent does on that machine, blocks dangerous commands and sensitive file reads, and makes a person approve each risky action. Its job is to stop damage, not to judge the quality of model answers.
Architecture:
A self-hosted proxy that wraps the agent. The README lists a Node.js, TypeScript and Express proxy server, file system hooks and command parsing for monitoring, a SQLite database for audit logs, and a Next.js 14 and Tailwind CSS dashboard. It works at the action level: it lets a prompt injection reach the agent and stops the file read, command or network call that follows.
Pricing Model:
AgentVault is free and open source under the MIT license: a GitHub project that wraps AI agents with action-level security monitoring for OpenClaw. It has no paid plans and publishes no prices; the agentvault.ai domain now belongs to a different company. Verified 2026-10-05 against https://github.com/hugoventures1-glitch/agentvault.
Ease of Use:
Setup is three commands from the README: npm install -g agentvault, agentvault wrap openclaw, and agentvault dashboard. The ongoing work is the approvals: every risky action waits for a person. Read the code first, because a global npm package will sit between your agent and your credentials.
Scalability:
Built to wrap one OpenClaw agent. Multi-agent monitoring and custom security policies are on the README roadmap as "coming soon", not shipped. The approval step also limits scale on purpose: the agent pauses each time it tries a risky action, so throughput depends on how fast a person answers.
Community/Support:
A single-maintainer GitHub project under MIT. The repository was created on 2026-02-09 and last pushed on 2026-02-10, and the README says "This is day 1. Issues and PRs welcome." There is no company, paid support or SLA. The agentvault.ai and agentvault.chat sites are different products and do not describe this project.

Auditi

Best For:
Teams that build LLM applications and need to see and grade every model call. Auditi captures traces, runs LLM-as-a-judge evaluators on them, adds human annotation queues, tracks cost per call, and turns reviewed traces into datasets for fine-tuning and evaluation.
Architecture:
An open-source observability and evaluation platform. Per its README, a Python SDK sends traces over HTTP to a FastAPI and PostgreSQL backend, and a React and Vite frontend shows the dashboards. It traces agents, tools, LLM calls, embeddings and retrieval steps. It sits in your application code, not between the agent and the operating system.
Pricing Model:
Free and open source
Ease of Use:
Install is clone the repository, generate an ENCRYPTION_KEY and a JWT_SECRET, then run docker-compose up. Instrumentation is two lines of Python, auditi.init() and auditi.instrument(), after which OpenAI, Anthropic and Google Gemini calls are traced automatically. Decorators cover agents, tools and retrieval steps.
Scalability:
Traces are stored in PostgreSQL, so capacity follows the database and host you run. The project publishes no throughput figure. Async support and nested traces cover multi-step agent workflows, and production traces can feed fine-tuning datasets.
Community/Support:
An MIT-licensed GitHub project at deduu/auditi, last pushed on 2026-04-22. Help runs through GitHub Discussions and Issues. Priority support with an SLA is listed only for the Enterprise offering.

Public signals

Verified factual signals only. Bars appear only for like-for-like metrics with five weekly assessments for every tool; missing evidence stays explicit. These signals do not establish enterprise adoption, product quality, or total cost.

MetricAgentVaultAuditi
GitHub commits, 90d(Product adoption)
0
2
GitHub stars(Product adoption)
2
5
Product Hunt comments(Community interest)
1
1
Product Hunt reviews(Community interest)00
Product Hunt votes(Community interest)
2
4

As of November 16, 2026 — updated weekly.

Feature Comparison

Agent Security and Control

Dangerous command blocking

AgentVaultBlocks rm -rf, curl | bash and privilege escalation attempts before they run
AuditiNo blocking feature in its README; it records and grades calls instead

Sensitive file protection

AgentVaultIntercepts agent reads of files such as ~/.aws/credentials and ~/.ssh/id_rsa
AuditiEncrypts the LLM API keys it stores with an ENCRYPTION_KEY; it does not watch agent file access

Human review

AgentVaultHuman-in-the-loop approval before every risky action runs
AuditiAnnotation queues with custom score configs for reviewing outputs after the call

Network visibility

AgentVaultLogs all outbound connections and data exfiltration attempts
AuditiTraces calls to OpenAI, Anthropic, Google Gemini and OpenAI-compatible APIs

Rate limiting and credential scanning

AgentVaultNamed in the maker's project description; the README does not describe them, so test before relying on them
AuditiNo equivalent listed in its README

Observability

Activity records

AgentVaultSQLite database of every agent action, kept for forensics
AuditiTraces of agents, tools, LLM calls, embeddings and retrieval steps in PostgreSQL

Dashboard

AgentVaultWeb UI that shows what the agent is doing right now
AuditiDashboards with metrics, trends, correlations and anomaly detection

Failure analysis

AgentVaultAlerts you when it blocks an action; the audit log supports later investigation
AuditiFailure mode analysis that finds patterns and suggests fixes

Evaluation and Improvement

Automated quality evaluation

AgentVaultNot part of the project; it judges actions by risk, not answers by quality
AuditiLLM-as-a-judge evaluators run on traces (7+ per its maker), plus custom evaluators

Cost tracking

AgentVaultNo cost tracking in the README
AuditiAutomatic cost per call, with provider pricing you can update through its Pricing API

Datasets

AgentVaultNone; audit logs exist for investigation, not training
AuditiBuilds reusable datasets from annotations for fine-tuning and evaluation

Agent and provider coverage

AgentVaultWraps OpenClaw (agentvault wrap openclaw); other agents are not documented
AuditiOpenAI, Anthropic, Google Gemini and OpenAI-compatible APIs, with LangChain and FastAPI examples

Deployment

Hosting

AgentVaultSelf-hosted on the machine the agent runs on, installed with npm
AuditiSelf-hosted with docker-compose

Implementation stack

AgentVaultNode.js, TypeScript, Express, SQLite, Next.js 14 and Tailwind CSS
AuditiPython SDK (3.9+), FastAPI, PostgreSQL, React and Vite

License and Maturity

License and cost

AgentVaultMIT; no paid plans and no published prices
AuditiMIT; Enterprise offering by email with no published price

Project maturity

AgentVaultCreated 2026-02-09, last push 2026-02-10; roadmap items not yet shipped
AuditiLast push 2026-04-22; roadmap lists streaming, webhooks and cloud deployment templates as open

Which approach fits

AgentVault and Auditi watch different things. AgentVault watches what an OpenClaw agent does to the machine it runs on: files, shell commands and network calls. It stops risky actions until a person approves them. Auditi watches what an LLM application says and what each call costs, then grades the output with LLM judges and human reviewers. Both are free MIT projects that you host yourself, so price does not decide this choice. Pick by risk. If the danger is an agent stealing keys or running rm -rf, start with AgentVault. If the problem is answer quality and model spend, start with Auditi. A team that runs agents with system access in production will want both, because neither one covers the other's job.

When each approach fits

Choose AgentVault if:

Choose AgentVault if you run OpenClaw with real access to a laptop or server and your first fear is damage, not bad answers. It catches reads of ~/.aws/credentials and ~/.ssh/id_rsa, blocks rm -rf and curl | bash, logs outbound connections, and makes every risky action wait for your approval. It has no license fee and installs with three commands. Accept its limits before you depend on it: it is an early single-maintainer project (created 2026-02-09, last push 2026-02-10), it documents only OpenClaw, and custom policies and multi-agent monitoring are still roadmap items. Auditi loses this case because it cannot stop an action; it records and grades calls after they happen.

Choose Auditi if:

Choose Auditi if you ship an LLM feature and need to know whether the answers are good and what they cost. Two lines of Python trace OpenAI, Anthropic and Google Gemini calls. LLM-as-a-judge evaluators score the traces, annotation queues let people review hard cases, and reviewed traces become datasets for fine-tuning. It runs on your own PostgreSQL with docker-compose under MIT, and its README offers SSO/SAML and RBAC through an Enterprise offering by email. AgentVault loses this case because it has no evaluation, cost tracking or dataset features: it judges actions by risk, not output by quality. Auditi is also the more recently updated project, with its last push on 2026-04-22.

These scenarios reflect the available product evidence. Your requirements, existing stack, and team expertise should guide the final decision.

Frequently Asked Questions

What is the main difference between AgentVault and Auditi?

AgentVault is a runtime guard for OpenClaw agents. It sits between the agent and the machine, intercepts sensitive file reads, blocks dangerous commands, logs network calls, and asks a person to approve risky actions. Auditi is an observability and evaluation platform for LLM applications. It traces model calls, scores them with LLM-as-a-judge evaluators and human annotators, tracks cost, and builds datasets from the results. One controls what an agent does; the other measures what a model says. They solve different problems, so many teams can run both.

Which is better for small teams?

The answer turns on the risk you face, and it is clear for each case. A small team that lets an OpenClaw agent touch real credentials or a shell should start with AgentVault, because it is the only one of the two that can stop an action. Plan for one person to answer approval prompts and to read the code before installing it. A small team building a customer-facing LLM feature should start with Auditi, because two-line instrumentation, automatic evaluation and cost tracking match that work. Both have no license fee, so the real cost is setup time and the host.

Can I migrate from AgentVault to Auditi?

Not in the usual sense. The two projects store different data for different purposes, and neither documents an import from the other. AgentVault keeps a SQLite log of agent actions such as file reads, commands and approvals. Auditi keeps traces of LLM calls, tool calls and evaluation scores in PostgreSQL. A move from one to the other is really adding a second tool: you instrument your code with the Auditi SDK and keep, or retire, the AgentVault proxy based on whether you still need actions blocked at runtime.

What are the pricing differences?

There is no price difference in license terms. AgentVault is free and open source under MIT, with no paid plans and no published prices. Auditi is also free and open source under MIT, self-hosted with Docker Compose. Its README names an Enterprise offering with SSO/SAML, advanced RBAC and priority support that you request by email, and it publishes no price for it. For both tools, your costs are the server you run them on and the people who maintain them. With Auditi, LLM-as-a-judge evaluators also make model calls, and those calls are billed by your model provider.

Can AgentVault and Auditi run together?

Yes, and for an agent with system access in production that is a sensible setup. AgentVault wraps the OpenClaw agent and blocks risky file reads, commands and network calls at runtime. Auditi instruments the application code and records each LLM call for evaluation and cost tracking. They work at different layers, so they do not overlap. Keep their records apart: AgentVault writes to its own SQLite file and Auditi to its own PostgreSQL database, and neither project documents a link between the two.