350+ Tools CoveredSource Data Updated Weeklydates

Tolokaforge

Tolokaforge is a benchmarking harness for tool-using LLM agents, with multi-turn agent/user loops, sandboxed execution, deterministic grading, and telemetry through LiteLLM-supported providers.

Visit Tolokaforge →
Type
LLM Observability & Evaluation

By Egor Burlakov · Edited · Source dates below

Buying snapshot

Scroll sideways to see all columns. With a keyboard, focus this table and use the arrow keys.

AspectSupported detail

License

Apache-2.0.

Installation

Available as a Python package; optional extras add the terminal CLI, Playwright support, or all components.

Runtime isolation

Docker-backed shared and per-trial runtime modes are documented; per-trial mode uses a fresh stack through Testcontainers.

Model connectivity

The harness supports providers available through LiteLLM, including OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and OpenRouter.

Take

Consider Tolokaforge when an evaluation needs reproducible task environments alongside grading and run telemetry. It is especially suitable when task state can change between trials: the documented per-trial backend is intended to prevent database, filesystem, and service-state leakage. Teams seeking only a lightweight prompt-test workflow may find the Docker-stack operating model more infrastructure than they need.

Evaluate Tolokaforge

Comparisons

Where it stands out

  • It combines JSONPath assertions, state hashes, transcript rules, and optional LLM judges for deterministic grading workflows.

  • Coding-harness mode can run seven listed vendor coding-agent CLIs inside the trial container while retaining the same per-trial artifact bundle shape.

  • Task packs can declare their own Docker Compose stack, enabling task-specific services beyond the built-in environment.

Trade-offs to weigh

  • Every trial runs in a Docker stack, so evaluation setup depends on Docker-compatible runtime infrastructure.

  • Per-trial isolation is designed for mutable environments, but it creates a fresh stack for each trial; shared mode is documented as the lower-overhead option.

  • A run configuration and its referenced tasks travel together; the project documents no global default configuration.

Public signals

About these signals

Verified factual signals from public sources. They indicate observable activity or interest, not total adoption, product quality, or cost.

226 GitHub commits 90d24 GitHub stars0 vulnerabilities across 1 package

See all signals from 3 sources
Source
Signals
Last updated
GitHub
Commits 90d:226↓18Stars:24↑1
November 16, 2026
PyPI
Weekly downloads:1.7k↑907
November 16, 2026
OSV
Package vulnerabilities:0 vulnerabilitiesacross 1 package

PyPI · tolokaforge@0.34.8

November 16, 2026

Alternatives to Tolokaforge

The reviewed substitutes for Tolokaforge among the LLM observability & evaluation, and what would make each one the better answer.

Other approaches

A different approach to the same problem. Each substitutes only for the workload named beside it.

Arize Phoenix
Arize Phoenix is a local-first platform for tracing and evaluation built on OpenTelemetry, where datasets and experiments sit beside observability of a live application. Tolokaforge is a dedicated benchmarking harness that generates multi-turn agent episodes, runs the tools in a sandbox and records its own telemetry for grading. The choice depends on whether the buyer needs an observability platform with experiments added on, or a self-contained harness for comparing agent behaviour across LLM providers.Applies to: Running experiments that compare agent models
Braintrust
Tolokaforge is an open-source harness that runs tool-using agents through multi-turn conversations with a simulated user, executes their tool calls in a sandbox and grades the results deterministically. Braintrust is a hosted platform for scored experiments over versioned datasets, with LLM-judge, code and human scoring and gates in CI. The choice depends on whether the team needs reproducible agent benchmarks with simulated users and sandboxed tools, or a managed eval workflow that compares prompts and models and blocks releases.Applies to: Scoring agent and model variants before release
TruLens
TruLens is an open-source library that instruments LLM apps and agents and scores their runs with feedback functions, which are mostly LLM-based judges of groundedness, relevance and similar qualities. Tolokaforge builds the test episodes itself, with user and agent loops in a sandbox, and grades the outcomes deterministically. The deciding factor is whether the buyer wants to score traces from their own application or run controlled, repeatable tool-use benchmarks across providers.Applies to: Open-source evaluation of tool-using agents
Explore all Tolokaforge alternatives →

Sources

This review is based on the project's official GitHub repository capture, observed 2026-10-19.