Benchmarks
Benchmark suites
Each suite is a versioned set of tasks with published criteria, test conditions and scoring. Results always link back to the exact suite version, agent version and configuration that produced them.
What every benchmark contains
All categories follow this structure, so results can be compared and audited the same way.
- Task suite
- A fixed, versioned set of tasks. Changing a task creates a new suite version.
- Evaluation criteria
- Exactly what counts as a pass, decided before any agent is run.
- Test conditions
- Sandbox image, network policy, permitted tools, timeouts and step limits.
- Scoring methodology
- How attempts become metrics. Success, cost, latency and failures stay separate.
- Evaluation date
- When the run happened. Agents and models change, so older results are labelled by date.
- Agent & model versions
- The exact agent release and model version used in the run.
- Execution configuration
- Every parameter of the run, recorded so it can be reproduced.
- Result provenance
- Who ran it: this platform, the publisher or the community.
Result provenance labels
- Independently evaluatedMeasured by this platform's evaluation harness under documented, reproducible conditions.
- Measured by this platform's evaluation harness under documented, reproducible conditions.
- Publisher-reportedStated by the agent's developer. Not checked by this platform.
- Stated by the agent's developer. Not checked by this platform.
- Community-reportedReported by a third party or user. Not checked by this platform.
- Reported by a third party or user. Not checked by this platform.
- Not evaluatedNo independent result on a full benchmark suite yet.
- No independent result on a full benchmark suite yet.