Methodology · Rate AI Agent

Current status

The Coding Core v1 suite and this method are published, but the evaluation runner is not live yet. No real agent has been evaluated. Every result on the site belongs to a fictional example agent and was produced by a simulator, to show how real results will appear. Real agents are listed as Not evaluated.

Provenance labels

Every figure carries one of four labels. They are never merged, and publisher or community figures never feed a ranking.

Independently evaluatedMeasured by this platform's evaluation harness under documented, reproducible conditions.: Measured by this platform's evaluation harness under documented, reproducible conditions.
Publisher-reportedStated by the agent's developer. Not checked by this platform.: Stated by the agent's developer. Not checked by this platform.
Community-reportedReported by a third party or user. Not checked by this platform.: Reported by a third party or user. Not checked by this platform.
Not evaluatedNo independent result on a full benchmark suite yet.: No independent result on a full benchmark suite yet.

Test selection

Tasks reflect common real work: bug fixes, small features, refactors, test writing and dependency issues, across several languages.
Each task runs in a self-contained repository with a known-good reference solution and hidden tests written before any agent is run.
Task titles are public. Repositories and hidden tests stay private to limit contamination of model training data. Tasks are rotated in new suite versions.
Difficulty labels (easy, medium, hard) are set by how long the reference solution took a human to write, not by agent results.

Reproducibility

Each attempt runs in a fresh, disposable sandbox built from a pinned image digest.
Each run records the suite version, agent version, model and model version, timeouts, step limit, permitted tools, network policy, hardware class and runner version.
Each task is attempted three times. We report repeatability, the share of tasks with the same outcome on every attempt, because agent behaviour is not deterministic.
Transcripts, tool calls, diffs and grader output are kept for each attempt, so a disputed result can be audited.

Scoring

The headline figure is the task success score: passed attempts ÷ total attempts × 100. An attempt passes only if every hidden test passes and no existing test regresses. We also report:

Tasks attempted and tasks passed (a task counts as passed when most of its attempts pass)
Cost per task and cost per successful task
Median time per attempt
Failure rate: attempts ending in a timeout, crash or tool error
Repeatability across attempts

These are shown separately. We do not fold cost or speed into the success score.

Levels and use cases

A single score can't tell you whether an agent suits your work. Many agents are a good choice for some jobs and not others, so alongside the score we report plain-language levels by kind of work.

Every task is tagged with one or two use cases: Bug fixing, Building features, Refactoring, Testing, Security fixes, Reliability & performance, APIs & data. Tags are set when a task is written, not from agent results.
For each use case and each difficulty, the level comes from that group's attempt success rate: Strong at 80% or more, Capable from 50%, Inconsistent from 20%, otherwise Not yet reliable.
A group needs at least 3 different tasks to get a level. Smaller groups show their numbers marked “Too few tasks”, because a level based on one or two tasks would mislead.
Levels come from independent runs only, the same rule as ranking. The “Good at” filter matches agents that are Strong or Capable for every selected use case.
Partial progress is the average share of hidden checks passed on attempts that didn't fully pass. It shows how close a failure came. It never counts towards the success score or the levels: an attempt passes only when every check passes.

StrongUsually completes this kind of work on its own.
CapableCompletes most of this work; review what it produces.
InconsistentHit and miss; expect to step in.
Not yet reliableRarely completes this kind of work yet.
“Too few tasks”: fewer than 3 tasks in the group, so no level is given. Use cases: Bug fixing, Building features, Refactoring, Testing, Security fixes, Reliability & performance, APIs & data.

Composite score (disabled by default)

A single “overall” rating hides value judgements. A team with a large budget weighs cost very differently from a solo developer. For that reason the composite score is turned off. If it is enabled, the formula is a weighted mean of normalised components, shown next to its parts:

composite = Σ(weightᵢ × componentᵢ) ÷ Σ(weightᵢ for available components)
success   = success rate                           weight 0.6
cost      = 1 − min(1, cost per success ÷ $5)        weight 0.15
latency   = 1 − min(1, median time ÷ 20 min)        weight 0.1
stability = repeatability                          weight 0.15

Missing components are left out and the remaining weights re-normalised. If success is unknown, there is no composite.

Cost measurement

Cost is metered model usage (input and output tokens at the provider's list price on the run date), plus paid tool or API calls made by the agent.
Sandbox compute is excluded, because it doesn't vary by agent.
Subscription-priced agents are metered on the underlying usage where possible. Otherwise cost is reported as n/a, not estimated.
Cost per successful task is shown only when at least one attempt passed. If any attempt is missing cost data, no total is shown, because a partial total would understate it.

Latency measurement

Wall-clock time from task hand-off to the agent declaring completion, or to the timeout.
We report the median rather than the mean, so a few very long attempts don't distort it.
Provider rate limits and queueing are included, because users experience them too. Runs are scheduled at varied times to reduce bias.

Failure handling

Failures are classified as wrong output (ran fine, tests failed), timeout, crash or tool error.
Only timeouts, crashes and tool errors count towards the failure rate. All failures count against success.
Platform-side infrastructure faults are re-run and logged. They are never counted against the agent.
An agent that cannot be installed or started under the published conditions is reported as not evaluable, with the reason, rather than scored zero.

Model & version tracking

Each result is tied to one agent version, model version, suite version and configuration. When any of these changes, the result becomes history rather than being overwritten. Older results are labelled by date, and profiles show the full history.

When results are comparable

Two results are compared directly only if they used the same suite version, timeout, step limit, network policy, permitted tools, hardware class and number of repeats, and the same provenance. The model may differ, because the agent + model pairing is what users choose. When conditions differ, the comparison page greys out the values, explains why and leaves them out of its conclusions.

Limitations & known biases

Twenty tasks is a small sample. Differences of a few points are within noise, and the comparison page says so.
Self-contained repositories understate the difficulty of large, messy real codebases.
Hidden tests check behaviour, not code quality, readability or security.
Agents tuned to public benchmarks may do better here than in real use. Private tasks and rotation reduce, but don't remove, this risk.
Cost depends on provider pricing on the run date and goes out of date.
Interactive agents designed around human feedback are run without it, which may disadvantage them.

Conflict-of-interest policy

Payment never affects scores, rankings, badges or which results are published.
Publishers may pay for an evaluation run. Paid runs use the same suite, conditions and publication rules, and the profile shows who requested and funded each run.
Results are published whether favourable or not. A publisher cannot withdraw a completed run.
Reviewers cannot approve, verify or evaluate agents from organisations they belong to. This is enforced in the database, not just by policy.
No sponsored placements inside rankings and no paid “verified” badges. Any affiliate relationship will be disclosed on the profile.
Staff and contractors declare relevant financial interests. Declared interests will be published here.

Found a problem with a result? Use the submission form to report a correction.