Coding Core v1.0
Twenty self-contained repository tasks: bug fixes, small features, refactors and test writing across Python, TypeScript and Go.
Published 18 Oct 2026
Evaluation criteria
- A task passes only if all hidden unit and integration tests pass after the agent finishes.
- Pre-existing tests must still pass (no regressions).
- The agent may not modify the hidden test files or the grader.
- Execution failures (timeout, crash, tool error) count as failed attempts and are reported separately.
Test conditions
- Fresh container per attempt from a pinned image; repository checked out at a fixed commit.
- No outbound network access except the model provider endpoint.
- Permitted tools: shell, file edit, test runner. No web browsing.
- Per-task wall-clock timeout and step limit as listed in the run configuration.
- Each task attempted 3 times to measure repeatability.
Scoring methodology
Task success score = passed attempts ÷ total attempts × 100. Cost, latency, failure rate and repeatability are reported separately and are not blended into the score.
Full methodologyResults
Latest run per agent. Independent runs are listed first; publisher-reported runs are shown separately and not ranked against them.
No agents have been evaluated on this suite yet.
Task suite (20 tasks)
Task titles are public; repositories and hidden tests are kept private to reduce training-data contamination.
| ID | Task | Language | Difficulty | Use cases | Grader |
|---|---|---|---|---|---|
| cc-01 | Fix off-by-one error in pagination helper | Python | easy | Bug fixing | Hidden test suite |
| cc-02 | Handle empty input in CSV parser | Python | easy | Bug fixing, APIs & data | Hidden test suite |
| cc-03 | Correct timezone handling in date formatter | TypeScript | medium | Bug fixing | Hidden test suite |
| cc-04 | Add input validation to REST endpoint | TypeScript | easy | Security fixes, APIs & data | Hidden test suite |
| cc-05 | Fix race condition in job queue worker | Go | hard | Bug fixing, Reliability & performance | Hidden test suite |
| cc-06 | Implement retry with exponential backoff | TypeScript | medium | Building features, Reliability & performance | Hidden test suite |
| cc-07 | Add pagination to list API | Go | medium | Building features, APIs & data | Hidden test suite |
| cc-08 | Write unit tests for currency rounding | Python | easy | Testing | Hidden test suite |
| cc-09 | Refactor callback code to async/await | TypeScript | medium | Refactoring | Hidden test suite |
| cc-10 | Migrate config loader to typed schema | TypeScript | medium | Refactoring, APIs & data | Hidden test suite |
| cc-11 | Fix memory leak in event subscription | TypeScript | hard | Bug fixing, Reliability & performance | Hidden test suite |
| cc-12 | Add caching layer to slow lookup function | Python | medium | Building features, Reliability & performance | Hidden test suite |
| cc-13 | Resolve failing integration test after dependency bump | Python | hard | Testing, Bug fixing | Hidden test suite |
| cc-14 | Implement CLI flag for dry-run mode | Go | easy | Building features | Hidden test suite |
| cc-15 | Split large module into cohesive packages | Go | hard | Refactoring | Hidden test suite |
| cc-16 | Fix SQL injection in search query | Python | medium | Security fixes, APIs & data | Hidden test suite |
| cc-17 | Add structured logging to request handler | Go | easy | Building features | Hidden test suite |
| cc-18 | Implement rate limiter middleware | TypeScript | hard | Building features, Security fixes | Hidden test suite |
| cc-20 | Add database migration for new column with backfill | Python | hard | APIs & data, Building features | Hidden test suite |
| cc-19 | Fix incorrect sort stability in table component | TypeScript | medium | Bug fixing, Testing | Hidden test suite |