🔍 Read the full analysis: FrontierCode: What It Takes For An AI Pull Request To Be Considered on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, and tested 23 models on repository tasks judged against maintainer criteria. Claude Opus 5.5 led with a 52.2% score, while passing every blocking check in 57.7% of attempts. The results show that strong benchmark performance still leaves substantial room for human review.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, publishing results for 23 AI models on repository tasks designed to test whether maintainers would accept their code changes. Claude Opus 5.5 led with a 52.2% score, but it passed every blocking check in 57.7% of attempts, leaving a substantial share of attempts short of the benchmark’s acceptance threshold.
FrontierCode evaluates patches across six areas: hidden tests, command checks for regressions and cleanliness, whether an agent’s tests fail on the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM review against maintainers’ criteria. Each check is marked blocking or non-blocking. Failing any blocking check scores zero; attempts that pass all blocking checks receive a score between 70% and 100%, depending on quality checks. Cognition’s trajectory scanner also assigns zero to attempts flagged for looking up solutions.
The benchmark was built by Cognition with open-source maintainers. More than 20 maintainers wrote tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task, according to the source material. The full benchmark contains 150 private tasks; Artificial Analysis runs the 100-task Main set. Its setup uses the open-source Stirrup agent harness, a cap of 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) served as the review judge.
Opus 5.5’s 52.2% score put it ahead of GPT-6.1 Sol at 49.1% and Claude Fable 5.1 at 49.0%. GPT-6 Astra scored 47.0%, followed by Claude Sonnet 5.5 at 45.8%. Xiaomi’s MiMo-V2.6-Pro scored 45.4%, while DeepSeek V4.1 Flash and Kimi K3 scored 44.0% and 43.6%. Nine models scored between 43% and 52%; results then fell to 33.8% for Grok 4.7 and lower for several other tested systems.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
What the Acceptance Scores Show
The results measure a stricter outcome than whether code passes visible tests: they ask whether a patch meets a set of correctness, scope and review requirements. Opus 5.5 passed all blocking checks in 57.7% of attempts, according to the reported results. That is the benchmark’s pass rate, not a guarantee that a model will succeed on a particular team’s codebase or workflow.
The figures suggest that AI coding agents can produce acceptable changes on a meaningful share of these tasks, while still failing blocking checks in many attempts. For maintainers, those failures make review and validation part of the current process. The leaderboard does not establish that autonomous merging is reliable; it reports performance under this benchmark’s tasks, harness and scoring rules.
Cost also varies sharply. Artificial Analysis reported per-task costs from $0.09 for GPT-6 Luna to $12 for Claude Fable 5.1. It identified four models on its cost frontier: GPT-6 Luna at 41.9% for $0.09, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Opus 5.5 at 52.2% for $10.15. In this comparison, moving from Sol to Opus adds 3.1 score points at 12.5 times the reported cost.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Judges Patches
FrontierCode was announced in June 2026, with version 1.1 following in July. The source says that version added rules against agents looking up solutions online. Tasks are private, which is intended to keep benchmark solutions out of model training data. Artificial Analysis evaluated the 100-task Main set, rather than the full 150-task benchmark.
The scoring design accounts for issues that can make an otherwise passing patch difficult to merge. A test must distinguish the changed code from the original; the benchmark also checks whether the agent edits permitted files and keeps the diff within scope. Non-blocking quality checks affect the score after blocking requirements are met. Passing tests alone cannot secure a passing attempt under these rules.
The source compares results with the Artificial Analysis Intelligence Index and reports different rankings across the two evaluations. Claude Sonnet 5.5 ranked second among the listed models on that index but fifth on FrontierCode; Grok 4.7 fell to 18th on FrontierCode. DeepSeek V4.1 Flash and GPT-6 Luna ranked better on FrontierCode than their listed Intelligence Index positions might suggest. The comparison indicates that general model rankings do not directly predict performance on this repository benchmark.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
Limits of the Published Results
The published scores describe performance on FrontierCode’s selected tasks and evaluation setup. They do not show how the models would perform across every repository, programming language, team policy or production workload. Nine of the 100 Main-set tasks were not solved by any model on any attempt, according to the source material, but it does not identify those tasks or explain what made them difficult.
Several models had no FrontierCode result as of October 13: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the reported ranking does not cover every model readers may want to compare. The source also does not provide task-by-task results or enough detail here to assess variation across the three attempts for each task.
The benchmark uses an LLM reviewer, GPT-6 Luna (medium), as part of its grading. The available material describes that setup but does not report how judge agreement was measured or how sensitive scores are to the reviewer’s judgments. That leaves an open question about grading consistency, particularly for quality checks rather than blocking tests.
More Model Results to Watch
The next comparison point will be results for models that had no score as of October 13, particularly Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. The source specifically says Muse Wonder Full’s result will be worth watching given its performance on Terminal-Bench and the Coding Agents Index. No release date for those FrontierCode results is provided.
Readers can also watch for further benchmark results or methodological details from Artificial Analysis and Cognition, including task-level performance and information about review consistency. The reported evaluation establishes a current comparison under the published setup; how well those scores carry over to other repositories remains unknown.
Key Questions
What does FrontierCode measure?
It measures whether an AI coding agent produces a repository patch that meets blocking checks and maintainer-oriented quality criteria, including correctness, scope and review.
Which model ranked highest in the reported evaluation?
Claude Opus 5.5 (max with fallback) ranked first with a 52.2% FrontierCode score. It passed every blocking check in 57.7% of attempts.
Does a high score mean an AI patch can be merged without review?
The results do not establish that. They show performance on FrontierCode’s tasks and scoring setup; the benchmark’s leading model still failed to pass all blocking checks in a substantial share of attempts.
How much did model costs vary?
Artificial Analysis reported per-task costs ranging from $0.09 for GPT-6 Luna to $12 for Claude Fable 5.1. Costs and scores are specific to the evaluation setup described in the source.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
