🔍 Read the full analysis: FrontierCode: AI Pull Requests And The Maintainer’s Decision on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis published FrontierCode results for 23 AI models on October 12, 2026. Claude Opus 5.5 had the highest score, but cleared all blocking checks in 57.7% of attempts; the results measure performance on a 100-task set, not how often AI-generated changes are accepted in every real repository.
Artificial Analysis published results on October 12, 2026, from 23 AI models tested on FrontierCode, a coding benchmark that scores whether maintainers would accept a change rather than relying on passing tests alone. Claude Opus 5.5 ranked first with a 52.2% score, but cleared every blocking check in 57.7% of attempts, leaving substantial room for human review.
FrontierCode’s score is not the same as its pass rate. The leading model’s 52.2% benchmark score reflects both passing required checks and quality scoring; its 57.7% pass rate records attempts that cleared all blocking checks. Artificial Analysis tested each model using the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. The review stage used GPT-6 Luna (medium) as its judge.
GPT-6.1 Sol placed second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Nine models scored between 43% and 52%. The published results cover the 100-task Main set, drawn from a full benchmark of 150 tasks. According to the source report, nine of the 100 tasks were not solved by any model on any attempt.
Results also varied by cost. Artificial Analysis listed GPT-6 Luna (max) at 41.9% for $0.09 per task, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Claude Opus 5.5 at 52.2% for $10.15. These are reported per-task costs for this evaluation; they do not, on their own, establish what a model would cost in other workloads.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results highlight the gap between a patch that runs successfully and one that fits a repository’s standards. FrontierCode checks for correctness, regressions, test quality and scope, then includes an LLM review against criteria set by maintainers. A change can fail a blocking check and score zero even if other checks pass. That makes the benchmark relevant to teams weighing AI coding tools: it tests more of the work that can make an automatically generated pull request difficult to review.
At the same time, the results do not establish that the leading model would produce an acceptable change 57.7% of the time across all software projects. That figure is the pass rate in this specific evaluation, under its task set, harness and scoring rules. The cost comparison also shows a marked price difference near the top: Opus scored 3.1 percentage points above GPT-6.1 Sol while costing $10.15 versus $0.81 per task in this run. Teams would need to weigh that reported performance against their own workloads and review requirements.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Changes
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source report says maintainers spent more than 40 hours on each task. The tasks are private, intended to prevent models from having encountered their solutions during training.
The benchmark uses six evaluation areas: hidden tests, command checks, reverse tests that check whether an agent’s tests fail on the original code, adaptive tests for complex behaviour, limits on files and diff size, and an LLM code review. Checks can be blocking or non-blocking; an attempt that fails a blocking check scores zero. Cognition’s trajectory scanner can also assign zero if it flags an attempt for looking up a solution. Artificial Analysis evaluated the 100-task Main set rather than all 150 tasks in the full benchmark.
In Cognition’s announcement, Celery maintainer Tomer Nosrati said: “Where others grade like a CI, FrontierCode grades like a tech lead.” That description captures the benchmark’s stated distinction: it aims to assess maintainability and fit, not only whether code passes tests.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The leaderboard reports performance on a particular benchmark, not acceptance rates across real-world pull requests. The source material does not provide enough detail to establish how closely the 100-task Main set represents repositories, languages or maintenance practices outside the included tasks. It also does not show whether the reported model rankings would hold under different agent settings or review processes.
Several models had no published FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the table does not compare every prominent coding model. The source report also does not specify a date for later evaluations or say when additional results will be added.
As an affiliate, we earn on qualifying purchases.
Further Model Results to Watch
The next useful development will be results for models not yet represented, alongside any repeat evaluations or updates to the benchmark. The source report specifically identifies Muse Wonder Full as a model whose future FrontierCode score will be watched, but provides no release date for that result. Until more models are tested and the benchmark’s coverage grows, the October leaderboard should be read as a snapshot of the 23-model group evaluated by Artificial Analysis.
For maintainers and engineering teams, the reported figures offer a comparison point rather than a substitute for review. FrontierCode’s central finding is bounded but clear: even the leading model in this run did not pass every blocking check on many attempts, so the benchmark does not support treating generated pull requests as ready to merge without assessment.
AI coding assistant for developers
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates code changes using tests, repository-scope rules and an LLM review against maintainer criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the Artificial Analysis results with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts.
Does a 57.7% pass rate mean that model gets 57.7% of real pull requests accepted?
No. It is the pass rate reported for attempts in FrontierCode’s 100-task Main set under the evaluation setup. The result does not establish an acceptance rate across real-world repositories.
How many models were included?
Artificial Analysis tested 23 models. Several other named models had no FrontierCode result as of October 13, 2026, according to the source report.
What is still unknown?
The published results do not show how well the scores generalize to all repositories or whether rankings will hold under other agent settings. Results for several models were also unavailable at the time of publication.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
