🔍 Read the full analysis: How CWE-bench V2 Advances Coverage, Quality, And Transparency In AI on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Collinear AI has released CWE-bench v2, a benchmark of 100 security-fix tasks from real open-source repositories, along with its grader and evaluation harness. In the release’s results for 11 models, Gemini 4 Argon led at 67.8% pass@1, but the authors report that even the leader failed some tasks on every attempt.
Collinear AI has released CWE-bench v2, a public benchmark of 100 security-fix tasks drawn from real open-source repositories, alongside the tasks’ grader and evaluation harness. The release gives researchers a shared way to test coding agents across 91 CWE weakness categories; its reported results show the leading model reaching 67.8% pass@1, while still failing a portion of the tasks.
Each task gives an agent a checkout of a real repository and a short instruction to find and fix a security weakness without breaking existing functionality. The benchmark authors say scoring is fully programmatic and based on curated checks, with tasks packaged as self-contained Harbor environments. A registry file on Hugging Face is provided to help users run the tasks.
The release reports evaluations of 11 models. Gemini 4 Argon recorded the highest listed pass@1 score, at 67.8%, and a pass@4 score of 81.0%. Claude Opus 5.5 followed at 67.3% pass@1 and 78.0% pass@4. The authors say at least one evaluated model solved 95 of the 100 tasks; that figure does not mean a single model solved all 95.
The announcement also compares reported cost and performance. DeepSeek V4.1 Flash (1017) scored 57.0% pass@1 at a reported $0.04 per rollout. Claude Opus 5.5 scored about 10 percentage points higher at $1.83, while Gemini 4 Argon scored slightly higher still at $3.92. These are figures from the release; costs may depend on model pricing and evaluation setup.
What the Scores Say About Security Fixes
CWE-bench v2 addresses a practical measurement problem: a coding agent that can produce plausible code is not necessarily able to find and repair security flaws without disrupting software. Testing against real repository checkouts and requiring fixes to pass behavior checks makes the benchmark a way to compare more than code generation in isolation.
The results also put a limit on what strong benchmark performance can show. The top published pass@1 score is below 70%, and the release says the leading model fails 19% of the set on every attempt. That is evidence of remaining gaps on these tasks, not proof of how a model would perform across all security work. A score can support comparisons under a shared test, but it does not establish that an agent can safely handle every repository, vulnerability or deployment environment.
Public tasks and tooling give outside teams a chance to inspect the benchmark and repeat at least the public evaluations. That can make comparisons more transparent than results that cannot be independently run. At the same time, the benchmark remains curated: performance may reflect how well systems handle its task design, and does not by itself predict outcomes in live security reviews.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W
DJI Osmo Mobile 8 Gimbal with Mic Mini Transmitter, Infinity Black
As an affiliate, we earn on qualifying purchases.
How Version 2 Was Reviewed
Version 2 retains the earlier task format and expands the publicly available set to 100 tasks. The authors say they reviewed each task by hand, checking whether its instruction was realistic, whether the expected behavior could be inferred, and whether the verifier measured correct behavior rather than requiring a particular code structure.
According to the release, the team also checked that starter code fails the relevant tests, the reference fix passes, existing functionality remains intact, and verifiers reject memorized answers such as known upstream patches. LLM judge panels helped shape programmatic checks for task difficulty and fairness. These are the authors’ descriptions of their quality process, not an independent audit of the benchmark.
The release describes two Hugging Face leaderboard channels: a dataset-page leaderboard for open-weight models and a leaderboard Space for broader comparisons, including API-only models. Contributors can submit results through pull requests. The dataset is gated, meaning users must accept its access conditions before running the public tasks.
“A defender has to anticipate the ways an attacker could get in, and get it right every single time.”
— Collinear AI, in its CWE-bench v2 announcement on Hugging Face
Limits of the Published Comparisons
The supplied results do not include enough methodological detail to independently assess every comparison, including exact model configurations and the conditions behind each cost estimate. Scores may change as new results are submitted, and the announcement does not establish how reliably performance on these 100 tasks predicts results in production security work.
Collinear AI says it maintains a separate held-out set for tracking progress on tasks not seen by evaluated models. That set is not public; access requires contacting the company, so outside researchers cannot directly inspect or reproduce results on it. The release also does not establish whether independent teams have reproduced the reported public scores.
Independent Runs and New Submissions
Researchers and developers can run models against the public tasks using Harbor after accepting the dataset conditions and setting up Hugging Face authentication. The release invites feedback on the tasks and says leaderboard results can be submitted through pull requests. Collinear AI says teams interested in testing models on the held-out set can contact it.
Further evaluations may add to or change the public comparisons, while task feedback could reveal weaknesses in the benchmark itself. Independent reproduction of the scores and evidence that benchmark results track performance in real security reviews would help clarify how far the results generalize; the announcement does not set a date for either.
Key Questions
What is CWE-bench v2?
It is a 100-task benchmark from Collinear AI for evaluating coding agents on security fixes in real open-source repositories. The tasks span 91 CWE weakness categories.
Which model led the reported results?
Gemini 4 Argon led the release’s evaluation of 11 models with 67.8% pass@1 and 81.0% pass@4. These are results reported by the benchmark authors.
Does a 67.8% score mean the model can handle security work reliably?
No. It measures performance on this benchmark’s tasks under the reported evaluation. It does not establish performance across all repositories or real-world security conditions, and the release says the leading model failed 19% of the set on every attempt.
Can researchers run the benchmark themselves?
The public tasks, grader and evaluation harness are available through the release. Users must accept the dataset’s access conditions and set up Hugging Face authentication; the tasks are packaged for use with Harbor.
Can outside researchers check results on the held-out set?
Not directly from the public dataset. Collinear AI says the held-out tasks are private and that interested teams can contact the company to evaluate models on them.
Primary source: Hugging Face · via ThorstenMeyerAI.com
