What Claude Can Show Through Formal Verification—and What Remains Unclear

🔍 Read the full analysis: What Claude Can Show Through Formal Verification—and What Remains Unclear on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Anthropic’s Frontier Red Team and collaborator Theorem say Claude proved a no-crash property for 239 fuzz-test entry points across 123 open-source libraries, with proofs checked by the Rocq proof assistant. The work also uncovered bugs that earlier fuzzing had not found, but it does not establish that the libraries are bug-free: the results cover specified entry points in compiled RISC-V programs under stated conditions.

The researchers used existing fuzz tests as the properties to prove. They compiled each library and test into an instrumented RISC-V executable, then had Claude produce proofs in Rocq that the program would not crash for any input under 64 megabytes. The stated property excludes AddressSanitizer memory errors and failed assertions, subject to the models and assumptions used in the proof.

The method’s scale is a central finding. Anthropic reports that the proofs contain about 180 million lines of Rocq, produced over a few weeks. A typical proof was about 600,000 lines, for a binary with a median size of 18,000 RISC-V instructions. Rocq’s checker, rather than a human reading each proof, checks whether a proof follows from its specification.

Anthropic says the process found hundreds of bugs, including issues associated with large or unusual inputs, particular sequences of calls, and deeply nested data. The source also describes a logic error involving a file-loading restriction, a property that does not necessarily cause a crash. The account says even incomplete proof attempts could reveal bugs, but does not give a full breakdown of findings by severity or disposition.

What Machine-Checked Proofs Could Change

Fuzzing tests many inputs but cannot, by itself, show that every possible input has been covered. A proof of the stated property can address all inputs in the modelled range and execution path, making this a different kind of assurance than a record of successful tests. That matters for software components where memory errors can create security risks.

The report also speaks to the cost that has kept formal verification uncommon. Anthropic cites the seL4 microkernel project as an example of the human effort historically involved: two person-years to build and 20 person-years to verify. The researchers’ result suggests AI may help generate proof material at a much larger scale, while a small proof checker handles verification. That is a reported demonstration, not evidence yet that the approach is economical or practical across ordinary software projects.

The distinction is important for readers assessing security claims: a machine-checked proof can provide strong assurance about a precisely defined property, but the property and the model determine what the result actually covers. This work offers evidence about particular no-crash tests—not a general guarantee of security.

From Existing Fuzz Tests to Rocq

Formal verification requires a specification: a precise statement of what a program must do. Rather than define broad correctness for each library, Anthropic and Theorem used its existing fuzz tests. In a typical fuzzing setup, tools generate or mutate inputs and look for crashes; the approach can find bugs, but a test run cannot establish that every relevant path has been exercised.

In the reported pipeline, the test and library were compiled and instrumented with AddressSanitizer, while Theorem supplied formal models of RISC-V instructions and Linux processes. Claude then wrote a Rocq proof against the resulting specification. When an attempted proof failed, the failure could point to a potential bug; the team could patch the code, regenerate the specification and try again. Rocq checked proofs against the stated claims.

Anthropic describes the project as, to its knowledge, the first large-scale use of AI for formal verification of production software at assembly-code level. The source says the company is publishing unpatched proofs, some previously unpublished trusted components, and specifications for fuzz tests it could not prove. That release may let others examine the work, but it does not by itself validate the method’s broader applicability.

Where the Proofs Stop

The results are limited to the single-threaded, static RISC-V binaries modelled by the researchers, linked against musl. The source says most users encounter x86 or ARM builds linked against glibc, which were not covered. It is therefore not established that the same properties hold for those builds or for the software’s full range of real-world use.

The proof also addresses the behavior encoded in each fuzz-test specification. A no-crash result does not rule out every security flaw, incorrect output or undesirable behavior. The source notes that proving the test cannot crash may establish a higher-level property where the test checks one, but that depends on what the test asserts. It gives a file-restriction logic bug as an example of a problem that may not crash.

Anthropic says proofs can consume hundreds of millions of tokens, with newer models and shared proofs reducing the burden. The source does not provide a complete cost per library, a comparative evaluation against human verification, or independent replication results. It also identifies components that must be trusted, including the formal models, theorem statements and Rocq checker, alongside underlying hardware and software dependencies.

Proof Releases and Cost Tests

The next near-term step is scrutiny of the material Anthropic says it is publishing: proofs that did not require patching, previously unpublished trusted components, and specifications for tests the system could not prove. Those materials could help outside researchers inspect what was checked and identify gaps between the formal claims and the intended software behavior.

Further evaluation would need to test whether the method can be applied beyond this collection of fuzz-test entry points, whether it can cover builds used in production, and how much computing and model usage it requires in practice. The source describes newer models and reuse of proofs as ways to lower token costs, but does not establish the resulting savings. Until those questions are answered, the reported achievement is best read as a large-scale demonstration of a constrained verification workflow.

Key Questions

What did Claude prove?

Claude generated Rocq proofs for a specified no-crash property covering 239 fuzz-test entry points in 123 libraries. The claim applies to the modelled binaries, inputs and assumptions—not to every possible behavior of the libraries.

Does this mean the libraries are bug-free?

No. A proof establishes only the property stated in its specification. The work does not show that every security flaw, logic error or incorrect result has been ruled out.

How did the project use fuzz tests?

The team treated each existing fuzz test as a specification for the compiled program. Claude wrote proofs in Rocq, and Rocq’s proof checker checked them against those specifications.

Why did the proofs find bugs fuzzing missed?

Fuzzing samples inputs and execution paths, while a successful proof must account for all behavior within its formal model and input range. The source describes bugs involving large inputs, rare layouts and specific call sequences.

Do the results apply to the builds most users run?

That has not been shown. The reported proofs concern single-threaded, static RISC-V binaries linked against musl; the source says commonly used x86 or ARM builds linked against glibc were not covered.

Source: ThorstenMeyerAI.com