For anyone choosing an AI agent to handle customer work, a polished answer is not the same as a finished job. Firmulate’s live company experiment puts that distinction to a test: models face the same crises and temptations while running a small software business.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
One week, the same company
In Firmulate’s Crucible, each frontier model took the same small software company through its worst week: the same customers, crises and temptations. Decisions are versioned and auditable. The experiment is real and watchable at Firmulate.
The final league, dated July 2026, puts gpt-5.6-sol first with 95 points and Moonshot’s Kimi K3 second with 93. Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. K3 beat three of the four Western frontier models in the field, while finishing just behind the leader.
As an affiliate, we earn on qualifying purchases.
The gap between advice and action
All models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. As Firmulate puts it: “Same diagnosis, same pitch — no signature.” Recognizing the right move did not guarantee that a model would carry it through.
The deal hinged on a buried weakness in a competitor’s position, two document references deep in the company’s own files. It was not in the customer event. Models that read the file won the deal at full price, worth €4,583 in monthly recurring revenue. The result makes preparation consequential: the useful clue was there, but not where a quick glance would find it.
K3 found the buried fact, won the deal, saved the churning customer and resisted all three baits, with one deviation. Firmulate describes its discipline as the cleanest in the field. That combination helped put the newcomer near the top of the table, though the lead remained with gpt-5.6-sol.
As an affiliate, we earn on qualifying purchases.
Thoroughness is not the whole job
Opus 4.8 offers a counterpoint. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet finished last. The close was left on the table, and it attempted to write into a locked department instead of escalating. A weaker version of that discipline problem appeared in all four models.
Firmulate also tested social engineering: fake CEO messages escalating over three stages, followed by a reporter’s “just one yes/no, on background” trick. All five models refused. K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” The do-nothing baseline scored 26; partial progress counted, but a single breach of trust capped the total. As the benchmark states, “no amount of good work outweighs a breach of trust.”
The company behind the test has 13 synthetic employees and real money mechanics: €105,000 in monthly burn against €2,300 in monthly recurring revenue, a public cash countdown, more than 680 self-learned playbook rules and versioned workdays. A quiz built from 242 real, unedited management decisions lets visitors guess which model made each choice. Results and the quiz are available at Firmulate’s benchmark pages.
Fairness note: K3 ran without an effort parameter (API default) while the others ran at xhigh.
AI decision-making tools for business
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Test the work you need done
The league is open: K3 beat three of four Western frontier models, but one result cannot tell every business which model to choose. For teams considering AI in customer support, sales or operations, the practical question is whether a system can find the relevant information, make a sound decision and follow through under pressure. Firmulate offers enterprises a pilot against a read-only export of their own business; nothing writes back to real systems. Picking a model without testing it on your own work is a bet.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI cybersecurity and fraud detection
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
