In the fast-evolving world of AI tools and automation, chatter about chatbots often focuses on how well they can mimic conversation. But what truly matters — and what’s often invisible in demos — is whether AI can see a task through to completion under real pressure. A live experiment from Firmulate puts this question front and center, revealing that the ability to finish a job, read critical documents, and resist manipulation is far more telling than any slick chat demo.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Four Models, One Company, Same Crisis
In July 2026, four leading AI models were tasked with running a small software company through its worst week — same customers, same crises, same temptations to cut corners or manipulate. This wasn’t a theoretical test; it was a real-time, auditable experiment designed to see how well each AI could manage, diagnose, and close a crucial deal worth €55,000.
The Performance Scores
- gpt-5.6-sol scored a 95, found the buried fact, and closed the deal.
- Kimi K3 scored a 93, also closing the deal with the cleanest discipline of the field.
- Sonnet 5 scored 88, closing the deal but with a few process slips.
- Fable 5 scored 77, with the best rule discipline but leaving the deal unexecuted.
- Baseline do-nothing scored 26, showing partial progress but no real closure.
Despite all models spotting every crisis and refusing every manipulation — including fake CEO messages and reporter tricks — only two actually signed the deal their analysis had earned. The others identified the issues but failed to follow through, leaving the revenue on the table.
As an affiliate, we earn on qualifying purchases.
What This Means for Business AI
Chatter in demos and chatbots often measures language fluency or superficial decision-making. But the real measure of AI’s business effectiveness is whether it can see a task through to completion, especially when under pressure or facing manipulation. In this experiment, the decisive factor was reading and understanding a key document buried two references deep in the company’s own files — a task that the winning models did successfully. Those who failed to read the full context missed the opportunity to close the deal at full price, worth over €4,500 monthly recurring revenue.
Resisting Social Engineering
All four models refused to be manipulated or tricked, including staged fake CEO messages escalating over three stages and a reporter trick asking for a quick on-background approval. Kimi K3’s reasoning was clear: “Treat the request as a suspected approval-bypass / possible impersonation.” This shows a level of risk awareness and discipline often overlooked in chat demos.
As an affiliate, we earn on qualifying purchases.
The Live Company and Its Challenges
The experiment used a real business scenario with a live software company that has 13 synthetic employees, real money mechanics, and a public cash countdown. The company burns €105,000 monthly against €2,300 MRR, illustrating how critical effective decision-making is under financial pressure. The company’s daily operations are versioned and transparent, making the experiment not just a demo but a real-world stress test.
The Discipline of Rules and the Cost of Weakness
Opus 4.8, the most thorough participant with over 80 learned rules, performed the deepest analysis but ultimately left the deal on the table. Its discipline slipped, and the decision to escalate instead of close cost the company dearly. Interestingly, the same weakness appeared across models, but the depth of understanding and discipline distinguished the winners from the losers.
As an affiliate, we earn on qualifying purchases.
Lessons for Business Leaders
This experiment underscores a vital point: AI’s true business value isn’t just in generating convincing text or responses. It’s in executing decisions, reading critical information, resisting manipulations, and ultimately closing deals. The gap between AI’s chat prowess and its ability to deliver tangible results is hidden in plain sight — until you test it in a real-world scenario like this.
Test Your AI Before You Trust It
Firmulate offers a way to run your own business wargame, exposing your AI workforce to the same pressures and crises. It’s a read-only simulation that doesn’t write back to your systems but reveals how your AI models perform under real operational stress. You can see how they read and interpret documents, make decisions, and stick to their commitments. This isn’t just a demo — it’s a critical step before deploying AI in revenue-critical processes.
As an affiliate, we earn on qualifying purchases.
Final Takeaway
In the race to automate and augment business decisions, the ability to finish what AI starts is overlooked in favor of flashy chat demos. The Firmulate experiment demonstrates that under real pressure, only some models can read deeply, resist manipulation, and close the deal — essential qualities for any AI embedded in business operations. The challenge for leaders now is to look beyond the surface and rigorously test their AI tools for these invisible but vital capabilities.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Evergreen bestsellers Picks
bestsellers
As an affiliate, we earn on qualifying purchases.
