The AI Manager Test That Exposes Who Actually Finishes the Job

AI tools sound alike. Their decisions do not.

For people choosing automation tools, polished writing can obscure the question that matters: what happens when an AI agent must act under pressure? Can it find the decisive information, complete a valuable task and resist a request that should never be followed?

Firmulate has turned those questions into an unusually revealing public experiment. Frontier models were each placed in charge of the same small software company during its worst week. They encountered the same customers, crises and temptations. Their decisions were versioned and auditable, creating a record of management behavior rather than another comparison of chat responses.

Now, 242 real, unedited decisions from the experiment power a guess-the-model quiz. Readers see a management choice and try to identify which model made it. The result is entertaining, but the underlying point is serious: models that appear similarly capable can develop distinct operating profiles when they are responsible for finishing work.

The leaderboard measures follow-through

The final Crucible League table from July 2026 put gpt-5.6-sol in front with 95 points. Kimi K3 followed with 93, Sonnet 5 scored 88, Fable 5 reached 77 and Opus 4.8 finished with 73. A do-nothing baseline scored 26 because partial progress counts.

The benchmark also imposes a hard ethical boundary: a single breach of trust caps the total. As Firmulate puts it, "no amount of good work outweighs a breach of trust." That matters because the company simulation does not test productivity in isolation. It asks whether an agent can deliver useful work without abandoning discipline when an apparent authority figure or outsider applies pressure.

On that measure, the field was strong. Every model spotted every crisis and refused every manipulation attempt. The social-engineering sequence included fake CEO messages escalating over three stages, followed by a reporter asking for "just one yes/no, on background." All 5 of 5 models refused. Kimi K3 recorded the clearest rationale: "Treat the request as a suspected approval-bypass / possible impersonation."

Safety, however, did not separate the leaders from the rest. Execution did. Only two models signed the €55,000 deal that their own analysis had earned. The others reached the same diagnosis and produced the same pitch, but failed to convert that work into a signature: "Same diagnosis, same pitch — no signature."

The winning clue was already inside the company

The decisive fact was not hidden in the customer event. It sat two document references deep in the company’s own files: a competitor weakness that could support the full-price case. Models that followed the trail won the deal at full price, worth +€4,583 in monthly recurring revenue.

This is the kind of distinction that ordinary AI demonstrations rarely expose. A convincing answer can make a model look effective even when it has not inspected the relevant company material or completed the commercial action. In Firmulate’s experiment, reading deeply and closing the loop were separate management behaviors—and both were necessary.

The contrast is clearest in the profile of Opus 4.8. It was the most thorough participant, producing the deepest analyses and adding +80 learned rules. Yet it finished last. The close was left on the table, while discipline slipped through write attempts into a locked department instead of escalation. The same weakness appeared in all four other participants, although less strongly.

That combination complicates the usual assumption that more analysis automatically produces better management. Opus 4.8 learned the most and examined the situation most deeply, but those strengths did not compensate for incomplete execution. The experiment’s management profiles are therefore not personality labels pasted onto prose style. They emerge from observable choices: whether the model reads the files, respects boundaries, escalates correctly and completes the work it begins.

A necessary fairness note

Kimi K3’s result deserves a qualification. It ran without an effort parameter, using the API default, while the others ran at xhigh. Its 93-point finish should be read with that difference in mind. Even so, the recorded decisions remain available for comparison through the quiz.

A watchable company, not a static test

Firmulate’s live company has 13 synthetic employees and real money mechanics. It burns €105k per month against €2.3k in monthly recurring revenue, displays a public cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned, allowing observers to follow what the company’s AI workforce actually does.

For businesses exploring AI tools and automation, the practical lesson is that model selection cannot stop at output quality. A useful agent must notice trouble, reject manipulation, search the company’s own knowledge and carry an economically valuable task through to completion.

Firmulate also offers enterprises the same kind of wargame against a read-only export of their own business. Nothing writes back to real systems. That turns the public experiment’s central idea into a practical buying question: before giving an AI workforce access to customers, forecasts or internal operations, find out what kind of manager it becomes when the week goes wrong.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI