🔍 Read the full analysis: Can Database Grading Help Post-Train Agents With ThinkingBox? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub and uses executable checks of final database state as rewards for agent training. In one 50-step GRPO run, Qwen3.8-27B’s average success rate rose from 51.70% to 60.78%, while tasks passed in all 20 attempts increased from 38 to 107; tasks solved at least once fell from 453 to 446.
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub, with executable checks of an agent’s final database state used as rewards during post-training, as detailed in the original analysis. In a reported 50-step GRPO run, Qwen3.8-27B achieved a higher average success rate on a 507-task benchmark, though it solved slightly fewer tasks at least once—a result that points to improved repeatability without broader task coverage.
The teams evaluated the model on 507 benchmark tasks, each run 20 times. Before training, it succeeded in 5,242 of 10,140 attempts, or 51.70%. After training, it succeeded in 6,163 attempts, or 60.78%. The number of tasks passed on every recorded attempt rose from 38, or 7.50%, to 107, or 21.10%.
The results were not better on every measure. The share of tasks solved at least once across the 20 attempts—reported as pass@20—fell from 453 tasks, or 89.35%, to 446, or 87.97%. The authors’ figures therefore show more consistent success on a larger set of tasks, alongside a small decline in the number of tasks the model managed to solve at least once.
For training, the team used a pool of ThinkingBox-format practice tasks separate from the evaluation benchmark. The post says it selected tasks using an estimate of how difficult each was for the base model, then kept the pool fixed during training. The authors report that the model was trained for 50 GRPO steps and evaluated on the separate benchmark afterward.
Reliability Gains and Coverage Tradeoffs
Database-writing agents can produce convincing replies while leaving business records incomplete or incorrect. Scoring the final database state gives a training system a way to check whether the requested outcome is reflected in the backend, rather than relying on an agent’s account of what it did or on whether it followed one prescribed sequence of tool calls.
The reported results matter because they separate two aspects of performance: repeatability and coverage. A higher average success rate and more tasks passed in all 20 attempts suggest gains in consistency on the benchmark. But the lower pass@20 count means the model did not expand the set of tasks it could solve at least once. The figures do not establish that the approach will produce the same gains in live business systems.
The authors also describe a hotel-booking task in which a traveler requested a room downgrade requiring manager approval. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. Afterward, it preserved the booking and payment state in all 20 attempts; the authors say all expected records matched in 19. This is an example from one task, not evidence of an overall rate across hotel bookings.
AI training reinforcement learning tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
From Database Evaluation to Training
ThinkingBox was developed as a way to evaluate agents by the records they leave behind. The benchmark described by the authors includes 507 tasks, each with a backend, a set of MCP tools and a simulated user with private context. Executable checks compare the resulting database state with the expected outcome after the agent finishes.
The teams’ central idea is to use those checks not only to grade agents but also to train them. In their account, the checks reward the required outcome without requiring the agent to reproduce a specific set of tool calls. That can allow different valid methods while still testing whether the intended records are present.
The authors describe the evaluation benchmark as out of distribution: its domains, databases and most MCP tools were not encountered during training. That design is intended to test performance beyond the practice pool, but the source material does not provide production deployment results or evidence from other models.
“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”
— Microsoft and Hugging Face, in their joint post
As an affiliate, we earn on qualifying purchases.
What the Single Run Cannot Show
The reported figures come from one 50-step training run. The supplied account does not say whether the results hold across different random seeds, training-pool selections or run lengths, and it does not provide a full comparison with alternative reward designs or training methods. Those gaps make it difficult to isolate how much of the change came specifically from database-state rewards.
The reason pass@20 fell by seven tasks is also not established in the material. Nor are results reported from a live production environment. Generalization, deployment reliability and training cost remain open questions; the benchmark numbers alone do not resolve them.
AI agent performance benchmarking software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Reproducing Results Beyond the Benchmark
With the stack available on GitHub, researchers and developers can inspect the implementation and attempt their own training runs. The next useful evidence would be independent reproduction of the reported gains across more runs, models and task pools, including a closer look at whether higher consistency continues to come with lower task coverage.
Teams considering deployment would also need evaluations tied to their intended workflows. Those tests should check whether required records are correct and whether agents avoid incomplete or unauthorized changes, rather than relying only on benchmark averages. The source material does not specify a date for further results or a production release.
machine learning model evaluation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is ThinkingBox-Training?
It is a Microsoft and Hugging Face training stack, described as available on GitHub, that uses executable checks of an agent’s final database state as rewards during post-training.
How did Qwen3.8-27B perform after training?
In the reported 50-step run, its average success rate across 10,140 attempts on 507 tasks rose from 51.70% to 60.78%. The number of tasks passed in all 20 attempts rose from 38 to 107.
Did the model solve more tasks overall?
Not by the reported pass@20 measure. The number of benchmark tasks solved at least once fell from 453 to 446, despite the gains in average success and all-20-attempt consistency.
Do these results show that the method works in production?
No. The supplied account reports benchmark results from one training run, not performance in live production workflows. Production reliability and broader generalization remain untested in the reported material.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
