Turning Database Grading Into RL Rewards: Post-Training Agents With ThinkingBox

TL;DR

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft and Hugging Face say they used executable checks of final database states as reinforcement-learning rewards to fine-tune Qwen3.8-27B. On ThinkingBox-Bench, pass@1 rose from 51.70% to 60.78%, and tasks passed on all 20 recorded attempts increased from 38 to 107; pass@20 fell slightly.

Microsoft and Hugging Face report that they used executable checks of an agent’s final database state as reinforcement-learning rewards, then fine-tuned Qwen3.8-27B with the approach. On ThinkingBox-Bench, the model’s average success across recorded attempts rose from 51.70% to 60.78% after 50 GRPO steps, while the number of tasks it passed on all 20 attempts increased from 38 to 107.

The method builds on ThinkingBox, a benchmark that evaluates agents on whether they leave a backend database in the expected state. Each of its 507 tasks gives an agent a real backend and tools; executable checks then inspect the terminal database records. The authors argue that this outcome-based grading can serve as a training reward, rather than merely scoring an agent after it finishes.

For the training experiment, the team selected tasks in a separate pool using what it calls offline policy-relative difficulty. It estimated each task’s success probability with the base model before training and kept the selected pool fixed. The report says Qwen3.8-27B was then fine-tuned for 50 GRPO steps and evaluated on the 507 benchmark tasks, which it describes as unseen during training.

The reported results show a tradeoff across measures. Pass@1—the share of successful attempts across 10,140 attempts, or 507 tasks run 20 times each—rose by 9.08 percentage points. Tasks passed on every recorded attempt rose from 38 to 107. But pass@20, the number of tasks passed at least once, slipped from 453 to 446, or from 89.35% to 87.97%. The report therefore describes stronger repeatability alongside a small decline in at-least-once task coverage.

Repeatability Gains, With a Coverage Tradeoff

The findings speak to a known limitation of evaluating agents by whether they succeed once. For tasks that change bookings, payments, tickets or other records, a result that varies across repeated runs may be less useful than one that consistently leaves the system in the required state. The report’s 20-attempt measure is intended to show that distinction: after fine-tuning, many more benchmark tasks passed in every recorded run.

Using a database’s final state as a reward could also let training focus on whether the requested outcome was achieved, rather than on copying a particular sequence of tool calls. That matters because an agent might reach the correct result through a different valid path. However, the reported decline in pass@20 coverage shows that higher consistency on some tasks did not translate into improvement on every measure. The experiment is evidence from one model and one reported setup, not proof that the method will improve agents across systems or real-world deployments.

From Database Grading to Training

ThinkingBox was introduced as a way to catch cases where an agent reports completion but leaves an incorrect database record. In an earlier example cited by the authors, an agent marked a customer’s ticket resolved while the underlying carrier exception remained open. The benchmark checks the resulting records rather than treating plausible dialogue or a series of apparently competent tool calls as success.

The training report extends that idea from evaluation to post-training. The authors’ central observation is that many tasks can produce a direct executable outcome—a pass or fail based on the final database state—and that outcome can be used as a reward signal. They selected practice tasks based on the base policy’s estimated likelihood of success, on the premise that tasks already easy for the model or far beyond its current ability may provide less useful training.

The report also gives a single-task illustration involving a hotel booking. Company policy required manager approval before a requested room downgrade. The base model changed the booking and issued refunds without approval in 15 of 20 attempts; after fine-tuning, it preserved booking and payment state in all 20. The authors say the expected records matched in 19 of those post-training attempts, a reminder that the task-level example does not show a perfect result across every check.

“That is not just a grader. This can be a reward function.”

— Microsoft and Hugging Face, in the report

Limits of the Reported Evaluation

The source provides results for one model, one 50-step training run and one benchmark evaluation. It does not establish whether the gains will hold across other model families, training budgets, tool environments or live production workloads. The report says the benchmark tasks were not used for training, but the supplied material does not provide enough detail to independently assess the full training pool, task-selection thresholds, run-to-run variation or statistical uncertainty.

It is also unclear from the supplied report excerpt how the method performs over longer training or whether the measured decrease in pass@20 coverage persists under other settings. The hotel example illustrates one policy-sensitive task, but does not establish a general reduction in unsafe database changes. The results are reported by the project’s authors; no independent replication or peer-reviewed assessment is identified in the source material.

Code and Further Testing

The authors say ThinkingBox-Training is available on GitHub, giving other researchers and developers a way to inspect the training implementation. The report does not announce a scheduled follow-up experiment or deployment. The next useful evidence would include replication across multiple training runs and models, clearer reporting of the task-selection and reward setup, and evaluation on additional held-out environments.

For now, the reported benchmark results support a narrower conclusion: executable checks of final database state can be used as rewards in this post-training setup, and the authors observed improved pass@1 and all-20-attempt success for Qwen3.8-27B. Whether that improvement transfers broadly—and how to balance repeatability against task coverage—remains open.

Key Questions

What is ThinkingBox?

ThinkingBox is a benchmark that gives agents tasks involving a backend and tools, then checks whether the final database records match the expected outcome.

How did the researchers use database checks in training?

They used executable pass-or-fail checks of final database states as reinforcement-learning rewards while fine-tuning Qwen3.8-27B with GRPO.

What results did the report give?

After 50 GRPO steps, reported pass@1 increased from 51.70% to 60.78%, and tasks passed on all 20 attempts rose from 38 to 107. Pass@20, tasks passed at least once, declined from 453 to 446.

Does the result show the method works for all AI agents?

No. The report describes one model and one training setup. More tests are needed to establish whether the results transfer to other models, environments or production use.

Where is the training implementation?

The report says ThinkingBox-Training is available on GitHub. The supplied source does not include a specific repository address.

Source: rss