TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A Memdoor report says a workflow using five Qwen3.8-27B model roles averaged 9.8 correct solutions out of 10 across five runs on each of 10 LiveCodeBench Hard problems. The result is a small-sample parity check against the cited GVS5H paper’s harness score of 9.6, not evidence that the workflow outperforms Fable 5: the Fable 5 score was a separate, one-call result from the authors’ files.
Memdoor reports that a workflow using five Qwen3.8-27B model roles averaged 9.8 correct solutions out of 10 across five runs on each of 10 LiveCodeBench Hard problems. The result nearly matches the 9.6-out-of-10 score the project attributes to the GVS5H paper’s own harness, but does not establish that the workflow beats Fable 5: that comparison score came from the paper’s files and was not produced in Memdoor’s run.
The Memdoor project describes a team-solving workflow with a planner, two ideators, a solver, a manager and a repair worker. The source headline calls it a team of five models; the workflow description lists six roles, so the precise distinction between model instances and workflow roles is not fully explained in the supplied material. The agents start with fresh contexts and communicate through files on disk rather than relying on shared conversation history. The workflow is written as eight YAML files and does not name a particular model in its steps.
For its comparison, Memdoor says it rebuilt the paper’s harness using Qwen3.8-27B, the paper’s prompts and loop, and a 128K reply cap. It ran the first 10 problems on the LiveCodeBench Hard list five times each. A corrected grader checked programs against hidden tests; the project says it retained request and reply logs, generated programs, grader output and cost for all 50 runs. The reported mean was 9.8 out of 10 for Memdoor’s workflow, 9.6 for the paper’s harness and 8.0 for Fable 5.
Memdoor reports a median run time of 40 minutes and a total cost of $35.79 for the 50 runs. It says one problem was lost after the manager proposed an incorrect counterexample and the team replaced a correct program. On two other problems, the workflow overwrote a correct program before recovering it. The project states these gaps were part of the authors’ original loop and were left unchanged in the parity check.
What the Score Can Establish
The result is useful as a reproducibility check: Memdoor says it implemented the cited paper’s method and obtained a similar score under a repeatable setup, with artifacts available for inspection. That matters to developers evaluating whether a multi-agent approach can be expressed as a workflow and tested with recorded inputs, outputs and grading results, rather than accepted on the basis of a summary score alone.
But the reported margin is not evidence of a meaningful performance advantage. The difference between 9.8 and 9.6 is only two-tenths of a problem on a 10-problem set, and Memdoor explicitly says that the sample is too small to treat the numbers as different. The experiment also does not show that a group of agents is generally better than a single model: the Fable 5 number was a one-call result from the paper’s files, not a run in Memdoor’s harness.
For readers comparing coding systems, the key takeaway is narrower: the project reports close parity with the paper’s harness on a small set, while exposing the workflow’s costs, failure modes and test artifacts. It does not settle how the method performs across the paper’s full 100 problems or on newer, unseen tasks.
Products Worth Considering
Apple MacBook Air 15-inch (2026) with M5 chip, 16GB memory, 512GB SSD, Silver
Samsung Galaxy Tab S10 FE 128GB Wi-Fi Android Tablet, Blue
Bose QuietComfort Ultra Wireless Noise-Cancelling Headphones with Spatial Audio, Black
As an affiliate, we earn on qualifying purchases.
How Memdoor Recreated the Workflow
The workflow follows a sequence of planning, parallel proposals, solution generation, testing, review, repair and verification. The planner writes a plan; two ideators propose approaches from different algorithmic families; and the solver produces a program. A test command runs the supplied sample tests, while a reviewer reads the output and writes repair instructions if needed. A repair agent can change the program, after which a verification command determines whether the run passes.
Each role is instructed to begin with a fresh context and read shared files instead of relying on prior chat. The project says the method is based on GVS5H, credited to Lee (2026), and that its implementation uses the Mario workflow engine. Its reported benchmark uses public samples during the loop, while hidden tests determine the score. The underlying contests are dated to spring 2025; Memdoor says possible training-data contamination cannot be ruled out for any of the compared results.
“On a sample this small, 9.8 and 9.6 are the same number.”
— Memdoor project report
Limits of the Benchmark Comparison
The reported test covers 10 problems, not the paper’s 100, and repeats each five times. The material does not provide enough information to establish how representative those problems are of the full benchmark or of future programming tasks. The project itself says the sample is too small to distinguish the 9.8 and 9.6 averages.
The Fable 5 figure also has a different provenance: Memdoor describes it as a one-call-per-problem score from the authors’ files, not a result generated under the same Memdoor setup. It is therefore not a controlled head-to-head comparison. The supplied report does not independently establish the validity of the paper’s original figures or describe external verification of Memdoor’s grading.
Training contamination remains another open issue because the contests date from spring 2025 and the models may have encountered them during training. The project says contamination cannot be ruled out, but supplies no estimate of its likelihood. The report also refers to a team of five models while listing six workflow roles; it does not clarify whether some roles share a model instance.
Full Runs and Independent Checks
Memdoor’s repository provides run logs and benchmark artifacts for the 50 reported trials, along with notes about the parity check, execution hosts and changes needed to recreate the first run. Readers can inspect those materials and run the workflow on their own programming task. The project says it can be connected to hosted model services or local inference software, with the model selected outside the workflow steps.
The clearest next test would be to run the method across the full 100-problem set and compare all systems using the same problem versions, grading rules and execution conditions. Until such results are available, the current finding remains a limited reproduction report rather than proof of a general coding advantage.
Key Questions
What did Memdoor report?
It reported that a multi-role Qwen3.8-27B workflow averaged 9.8 correct solutions out of 10 over five runs on each of 10 LiveCodeBench Hard problems.
Did the workflow beat Fable 5?
The reported averages were 9.8 for Memdoor and 8.0 for Fable 5, but they were not produced in a matched test. Memdoor says the Fable 5 figure was a one-call result from the paper’s files, so the report does not establish a controlled head-to-head win.
How does the team-solving process work?
Separate workflow roles plan the task, propose approaches, write a solution, run sample tests, review results and, if needed, repair and verify the program. The agents communicate through files on disk and are instructed to use fresh contexts.
How large was the test?
The run covered 10 problems with five passes each, or 50 runs in total. The report says the paper’s evaluation included 100 problems, which Memdoor did not reproduce in this test.
What remains uncertain?
The results may not generalize beyond the small sample, possible training contamination has not been ruled out, and the Fable 5 score was not rerun under the same conditions. The report also does not explain the difference between its headline description of five models and its list of six workflow roles.
Source: github.com
