What Microsoft’s Doctors-versus-AI Study Can Actually Show

🔍 Read the full analysis: What Microsoft’s Doctors-versus-AI Study Can Actually Show on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic system scored 83.2% on a benchmark of 304 difficult medical cases, compared with 36.1% for generalist physicians using textbooks and the internet, according to a peer-reviewed Science paper published Oct. 22. The findings show strong performance in a controlled, text-based test; they do not establish how the system would perform with real patients or alongside clinical teams.

The study also compared MAI-DxO with physicians on 325 emergency department cases. Microsoft reported scores of 80.3% for the system and 46.8% for physicians. On the 304 complex cases, doctors without reference materials scored 22.3%, while those with textbooks and internet access scored 36.1%. These figures are results reported by the study’s authors, all of whom are affiliated with Microsoft.

The benchmark, called the Sequential Diagnosis Benchmark, begins with a short case description. The system or doctor must request additional information, such as a patient-history detail or test result, rather than seeing the full case file at the outset. The test evaluates the final diagnosis and the estimated cost of obtaining more information. MAI-DxO uses a language model in a multi-agent process that proposes diagnoses, selects questions or tests and critiques possible answers over successive rounds.

A key comparison is that GPT-5 alone scored 78.9% on the complex cases, close to MAI-DxO’s 83.2%. Microsoft’s chart puts the system’s estimated testing cost at $2,397 per case, compared with $8,409 for GPT-5 alone. The figures suggest the orchestration approach may affect cost as well as accuracy within this benchmark, but do not show that it reduces costs in routine care.

What the Scores Mean for Care

The results matter because diagnostic systems could influence which questions clinicians ask, which tests they order and how quickly they reach a diagnosis. A benchmark that tests those steps is closer to the diagnostic process than an exam where all facts are provided at once. It also tries to account for the expense of gathering information, a concern for hospitals and insurers weighing possible uses of AI.

But the reported accuracy gap should not be read as evidence that MAI-DxO is ready to replace doctors. The comparison was between a coordinated AI system and individual generalists working alone, not specialists or teams, on cases selected for difficulty. The evaluation also did not include healthy patients or cases with harmless explanations, so it could not measure false alarms. In practice, excessive testing or incorrectly labeling a patient with a serious disease can carry costs and risks that this benchmark does not capture.

The study therefore offers evidence about performance under a particular test design, not a direct estimate of clinical benefit. Patients, providers and health systems would need evidence from real-world use—including safety, effects on care decisions and costs—before drawing conclusions about how the system should be used.

From Preprint to Science Paper

The Science paper, titled Sequential diagnosis with language models, was published on Oct. 22, 2026. An earlier version appeared on arXiv in June 2025 as a non-peer-reviewed preprint. That version reported a larger gap, with the AI scoring 80% against 20% for doctors on its comparison. The figures in the final paper differ, and details described in the preprint may not all be identical to the final study.

Microsoft’s public messaging has highlighted ratios between its system and physicians, including results on emergency cases and complex cases with or without references. Those are company-reported comparisons. Publication in a peer-reviewed journal means the paper underwent review; it does not by itself show that independent researchers have reproduced the results.

The test’s design also relies on estimates and model judgments. According to the earlier preprint, a separate model graded answers against published diagnoses, and researchers used estimated test prices based on billing codes and a 2023 U.S. hospital price list, plus a fixed physician-visit cost. When a requested test result was absent from a case record, a model generated a synthetic result. Those design choices help make the benchmark workable, but they are not equivalent to measuring actual clinical decisions and bills.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

Limits the Benchmark Cannot Resolve

It is not yet clear whether MAI-DxO’s reported advantage would persist with real patients, physical examinations, imaging and clinical teams. The cases were historical and text-only, and the physicians in the comparison were generalists rather than specialists. The study also did not test false-positive diagnoses against healthy patients or benign conditions.

The accuracy figures depend partly on a model-based grading process. The earlier preprint says physician reviewers agreed with the grader on a sample at a good, but imperfect, level. The exact final-paper methods may differ from the preprint, whose full text provides some of the methodological detail because the Science article is behind a paywall. Independent replication has not been reported in the supplied material.

It also remains uncertain how much of the result comes from MAI-DxO’s multi-agent process rather than the underlying language model. GPT-5 alone came within several percentage points of the system on the complex cases. The test’s estimated cost advantage does not establish savings in a hospital, where staffing, patient outcomes, repeat tests and other expenses matter.

Testing With Patients in Boston

Microsoft has said a prospective study at Beth Israel Deaconess Medical Center in Boston is planned. In that study, consenting patients are expected to interact with the system alongside their usual care. Unlike the historical-case benchmark, a prospective evaluation could provide evidence about how the tool behaves in a clinical setting, although the supplied material does not give a start date, enrollment target or detailed outcome measures.

The next evidence to watch for includes whether the system’s recommendations change clinicians’ decisions, whether those changes improve patient outcomes, how often it prompts unnecessary testing, and what its use costs in practice. Until those results are available, the Science paper supports a narrower conclusion: MAI-DxO performed strongly on the study’s difficult case records, but its real-world safety and clinical value remain to be tested.

Key Questions

What did Microsoft’s study find?

The Science paper reports that MAI-DxO scored 83.2% on 304 difficult medical cases. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without those resources.

Was the study conducted with patients?

No. The benchmark used historical, text-only case records in a simulator. It did not involve real patients, physical examinations or imaging.

Did MAI-DxO outperform every type of doctor?

The reported comparison was with generalist physicians working individually. The study did not compare the system with specialists or physician teams, so it does not establish how it would perform against them.

How much did the multi-agent system add?

On the complex cases, GPT-5 alone scored 78.9%, compared with 83.2% for MAI-DxO. The study’s figures suggest a modest accuracy difference in that test, but do not establish which parts of the system would add value in clinical care.

What is the next test of the system?

Microsoft has said a prospective study is planned at Beth Israel Deaconess Medical Center in Boston, where consenting patients would interact with the system alongside normal care. Details such as timing and study measures were not provided in the source material.

Source: ThorstenMeyerAI.com