🔍 Read the full analysis: The Measure That Shaped Microsoft’s AI Beats Doctors Claim on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft’s MAI-DxO diagnostic system scored 83.2% on 304 difficult NEJM case records, compared with 36.1% for generalist physicians using reference materials. The peer-reviewed study used historical, text-based cases rather than patient care; the authors say a prospective study alongside normal care is planned.
Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 difficult case records from the New England Journal of Medicine, compared with 36.1% for generalist physicians who could use textbooks and the internet, according to a Science paper published October 22. The result is a benchmark comparison, not evidence that the system diagnoses real patients better: the study used historical, text-only cases and compared a multi-agent AI system with individual generalists, not specialist teams.
The paper, “Sequential diagnosis with language models,” reports results on two case sets. On the NEJM records, MAI-DxO achieved 83.2% diagnostic accuracy. Physicians scored 36.1% with reference materials and 22.3% without them. On a separate set of 325 emergency department cases, the system scored 80.3%, compared with 46.8% for physicians. These figures are reported by Microsoft’s research team; they describe performance under the study’s test conditions.
The benchmark, called the Sequential Diagnosis Benchmark, gives each participant a short case summary. To obtain more information, the participant must request questions or tests from a model holding the full case record. Tests carry estimated costs, so the benchmark considers both the final diagnosis and the resources used to reach it. MAI-DxO uses a language model to run a panel of virtual physicians that proposes diagnoses, requests information and challenges its own reasoning.
The headline score does not isolate the benefit of that orchestration. Microsoft’s chart reports that GPT-5 alone scored 78.9% on the difficult cases, compared with 83.2% for MAI-DxO. The source account says the system’s added diagnostic gain was about four percentage points and that its orchestration reduced estimated test costs. Those cost estimates are not actual bills or observed spending in clinical practice.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
Why the comparison matters
The results add to evidence that language models can perform strongly on structured diagnostic challenges, including cases that require deciding what information to seek next rather than simply choosing from a fully provided list of facts. That distinction matters because clinical work involves gathering evidence as well as interpreting it. The benchmark’s cost component also offers a way to examine whether a system reaches an answer with fewer or more tests, although its estimated prices do not establish savings for hospitals or patients.
For clinicians and health systems, the study is a research result, not a readiness assessment for deployment. A high score on selected records does not show how a system performs with incomplete or conflicting information, patients with common benign conditions, or the many practical demands of care. The reported comparison also does not answer whether AI can improve outcomes, reduce diagnostic delays or fit safely into a clinician’s workflow. Those are separate questions requiring evaluation with patients and clinical teams.
Patients should read “outperformed doctors” with the comparator in view: generalists working individually, rather than specialists or teams. The finding may justify further testing, but it cannot establish that a patient should rely on an AI system instead of a qualified clinician.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Microsoft built the test
The paper appeared in Science, volume 394, pages 423–428, on October 22, 2026. The source account says the manuscript was received in April and accepted in August, and that all 15 listed authors had Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman. Peer review means the paper was evaluated for publication; it does not by itself show that independent researchers have reproduced the results.
An earlier, non-peer-reviewed version appeared on arXiv in June 2025. It reported a larger gap using an earlier model configuration. The later Science paper is the relevant published report, and its figures should not be merged with the earlier preprint’s results. Microsoft’s public presentation emphasizes ratios between system and physician scores, while the paper’s raw percentages show the comparison groups and test conditions more directly.
Some methodological details described in the source account come from the preprint, and may differ in the final paper. In that earlier version, a separate model judged diagnoses against published answers; physicians reviewing a sample showed good, but imperfect, agreement with that scoring. When a requested test result was absent from a case file, a gatekeeper model generated a synthetic result. Test prices were estimated using billing codes and a 2023 U.S. hospital price list, alongside a fixed physician-visit charge. The authors described these as first-order approximations.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch
What the study cannot establish
The benchmark did not involve real patients, physical examinations or imaging. It used historical case records, and the test cases were difficult teaching cases. The source account says the NEJM set contained no healthy patients or harmless explanations, meaning the study could not measure false alarms across those groups. Performance on this sample therefore cannot establish how the system would behave across the full range of patients seen in routine care.
The physician comparison also has limits. According to the source account, the doctors were primary care physicians and hospital generalists working alone; specialists and clinician teams were not included. In the earlier preprint, physicians were tested on 56 held-out cases while the AI was assessed on all 304. The published Science paper reports its own comparison figures, but the available source material does not provide enough detail to confirm whether every preprint design detail carried into the final version.
It also remains unclear whether independent teams will reproduce the scores, how the system would perform prospectively, and whether it would change patient outcomes or total care costs. A peer-reviewed publication is not an independent replication, and estimated test costs in a simulation are not evidence of savings in hospitals.
The planned patient study
The source account says Microsoft’s next step is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their normal care. That design could provide evidence about how the tool functions in a clinical setting, but the supplied material does not give a start date, enrollment target, study protocol or expected results date.
Readers should look for details on how clinicians oversee the system, what outcomes researchers measure, and whether the study compares AI-assisted care with usual care. Until those findings are available, the Science results remain evidence about performance on selected simulated cases—not a demonstration that MAI-DxO is ready to diagnose or treat patients independently.
Key Questions
What did Microsoft’s MAI-DxO score on the difficult cases?
The Science paper reports 83.2% accuracy on 304 NEJM case records. Generalist physicians scored 36.1% with reference materials and 22.3% without them in the reported comparison.
Was MAI-DxO tested on patients?
No. The reported benchmark used historical, text-based case records in a simulator. It did not include real patient interactions, physical exams or imaging.
Does the study show AI is better than specialist doctors?
No. The comparison was with generalist physicians working individually. The source account says specialists and physician teams were not included, so the study does not establish how MAI-DxO compares with them.
What happens after the published study?
The source account describes a planned prospective study at Beth Israel Deaconess Medical Center, where consenting patients would use the system alongside their normal care. Its timing and protocol were not provided.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
