Microsoft AI beats doctors in simulated diagnosis test

Ever leave a doctor’s appointment with more questions than answers? Maybe you’ve gone through several tests, only to hear that your doctor still isn’t sure what’s causing your symptoms. That uncertainty can be exhausting, especially when you want answers. Now, Microsoft says its new artificial intelligence system could help doctors solve some of medicine’s toughest diagnostic puzzles. And the early numbers are impressive.

In a study published on Oct. 22 in the journal Science, Microsoft’s AI diagnostic system correctly identified 83% of complex medical cases, compared with 36% for generalist physicians who could consult reference materials. That’s more than twice the accuracy under the study’s testing conditions.

However, there’s an important catch behind those numbers. The research took place in a simulation, and the doctors weren’t specialists working together on difficult cases. Still, the findings raise a fascinating question about how artificial intelligence could change your next visit to the doctor.

Missed CyberGuy LIVE? Watch the replay and discover 5 ways to travel smarter and save money.

Our free CyberGuy LIVE class, 5 Ways to Travel Smarter and Save Money, has ended, but you can still watch the full replay. Kurt “CyberGuy” Knutsson walks you through five practical ways to protect your phone and personal information, avoid common travel scams, find better deals, save money on bookings and use technology to make your travels easier and less stressful. No technical experience is needed.

Watch the free replay + downloadable checklist now at CyberGuyLive.com.

AI MAY SPOT DEADLY HEART RISK IN A ROUTINE ECG

How Microsoft’s AI diagnostic system works

Microsoft calls its technology the MAI Diagnostic Orchestrator, or MAI-DxO. The system uses multiple AI agents that work together much like a panel of doctors discussing a difficult medical case. Instead of jumping to an immediate answer, the AI begins with limited information. It asks follow-up questions and considers which medical tests might help identify the problem. Then, its AI agents challenge one another’s conclusions as new information becomes available.

Think about someone who has unexplained fatigue and several unusual lab results. A doctor might consider one possible explanation before ordering additional tests. Microsoft’s system can examine multiple possibilities and reconsider its initial assessment as the evidence changes. That approach could prove especially useful when symptoms overlap across different diseases.

According to Microsoft’s published findings, researchers evaluated the technology using 304 complex medical cases from The New England Journal of Medicine . These cases often involve unusual illnesses that challenge even experienced physicians. The results were striking. MAI-DxO correctly diagnosed approximately 83% of those cases. Generalist physicians working alone achieved 36% accuracy when they could use reference materials. Without those resources, their accuracy dropped to 22%. Researchers also tested the system using historical emergency department cases. There, the AI achieved approximately 80% accuracy, compared with 47% for the physician comparison group.

Why the AI versus doctors comparison needs a closer look

Before anyone starts canceling doctor appointments, there’s something important to understand about this research. The physicians in the comparison were generalists, including primary care doctors and general internists. They evaluated cases individually, even when they had access to medical reference materials.

However, hospitals typically involve specialists when patients develop particularly difficult medical problems. Those specialists may work together and review additional evidence before reaching a diagnosis. Microsoft did not directly compare its system against teams of medical specialists.

There’s another important detail. Microsoft’s AI was evaluated across 304 complex cases, while physicians worked through a smaller set of 56 cases held out from the system’s development process. That means the headline percentages don’t represent two groups completing exactly the same number of cases.

That limits how broadly we can interpret the findings. The researchers also relied on existing medical cases rather than treating patients in real time. While those cases provide a useful challenge, actual medical care can be far more unpredictable.

Patients sometimes struggle to describe symptoms. Medical records may contain gaps, and physical examinations can reveal clues that never appear in a written case summary. The simulation also relied on an AI system to supply information from existing case reports. When certain details weren’t available, it could generate plausible responses. That helped researchers create diagnostic conversations, although those interactions cannot fully reproduce the uncertainty doctors encounter with actual patients.

Furthermore, reaching the correct diagnosis represents only part of caring for someone. Doctors must determine the appropriate treatment and understand how individual circumstances affect medical decisions. The study’s authors also work for Microsoft, making independent testing especially important before hospitals consider relying on the technology for patient care. For now, Microsoft’s results show how well the technology performed under controlled testing conditions. Whether it can deliver similar benefits during everyday patient care remains an open question.

HIDDEN CLUES COULD HELP PREDICT A WOMAN’S BREAST CANCER RISK YEARS IN ADVANCE

What makes Microsoft’s medical AI different?

You might be wondering how this technology compares with ChatGPT or other AI assistants that already answer medical questions. The underlying AI models play a major role in Microsoft’s system. In fact, the researchers tested models developed by OpenAI, Google, Anthropic and other companies.

One finding deserves attention. OpenAI’s GPT-5 correctly solved approximately 79% of the complex cases when researchers used a basic prompt without Microsoft’s orchestration system. Adding MAI-DxO increased accuracy to approximately 83%. That’s an improvement, although the underlying AI model already performed remarkably well.

With the newer GPT-5.5 model, the gains were larger on complex cases. Microsoft’s orchestration system raised accuracy from approximately 72% to nearly 87% in that comparison. However, the additional system provided little improvement on emergency department cases using the strongest models. These results suggest that much of the diagnostic capability comes from powerful AI models themselves. Microsoft’s approach can improve its performance further, depending on the model and medical problem.

Meanwhile, Microsoft CEO of AI Mustafa Suleyman has described the company’s larger ambition as “medical superintelligence.” The idea involves developing AI systems capable of eventually exceeding today’s strongest medical teams in clinical reasoning and coordination. Microsoft is also collaborating with Mayo Clinic on AI technology that could eventually help clinicians anticipate how a patient’s medical condition might develop. Those ambitions extend well beyond the capabilities demonstrated in the current study.

Could Microsoft’s AI help lower medical costs?

Anyone who has received a surprisingly expensive medical bill knows how quickly diagnostic testing can add up. Microsoft believes its technology could eventually help doctors make better decisions about which tests patients actually need.

During the study, researchers assigned estimated costs to diagnostic procedures. They then measured whether the AI could reach accurate conclusions while avoiding unnecessary testing. In one cost-conscious configuration, MAI-DxO achieved approximately 81% accuracy on complex cases while generating an estimated $2,241 in diagnostic costs per case. The physicians using reference materials averaged approximately $4,725 in estimated diagnostic costs.

However, some configurations designed to maximize AI accuracy generated higher testing costs. The savings depended partly on how researchers adjusted the system. It’s important to understand what those dollar amounts represent. Researchers estimated the cost of diagnostic workups, including medical testing and standardized visit expenses. These figures do not represent actual patient bills or the cost of operating the AI. They’re estimates based on the simulation, so researchers still need to determine whether the savings would hold up in real hospitals. Nevertheless, reducing unnecessary procedures while preserving diagnostic accuracy could offer significant benefits if future clinical trials confirm the results.

Microsoft prepares to test its AI in real hospitals

The next challenge involves moving beyond simulated medical cases. Microsoft has announced a partnership with Beth Israel Deaconess Medical Center in Boston to evaluate MAI-DxO in clinical environments. Under the planned research, consenting patients will continue receiving standard medical care. They will also interact with Microsoft’s AI system as researchers compare its recommendations with physicians’ independent assessments. Importantly, doctors will remain responsible for patient treatment.

The study should help researchers understand how well the AI performs when it encounters actual patients and the unpredictable circumstances of a busy hospital. Meanwhile, Microsoft faces competition from other technology companies pursuing similar goals. Google recently published a study in The Lancet involving its own conversational medical AI system, called AMIE. In that study, 98 patients interacted with Google’s AI before urgent primary care appointments. Physicians monitored the conversations in real time, and none required a safety-related interruption under the study’s predefined criteria.

Participating clinicians also reported that the AI-generated information often helped them prepare for appointments. Although the research was relatively small, it provided an early look at how conversational AI might work alongside doctors. Microsoft’s upcoming clinical evaluation will offer another opportunity to examine whether these systems can perform reliably outside controlled simulations.

What this means to you

For now, Microsoft’s AI diagnostic system remains a research technology. You cannot rely on it as a replacement for a medical appointment. However, these findings offer a possible preview of where health care could be headed. If AI systems continue improving, your doctor might eventually have access to a powerful digital assistant that can review difficult cases and suggest overlooked possibilities. That could be especially helpful for people who have spent months trying to understand unexplained symptoms.

AI might also help clinicians evaluate test results more efficiently. In the future, that could reduce unnecessary procedures and potentially shorten the time patients spend waiting for answers. Still, accuracy alone should never determine whether a hospital adopts AI for patient care. Doctors need to understand when these systems make mistakes. Patients also deserve clear information about how their medical data gets used and who reviews AI-generated recommendations.

We’ve previously reported on how more Americans are turning to AI for health advice. That growing interest makes careful use even more important. If you’re already asking an AI chatbot about symptoms or medical test results, use its responses to prepare questions for your doctor. Avoid treating those responses as confirmed diagnoses. And before uploading medical records into any AI service, review how the company handles sensitive health information. Ultimately, the most useful role for this technology may be helping physicians reach better decisions while keeping patients involved in their care.

Want to put AI to work for your own health? Watch the free replay of my CyberGuy LIVE class, 5 Ways AI Can Help You Get Better Healthcare at cyberguy.com/cyberguylive/classes/, where I share practical ways to prepare for doctor visits and make sense of complicated medical information.

Kurt’s key takeaways

I find these results both exciting and a little unsettling. Anyone who has watched a loved one struggle through a difficult diagnosis knows how frustrating it can be to wait for answers. If AI can help doctors catch something they might otherwise miss, I’m all for it. But let’s be careful about declaring AI better than doctors based on simulated cases. I want to see how this technology performs with actual patients before we put too much trust in it. What interests me most is the possibility of giving doctors another set of highly capable eyes. A second opinion can change everything, and AI could make that available much faster. Of course, protecting sensitive medical information needs to be part of the equation, too.

CLICK HERE TO DOWNLOAD THE FOX NEWS APP

If an AI system disagreed with your doctor about a serious medical diagnosis, whose judgment would you trust? Would you want the AI involved in your care at all? Let us know by writing to us at CyberGuy.com

Get my best tech tips, urgent security alerts and exclusive deals delivered straight to your inbox.
For simple, real-world ways to spot scams early and stay protected, visit CyberGuy.com – trusted by millions who watch CyberGuy on TV daily.
Plus, you’ll get instant access to my Ultimate Scam Survival Guide free when you join.