Exam Scores vs Reality: The 2026 AI Index Report on Medicine

Sam: Did you catch the stat that a multi-agent AI system scored 85.5% on complex published medical case studies, while unaided physicians only scored 20%?

Maya: I did, and it is a fascinating finding to kick things off. Welcome to Smart Summaries. As a quick reminder, our voices are AI-generated, and today we are breaking down a public document. It is the Medicine chapter of the 2026 AI Index Report, published by the Stanford Institute for Human-Centered Artificial Intelligence.

Sam: Those exam scores really pull you in. If I am reading this right, Microsoft's AI Diagnostic Orchestrator, paired with OpenAI's o3 model, was tested against 21 practicing physicians with 5 to 20 years of clinical experience working without their usual tools. The AI hit 85.5% accuracy, and the doctors were at 20%. Why is there such a massive gap?

Maya: It comes down to the difference between structured cognitive evaluations and real clinical practice. The report shows multi-agent frameworks having diagnostic accuracy gains of 7% to over 60% over single-agent baselines. The AI models are phenomenal at parsing text in curated case studies. For example, on the New England Journal of Medicine Healer cases, OpenAI's o1-preview achieved a perfect revised-IDEA score in 78 of 80 cases.

Sam: How did the human doctors do on that exact same test?

Maya: Attending physicians scored 28 of 80, and residents scored 16 of 80. Even GPT-4 only got 47 of 80. And on management reasoning vignettes, the o1-preview model hit a median score of 86%. Physicians using conventional resources scored 34%, and those with access to GPT-4 scored 41%.

Sam: So the AI is undeniably outperforming most physicians on these specific reasoning tasks. But the report clearly states that strong model performance has not consistently translated into real-world clinical impact. Why is that?

Maya: Because real patients do not present as neatly packaged vignettes. The State of Clinical AI Report reviewed over 500 clinical AI studies and found that nearly half of them still rely on simulated scenarios or exam-style questions rather than real patient data. Only 5% of those studies used real clinical data.

Sam: Only 5%? That feels like a massive blind spot for the industry. If you are a hospital buyer, how do you trust a tool that has never been tested on actual patient records?

Maya: You do not, which is why the tools actually gaining traction in practice are the ones that support existing workflows, rather than trying to replace clinician judgment. The NOHARM benchmark is a great example of why caution is needed. It found that leading large language models produced 11.8 to 14.6 severely harmful recommendations per 100 clinical cases.

Sam: That is a staggering error rate for something that could theoretically touch patient care. What kind of harmful recommendations are we talking about?

Maya: The most insidious kind. The report notes that 76.6% of those severe harms were errors of omission. The model simply failed to recommend a critical test or intervention. That is exactly why autonomous AI agents are still strictly in the research phase.

Sam: Right, the report mentions autonomous and semiautonomous AI agents as a major development for 2025 and 2026. A scoping review found 43 studies evaluating agentic AI in healthcare, with 36 of those published just in 2025.

Maya: Exactly. But when you look at a benchmark like MedAgentBench, which evaluates these agents in a virtual electronic health record environment across 300 clinically derived tasks, the best performing model only achieved a task success rate of 69.7%. That is nowhere near ready for the ward.

Sam: So if autonomous diagnostic agents are not ready, what is actually scaling right now in enterprise health systems?

Maya: Ambient AI scribes. These are tools that automatically generate clinical notes from patient visits. In 2025, they saw the broadest adoption of any clinical AI category.

Sam: Let us put some real numbers to that adoption. The platform Abridge expanded from approximately 100 to over 150 health systems. They saw adoption reach 63% among hospitals using Epic's electronic health record system. Kaiser Permanente deployed it across 40 hospitals and more than 600 medical offices.

Maya: The operational metrics from these deployments are exactly what boardrooms are looking for. Northwestern Medicine reported a 112% return on investment. Physicians using the tool in more than half of their encounters saw 11.3 additional patients per month and had a 24% reduction in documentation time.

Sam: And what about the impact on the doctors themselves?

Maya: It is making a measurable difference. Sharp HealthCare reported an 83% reduction in note-writing effort, alongside a 3.5%–6% increase in work relative value units per encounter. The University of Chicago Medicine reported a 47% reduction in cognitive load and a 58% increase in undivided patient attention.

Sam: MaineHealth also reported a 23% reduction in time spent on clinical notes, using the tool in 70.3% of encounters. And a prospective study of 48 physicians at Stanford Health Care found a median time savings of 20 minutes per half day of clinic, with statistically significant reductions in task load and burnout.

Maya: Those numbers prove that AI does not need to make complex diagnoses to be incredibly valuable. Beyond scribes, AI-powered sepsis prediction is another area showing enterprise-scale clinical impact. 2 sepsis prediction systems reported actual mortality reductions in 2025.

Sam: Sepsis is notoriously difficult to catch early. What were the results?

Maya: The Targeted Real-time Early Warning System, developed at Johns Hopkins and used across 13 Cleveland Clinic hospitals, reported an 18.7% relative reduction in sepsis mortality. They achieved a 1.85-hour reduction in median time to first antibiotic order and correctly identified 82% of sepsis cases.

Sam: That is a huge window. An extra hour or two on antibiotics can easily mean the difference between life and death for a sepsis patient.

Maya: Exactly. The adoption rate among clinicians was 89%, and they even saw a 10% reduction in intensive care unit utilization. Another model, COMPOSER at UC San Diego Health, monitors over 150 variables per patient and reported a 17% reduction in sepsis mortality, which is a 1.9% absolute reduction, across 6,217 admissions.

Sam: They also noted a 5% increase in sepsis bundle compliance and estimated 50 lives saved annually just from that one tool.

Maya: We are also seeing generative AI being embedded directly into clinical workflows. A system called ChatEHR generates plain-language summaries of patient records. Within 3 months of its broad rollout, it logged 23,000 sessions across 1,075 trained users. 60% of that usage occurred through automated prompts, and 40% through interactive interfaces.

Sam: Another AI tool aimed at generating plain-language explanations of laboratory, imaging, and pathology results had great feedback. Out of 93 survey respondents, 85% found the tool user-friendly, 72% found it beneficial for laboratory results, and 63% for imaging results. OpenEvidence, a real-time evidence retrieval platform, reported adoption by 40% of U.S. physicians.

Maya: All of this clinical adoption brings us to a really critical point about governance and regulation. The FDA authorized 258 AI medical devices in 2025. That already surpasses all prior full-year totals, bringing the cumulative total to 1,357 authorized devices.

Sam: And 98 new companies entered the space in 2025 alone, following 109 in 2024 and 103 in 2023. Radiology is completely dominating, accounting for 1,039 of those 1,357 devices, which is 76.6%.

Maya: Yes, cardiovascular is second with 130 devices, or 9.6%, and neurology has 61 devices, at 4.5%. But here is the statistic that every hospital system leader needs to pay attention to: a peer-reviewed analysis of 1,016 authorizations found that only 2.4% of devices with clinical studies were supported by randomized controlled trial data.

Sam: Wait, only 2.4%? How are they getting clearance to be used on patients without new clinical trials?

Maya: The vast majority enter the market through the FDA 510(k) pathway. That pathway only requires manufacturers to demonstrate that a new device is substantially equivalent to one already on the market. In 2025 alone, 246 devices were cleared through this specific pathway, relying on existing safety and efficacy evidence rather than new randomized trials.

Sam: The FDA also issued draft guidance in January 2025 on AI-enabled device software functions applying a Total Product Life Cycle approach. And Predetermined Change Control Plans were used in approximately 10% of 2025 clearances.

Maya: But FDA clearance does not automatically equal clinical evidence. The report notes that healthcare systems typically have to complete their own financial clearance and cost-effectiveness justifications before implementing commercial tools. The authorized device market is heavily concentrated at the top. GE Healthcare leads with 93 devices, Siemens Healthineers has 82, Shanghai United Imaging Healthcare has 38, Philips Healthcare has 36, Canon Medical Systems has 35, and Aidoc Medical has 30.

Sam: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Maya: And now, back to the document.

Sam: Yet out of the 626 companies with at least 1 authorized device, the large majority hold only 1 or 2. Let us pivot to medical imaging AI. We know radiology dominates the clearances, but the report points out a major data scarcity issue compared to general-domain AI.

Maya: Right. Training data for medical imaging AI remains roughly 100 times smaller in raw sample count than for nonmedical AI. For context, MAIRA-2 is a radiology-focused model, but its dataset is tiny compared to DINOv3, a general-purpose vision transformer.

Sam: The gap is just as stark for multimodal models when you compare RadFM's medical scans to OpenCLIP's massive image and text pairs.

Maya: Data scarcity is especially pronounced for 3D modalities like CT and MRI. Fragmentation across institutions makes it incredibly hard to develop large-scale medical foundation models. However, we are seeing efficiency breakthroughs. The Merlin model demonstrated that a highly capable CT foundation model could be trained on a single 40GB GPU by leveraging both radiology reports and ICD codes during training.

Sam: That is a big deal for making medical AI accessible in resource-constrained settings. And despite the data challenges, the clinical evidence base in imaging is slowly growing. The number of prospective trials validating medical imaging AI models grew by 28.5% year over year, going from 417 in 2024 to 536 in 2025.

Maya: That growth is crucial because it signals the field is finally moving toward live clinical trials. The report highlights the MASAI study, a randomized screening accuracy study of AI-assisted mammography, and the NOTIFY-1 and NOTIFY-EXTEND trials which tested whether AI flagging early signs of heart disease on routine CT scans led doctors to prescribe more preventive cholesterol medication.

Sam: Let us move upstream from the clinic into the laboratory. The report tracks how AI is used for everything from gene sequences to protein design. AI-driven protein research grew approximately 71% between 2024 and 2025. Total publications rose from 2,259 to 3,855.

Maya: The distribution of that research is shifting. Protein-drug interactions represented the largest share, rising to 54.4% of papers in 2025. Protein structure prediction declined in relative share to 23.9%, function prediction was at 10.4%, and synthetic protein design was at 11.3%.

Sam: The overarching theme in molecular biology this year is a tension between scale and specialization. Smaller or more targeted models actually matched or outperformed much larger, general-purpose systems.

Maya: It is a massive shift. In 2024, the focus was on scaling model size, culminating in ESM3. But in 2025, the focus turned to smaller architectures. MSAPairformer surpassed previous leading methods on the ProteinGym benchmark using a fraction of the parameter budget.

Sam: We saw the same thing with genomic foundation models. GPN-Star, a smaller model focused on functional and regulatory genomics, outperformed Evo 2 on multiple variant effect prediction tasks.

Maya: It proves that in biology, bigger is not always better. A fine-tuned ProGen model was used to design OpenCRISPR-1, a novel CRISPR-Cas protein that demonstrated improved specificity over standard SpCas9. The bottleneck now is clearly data, rather than model architecture.

Sam: Because rapidly collecting new biological data is expensive. Models are increasingly using distilled datasets of AI-predicted structures to supplement experimental data, significantly scaling the size of training sets.

Maya: We are also seeing new experimental datasets debuting. A new single-cell sequencing dataset contains measurements from over 50 cancer cell types exposed to more than 1,100 drugs.

Sam: Another new dataset features a massive repository of genes obtained through metagenomic mining. These datasets fuel cofolding models, which predict 3D structures formed by combinations of proteins, nucleic acids, and drugs.

Maya: AlphaFold 3 and open-source replications like Boltz-2 and OpenFold3 are leading this space. AlphaFold 3's performance on the FoldBench benchmark has yet to be significantly surpassed even though several larger models have been released since.

Sam: The application of this technology in generative protein design is fascinating, but sobering when subjected to real-world tests. Adaptyv Bio hosted a protein design challenge in 2025 to design a binder targeting the Nipah virus.

Maya: Of the designs tested in the challenge, 99 proteins were confirmed to bind, but none neutralized the targeted protein. A specialized method called Mosaic, which combined multiple tools with expert tuning, outperformed general-purpose approaches.

Sam: Are we seeing any AI agents acting autonomously in biomedical research, similar to the clinical diagnostic agents we discussed?

Maya: We are. Robin is an automated discovery framework that linked literature-based hypothesis generation with experimental data analysis. It identified the ROCK inhibitor ripasudil as a novel candidate for dry age-related macular degeneration.

Sam: Multi-agent frameworks are entering the lab too, right? The Virtual Lab uses an LLM Principal Investigator to orchestrate specialized scientist agents, producing 92 novel nanobody binder designs for SARS-CoV-2.

Maya: Yes. Agent Laboratory assigns distinct roles to PhD, Postdoc, and ML Engineer agents within a simulated lab structure. And Biomni, developed at Stanford University, mapped a unified biomedical action space across 25 subfields, integrating 150 specialized tools, 105 software packages, and 59 databases. We are also seeing virtual cell models emerge as a new frontier, like Evo 2, STATE, and DeepMind's AlphaGenome.

Sam: Let us talk about the people at the center of all this healthcare innovation: the patients. AI-generated summaries, which Google calls AI Overviews, now appear at the top of 84% to 92% of health-related Google searches.

Maya: That is a fundamental shift in how people access medical information. Symptom queries and common health questions trigger an AI Overview 92% of the time. Treatment queries trigger them 90% of the time, and condition-based queries hit 84%–88%.

Sam: What do patients actually think about AI being used in their healthcare? Publication volume on patient perspectives grew tenfold between 2020 and 2025.

Maya: The prevailing attitude is conditional acceptance. Patients endorse AI in assistive roles, but they do not want it making autonomous decisions, especially in high-stakes contexts. Trust in AI is heavily clinician-mediated. If their provider endorses it, patients are much more likely to accept it.

Sam: That brings up a lot of ethical considerations. The report notes that ethics discussions in medical AI publications more than doubled, growing from 37.1% of publications in 2024 to 43.4% in 2025.

Maya: But the conversation is incredibly narrow. Governance totally dominates the discourse, with 1,228 publications in 2025. Algorithmic concerns had 896 publications, and societal concerns had 874. Meanwhile, despite all the policy panic about biosecurity, it remains vastly underexplored. Only 14 publications in 2025 discussed biosecurity.

Sam: Global health is an exception to that governance-dominated pattern. Among publications addressing global health in 2025, 51.8% also mentioned ethics topics, which is 100 out of 193 publications. And societal concerns like equity, justice, and accessibility ranked highest in the global health context.

Maya: The geographic distribution of those studies is telling. Europe led with 38 publications, followed by East Asia with 31, and North America with 28. Sub-Saharan Africa, Latin America, and Oceania each produced fewer than 5.

Sam: Before we wrap up, I want to touch on medical digital twins. Patent filings in healthcare digital twins skyrocketed from 30 in 2016 to 4,926 in 2025. Publication counts rose from 0 in 2015 to 372 in 2025.

Maya: The hype is real, but the conceptual clarity is lacking. A scoping review looked at 149 human digital twin studies and found that only 12.1%, or 18 studies, satisfied the strict definition set by the National Academies of Sciences, Engineering, and Medicine. Only 19% of those systems were tested in real healthcare environments.

Sam: But where rigorous trials do exist, the results are promising. There was a randomized controlled trial of 150 diabetes patients using Twin Health's platform. Within 12 months, 71% of participants achieved an HbA1c below 6.5% while safely reducing their blood sugar-lowering medications.

Maya: There was also an 80-patient randomized trial in ovarian cancer using adaptive therapy. It is a space with immense potential, provided we stick to rigorous clinical validation.

Sam: And that brings us full circle. The big takeaway for me is that while AI can ace medical exams and predict protein structures at scale, the true measure of its value is clinical evidence, not just benchmark scores.

Maya: Exactly. Whether you are on the ward or in the boardroom, look for tools that support existing workflows and demand randomized trial data before deploying autonomous systems. That is a wrap for today. You can find a link to the Stanford AI Index Report in our show notes. As always, this podcast is for informational purposes and is not medical advice.