16 November 2026 · 18 min

The RCT Reality Check: Does Healthcare AI Actually Improve Patient Outcomes?

Health Systems & OperationsEvidence & ResearchEconomics & Adoption

This episode unpacks a major systematic review of 54 randomized controlled trials evaluating AI in real-world clinical settings. We explore which types of AI actually improve patient outcomes and why passive workflow alerts often fail. Listeners will learn what the latest data means for clinicians building care pathways and executives deciding which digital health tools to buy.

Key points

Source: Effectiveness of Artificial Intelligence Systems in Healthcare: A Systematic Review of Clinical and Organizational Outcomes - Healthcare (Basel, Switzerland), 2026 (CC BY)

This spot is available. Reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsor the show

This episode is an AI-generated conversation summarising a public document; the hosts' voices are synthetic. It is for information only and is not medical advice. Always refer to the original source.

Authors: Andrea Ceccarelli, Tommaso Montagnani, Tristan Darius Marin, Vincenzo Cassetta, Marco Senni, Marco Montalti et al. · DOI: 10.3390/healthcare14203532

Your company here. This podcast is looking for its first sponsors: reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsorship options and rates →

Transcript

Sam: Wait, really? Out of 54 randomized controlled trials in real-world healthcare settings, 74.1% reported a positive direction of effect for AI?

Maya: Exactly. That is the headline finding from a new document we are unpacking today. It is called Effectiveness of Artificial Intelligence Systems in Healthcare: A Systematic Review of Clinical and Organizational Outcomes. It was published in the journal Healthcare.

Sam: I should mention right here at the start that our voices are AI-generated, and this podcast is a summary of a public document. But Maya, that 74.1% figure feels massive. Hospital executives and health tech developers are desperate for actual trial data, not just marketing pitches.

Maya: Absolutely. This review is the reality check the industry has been waiting for. It looks past the lab tests to see what happens when you actually put AI into the hands of clinicians and patients on the ward.

Sam: So let us talk about what this document actually is and why it exists. The authors conducted a PRISMA 2020-compliant systematic review. They looked at randomized controlled trials published between January 2015 and January 2026.

Maya: Right, and they searched major databases like MEDLINE, CINAHL, Cochrane CENTRAL, and ClinicalTrials.gov. They started with 1773 records and narrowed it down to 54 RCTs. And the inclusion criteria were incredibly strict.

Sam: That is what caught my eye. The AI had to be prospectively and actively deployed as the primary or co-primary intervention. It could not just be retrospective model development or post hoc predictive modeling.

Maya: Exactly. It had to be implemented in a real-world healthcare setting with actual patients. No simulated consultations, no lab-only radiology tests without a clinical component. They wanted to see real clinical or organizational outcomes.

Sam: Because of the sheer variety of the technologies, the authors note that meta-analytic pooling was inappropriate. So instead, they did a structured narrative synthesis. And they classified the AI interventions into 3 functional categories. What were those?

Maya: They broke them down into perceptive AI, predictive or decision support AI, and conversational or adaptive AI. Let us start with perceptive AI. There were 17 studies here. Perceptive AI systems are those that recognize, detect, classify, or quantify clinically relevant patterns from complex unstructured data. Think medical images or physiological signals.

Sam: The gastrointestinal data in this section is wild. 6 of the studies looked at computer-aided detection, or CADe, systems for colonoscopies. And the improvements in the adenoma detection rate were very consistent.

Maya: They really were. A study by Repici et al. in Italy with 685 adults found that real-time CADe increased the adenoma detection rate from 40.4% to 54.8%. Another study in China by Gong et al. saw the rate double from 8% to 16%.

Sam: And Su et al. found an increase from 16.5% to 28.9%. Even in the studies that had mixed results on their primary endpoints, the underlying metrics were still compelling. For example, Ahmad et al. in the UK studied 614 patients. Their primary adenoma detection rate endpoint narrowly missed statistical significance at a p-value of 0.09, but the significant polyp detection rate still jumped from 71.6% to 79.2%.

Maya: We also saw Wallace et al. evaluate AI-assisted tandem colonoscopy across 230 subjects in a multicenter setting. They demonstrated a significantly lower adenoma miss rate in the AI-first arm, specifically 15.5% versus 32.4%.

Sam: So perceptive AI clearly works for image detection. The review also looked at cardiovascular settings. Halcox et al. studied 1000 patients and found that an AI-based handheld ECG device increased new atrial fibrillation diagnoses with a hazard ratio of 3.9 compared to routine care.

Maya: But perceptive AI is also where we see the first major caveat of this document, which is human behavior. Let us talk about the Blomberg et al. study from Denmark. They looked at 5242 emergency calls to a medical dispatch center.

Sam: The machine learning model could identify out-of-hospital cardiac arrest with 85% sensitivity. That sounds like a massive success on paper.

Maya: Technically, yes. But the primary endpoint was the dispatcher recognition rate. That rate was 93.7% in the AI group versus 90.5% in standard care, which the authors classified as a neutral result because the p-value was 0.15. The human operators frequently disregarded the AI alerts, negating the potential benefit.

Sam: That is such a critical lesson for people building these systems. If the human operator does not trust the alert, or ignores it, the clinical outcome simply does not change.

Maya: Other perceptive AI tools showed mixed or neutral results too. He et al. evaluated a 3D pose estimation system for remote rehabilitation in 75 older adults with sarcopenia. It was non-inferior to in-person training across all functional outcomes, but did not demonstrate superiority, yielding a neutral result.

Sam: But then you have Huan Hwa Chen et al. reporting that an AI-assisted medication adherence monitoring system for 94 participants with schizophrenia significantly improved adherence to 94.72% versus 64.43% in routine care. And Xiao et al. showed significantly greater reductions in pain with AI-guided home telerehabilitation for 38 patients with chronic low back pain.

Maya: We also saw a study by Sharma et al. in the UK with 396 adults looking at tele-ophthalmology for glaucoma referrals. It reported a 59% reduction in false-positive urgent referrals versus standard care, although the AI model's sensitivity and specificity were actually lower than those of human experts.

Sam: That perfectly transitions us to the second functional category: predictive and decision support AI. There were 15 studies here, and the results were highly dependent on workflow integration.

Maya: When it works, it really works. A study by Dean et al. looked at 6848 adult patients arriving at the emergency department with pneumonia. When they used a clinical decision support system, or CDSS, the group saw a massive reduction in 30-day all-cause mortality, with an adjusted odds ratio of 0.62. They also saw a 17% increase in direct home discharge.

Sam: Another powerful example is Manz et al., who studied 20506 cancer outpatients. They used machine learning to trigger behavioral nudges for clinicians to initiate serious illness conversations. The rate of those conversations jumped from 3.4% to 13.5%.

Maya: And in terms of organizational efficiency, Sadeh-Sharvit et al. showed that an AI behavioral intervention platform for 47 adults reduced therapist administrative time from 69 to 14 hours, without compromising clinical outcomes. That is the dream scenario for hospital administrators looking for real return on investment.

Sam: Even for medical education, Fazlollahi et al. studied 70 medical students with no prior surgical experience. The AI-based surgical tutoring significantly outperformed remote human expert instruction with an expertise score improvement of 0.66 points.

Maya: But here is the catch. The document notes a stark contrast between AI that is structurally embedded into clinical workflows, like those examples, and AI that just delivers passive alerts.

Sam: Alert fatigue is a massive issue. Let us look at the study by Wright et al. They analyzed 288832 clinical opportunities across multi-site academic medical centers. The CDSS identified 4.6 times more missing diagnoses than standard care.

Maya: That is incredible technical performance. But the alert acceptance rate was only 22.1%. Because the overall alert acceptance remained low, they saw no significant improvement in downstream quality measures.

Sam: We see the exact same thing in the Harry et al. study. They looked at 5054 young adults due for the HPV vaccine. The electronic health record triggered a CDSS alert. Did it increase vaccination completion? No. The rate was 2.3% in the intervention group versus 2.2% in the control group. A completely neutral result.

Maya: Fritz et al. found a similar pattern with 5071 surgical patients and 89 clinicians. Providing machine learning risk scores for 30-day mortality did not significantly improve clinician predictions. The AUROC for mortality prediction was 0.793 for the AI-assisted group versus 0.780 for the unassisted group.

Sam: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Maya: And now, back to the document.

Sam: And Seol et al. looked at 184 children with asthma. The CDSS caused a dramatic reduction in physician chart review time, dropping from 11.3 to 3.5 minutes, but it produced no significant change in asthma exacerbations.

Maya: The document explicitly states that presenting AI-generated risk scores as additional information, without workflow integration, may not yield improved decision-making. Improved information generation and improved healthcare outcomes should not be considered interchangeable endpoints.

Sam: I mean, that is the defining takeaway for any health-system leader right there. If you are buying an AI tool, you have to ask how it integrates into the workflow. If it is just another pop-up window, the trial data shows it probably will not work.

Maya: And sometimes the AI simply performs poorly. Knitza et al. studied 600 patients awaiting consultation at rheumatology centers. They tested two AI-based triage tools for inflammatory rheumatic diseases. The diagnostic accuracy concordance was just 58% and 52% compared with physician diagnosis, which was the only study in the review to report a negative direction of effect.

Sam: That brings us to the largest subgroup in the review: conversational and adaptive AI. There were 22 studies in this category, and it showed the most consistent positive signal.

Maya: 19 of the 22 studies reported a positive direction of effect. These are systems that interact directly with patients to deliver education, coaching, or behavioral support. They proved particularly effective for chronic disease management and therapeutic adherence.

Sam: For example, Turino et al. studied 60 patients with severe obstructive sleep apnea in Spain. AI-assisted remote monitoring increased CPAP compliance by 1.14 hours per day without additional costs.

Maya: Nayak et al. looked at 32 adults with type 2 diabetes adjusting their basal insulin. The AI-guided titration helped them reach their optimal dose in 15 days, compared to more than 56 days in standard care. And 81.3% reached glycemic control versus just 25.0%.

Sam: Aguilera et al. tracked 168 adults with diabetes and depression. An adaptive messaging system increased their daily step count by 19%, compared to just a 3.9% increase in the control group.

Maya: It seems like these conversational agents excel at providing continuous support between formal doctor visits. Piette et al. demonstrated this perfectly. They studied 439 adults with chronic back pain. They found that AI-supported cognitive behavioral therapy using reinforcement learning was non-inferior to standard telephone-delivered CBT. But it required approximately 50% lower therapist time.

Sam: That is a massive organizational win. The patients get the same pain reduction, measured at 3, 6, and 12 months, but it frees up half the therapist's time.

Maya: Mental health and smoking cessation also saw strong results. Carrasco-Hernandez et al. ran a trial with 240 adult smokers in Spain. The per-protocol smoking abstinence rate at one year was 64.7% with the AI chatbot, compared to 40.0% with standard counseling.

Sam: Olano-Espinosa et al. did a similar study in primary care with 513 adult smokers. The abstinence rate at 6 months was 26.0% versus 18.8%. Still a statistically significant benefit.

Maya: We also saw positive results for anxiety and substance use. Prochaska et al. studied 180 adults with positive screening for substance misuse. The AI intervention group saw a reduction of 9.1 occasions of substance use in the past month, versus a reduction of 3.3 in the waitlist group.

Sam: Sharp et al. looked at 60 patients on a waitlist for eating disorder treatment. In-person treatment initiation was 93% for the chatbot group versus 70% for the control group. And Shade et al. found that older adults with chronic pain had morning routine initiations of 74% with an AI voice assistant compared to 58% with a standard one.

Maya: But not every conversational AI study was a landslide victory. Mathioudakis et al. evaluated an AI-powered lifestyle intervention versus human coaching for 368 adults with prediabetes. They demonstrated noninferiority on the composite primary endpoint, reaching 31.7% in the AI arm versus 31.9% in the human coach arm, yielding a neutral result.

Sam: And that is a major limitation we need to discuss for this category. The document explicitly warns that the positive signal here must be interpreted cautiously because of the risk of bias.

Maya: Exactly. The authors used the Cochrane Risk of Bias 2 tool to assess the trials across 5 domains. Across all 54 studies, 16 were rated as having a low risk of bias, 26 as having some concerns, and 12 as having a high risk.

Sam: But the high-risk studies were heavily concentrated in the conversational AI group, right?

Maya: Yes. Of the 22 included conversational and adaptive AI studies, 9 were rated as having a high risk of bias. These studies often had open-label designs because you cannot blind a patient to the fact that they are texting a chatbot. They also relied heavily on self-reported outcomes and had short follow-up periods.

Sam: Let us talk more about the organizational outcomes across this entire review. The document notes that they are substantially underrepresented.

Maya: That is an understatement. Out of 54 studies, only 3 reported exclusively organizational outcomes. 17 reported mixed outcomes spanning both clinical and organizational domains. The other 34 focused entirely on patient-level clinical outcomes.

Sam: Even in the mixed outcomes, we see strange trade-offs. Sibley et al. evaluated 51 adolescents with ADHD. The AI-assisted group saw more efficient treatment delivery, but fewer appointments attended and a significant decline in motivational interviewing quality over time.

Maya: And when it comes to the bottom line, the financial impact, there is almost no data. The review states that cost data were reported in only 2 studies. Halcox et al. and Turino et al. Neither demonstrated a statistically significant cost advantage.

Sam: This is a critical gap. The authors note that the scalability and sustainability of AI depend ultimately on whether clinical benefits can be achieved without generating disproportionate implementation costs. Right now, there is insufficient evidence to draw conclusions about cost-effectiveness.

Maya: There are also some major geographical limitations to this data. The evidence is heavily concentrated. 17 studies were from the USA, and 8 were from China. This limits the ability to transfer these conclusions to other healthcare systems, especially publicly funded systems in Europe or systems in low- and middle-income countries.

Sam: Another limitation is the duration of these trials. The document notes that the median follow-up period across all included studies was approximately 12 weeks. We simply do not know the long-term safety or whether these behavioral benefits are sustained over years.

Maya: Also, the comparators varied wildly. Some trials compared the AI to a waiting list, while others compared it to active, human-delivered treatments. That makes it very difficult to compare effect sizes across different studies.

Sam: Maya, as a former clinician, what is your most memorable takeaway from this systematic review?

Maya: It is the authors' focus on contextual effectiveness. Technical performance, like a model that predicts cardiac arrest with 85% sensitivity, does not automatically equal real-world effectiveness. The AI only works if the humans using it accept it, trust it, and have it smoothly integrated into their daily workflow.

Sam: Mine is the alert fatigue data. The fact that a system can find 4.6 times more missing diagnoses, but doctors only accept 22.1% of those alerts, shows that we have a massive user interface and workflow problem in healthcare AI, not just an algorithm problem.

Maya: To recap our deep dive: A review of 54 randomized controlled trials found that 40 studies, or 74.1%, report a positive direction of effect for AI in healthcare settings. Perceptive AI shines in diagnostics, conversational AI excels in behavioral support, and predictive AI relies entirely on workflow integration.

Sam: You can find a link to the full document in our show notes. And as always, a reminder that this podcast is a summary of a public document and is not medical advice. Thanks for listening to AI in Medicine - Smart Summaries.