Beyond Time Saved: The Hidden Risks of Evaluating AI Scribes on Efficiency Alone

Maya: Across 6 independently peer-reviewed, real-world implementations of ambient AI scribes, not a single one systematically evaluated if these tools actually preserve the quality of the clinical narrative or patient communication.

Sam: That is a massive blind spot for health system leaders. Before we dive into exactly why evaluating these tools solely by minutes saved is a dangerous trap, we need to remind everyone that our voices are AI-generated. This podcast is a summary of a public document, and nothing we say is medical advice.

Maya: Today we are unpacking a viewpoint on the comprehensive evaluation of AI clinical documentation, published in JMIR medical informatics in 2026. The author argues that focusing just on efficiency leaves domains central to patient safety entirely unmeasured.

Sam: It is easy to see why hospital executives are laser-focused on efficiency right now. Clinician burnout is an absolute crisis. We hear about it all the time, but the numbers in this document are still shocking.

Maya: They are. Physicians spend approximately 2 hours on electronic health record tasks for every hour of direct patient care. That work often extends 1 to 2 hours into the evenings. This burden links directly to patient safety and contributes to significant annual costs from turnover and reduced clinical hours.

Sam: So it makes total sense that when ambient AI scribes came along promising to automate the clinical note, health systems jumped at them. I mean, the early vendor claims and pilot studies made it sound like a miracle cure for burnout.

Maya: Initial reports definitely described substantial efficiency and well-being gains. For example, the document highlights a Kaiser Permanente deployment across more than 7000 physicians. That implementation reportedly saved nearly 16,000 documentation hours over 10 weeks, with 70 percent of users reporting reduced burnout.

Sam: Okay, saving 16,000 hours in 10 weeks sounds incredible. But the author notes that more recent evidence is more measured, right? What happens when you look at the broader picture beyond those early single-system reports?

Maya: The picture gets complicated. A multisite study found only modest decreases in total electronic health record time and documentation time, alongside a modest increase in weekly visit volume. Furthermore, a systematic review and meta-analysis of 23 studies demonstrated a moderate pooled reduction in documentation workload across 14 studies.

Sam: So we have a tool that saves some time, but maybe not as much as the initial hype suggested. Still, if it is saving any time at all, why is the author so concerned? What is the trap for hospital leaders here?

Maya: The trap is that current evaluation frameworks focus almost entirely on those efficiency metrics. To illustrate this, the author conducted an analytic review mapping the evaluation domains of a purposive sample of 6 prominent, publicly reported ambient AI scribe implementations published through April 2026.

Sam: And these are real-world implementations, not just lab simulations, right?

Maya: Exactly. They included studies from Kaiser Permanente, a multicenter ambulatory study, a large academic medical center at the University of Rochester, and an Australian allied health private practice. Across all 6 studies, evaluation consistently measured documentation time, after-hours work, and clinician satisfaction.

Sam: Which is great for the baseline. But what did they fail to measure?

Maya: None of them reported systematic assessment of patient-clinician communication using validated instruments. None used structured comparison of AI-generated against human-generated notes for narrative fidelity. None did longitudinal evaluation of cognitive effects on trainees, and none ran equity analyses stratified by language and demographic group.

Sam: Wait, really? They are rolling these out to thousands of doctors, fundamentally changing how a medical visit is documented, and nobody is checking if the actual story of the patient is being captured accurately or equitably?

Maya: To be fair, the author notes that the absence of public reporting in this sample does not prove that no organization has measured these dimensions internally. But as far as the independently peer-reviewed, published evidence goes, there is a massive systematic measurement gap.

Sam: Let us break down the 4 evaluation domains the author proposes investigating. The first one is communication. I can imagine that putting a recording device in the room changes the dynamic. Some patients might hold back.

Maya: That is a plausible pathway. If patients become guarded in disclosure, or if clinicians reduce empathic responses knowing they do not need to be documented, communication degrades. Conversely, if clinicians are freed from the mental load of concurrent documentation, they might provide more attentive listening. The problem is we just do not know at scale.

Sam: I know some implementations survey patients and doctors about this. Is not it enough if the doctor says they feel more present or the patient gives a thumbs-up on a satisfaction survey?

Maya: The document explicitly argues that subjective, self-reported impressions are not a substitute for rigorous assessment. Clinician and patient ratings of communication often show poor agreement. A general satisfaction question does not capture the subtle communication dimensions that predict treatment adherence, health outcomes, and malpractice risk.

Sam: So how should health systems actually measure this without bogging everyone down in red tape?

Maya: The tools already exist. The author suggests using multi-item patient-reported scales like the Communication Assessment Tool, which is a 14-item measure that gives visit-level clinician feedback. Or they can use the Consumer Assessment of Health Care Providers and Systems Clinician and Group communication composite, commonly known as CAHPS CG.

Sam: And the CAHPS CG is what hospitals already use for system-level benchmarking, right? So they could just bolt this onto their existing infrastructure.

Maya: Exactly. The baseline feasible approach is simply adding a single patient-reported communication item to existing patient experience surveys. The ideal approach is an interrupted-time-series design at the clinic level, adjusting for temporal trends, clinician, and patient case mix.

Sam: Alright, that covers communication. Let us move to the second domain: narrative preservation. What exactly is a clinical note losing when an AI summarizes it? Is not it just stripping out the fluff?

Maya: Clinical documentation preserves more than discrete biomedical facts. It captures temporal symptom patterns, patient explanatory models, and psychosocial context. An instrument-validation study comparing AI-generated drafts against reference standards actually found errors and frequent omissions, noting that accuracy degrades as clinical complexity increases.

Sam: That is concerning. If the AI is dropping the psychosocial context, subsequent providers might miss out on why a patient is actually struggling to adhere to a treatment plan. How do we audit for that kind of narrative fidelity?

Maya: The starting point is the Physician Documentation Quality Instrument, or PDQI-9. It assesses note quality across 9 dimensions, including up-to-date accuracy, thoroughness, usefulness, organization, comprehensibility, succinctness, synthesis, and internal consistency.

Sam: Are those 9 dimensions enough to catch what the AI might be missing?

Maya: Not entirely. The author notes that PDQI-9 fits best for outpatient consultation notes but does not adequately capture the narrative-specific features most at risk under automation. It requires validated extensions for temporal coherence, patient explanatory model, psychosocial context, and care-coordination utility.

Sam: And how do you even set the baseline? Manual documentation by humans is notoriously flawed anyway. We omit things and copy-paste old data all the time.

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: That is a crucial point. The relevant question is not whether AI matches an idealized transcript, but whether it preserves the narrative as well as the manual documentation it replaces. The framework suggests comparing AI output against pre-AI manual documentation by the same clinician, peer-clinician documentation, and verbatim transcripts to isolate the AI-specific failure modes.

Sam: This leads perfectly into the third domain: clinical reasoning and cognitive engagement. If the AI writes the note, the clinician still signs it. Does it really matter if the AI drafted it as long as the doctor reviews it?

Maya: It matters because of documentation automation bias. Reviewing a note happens under severe time pressure. Clinicians might accept an AI-generated narrative that silently restructures the encounter, which then shapes downstream reasoning, coding, and orders. Plausible failure modes include a missed red-flag symptom in the history of present illness or an incorrect speaker attribution that quietly redirects the differential.

Sam: Do we have proof that doctors will just blindly sign off on these errors?

Maya: We have evidence from the clinical decision support literature. For instance, an empirical study showed clinicians overrode correct initial judgments in approximately 5% of prescribing cases when presented with erroneous automated advice. Furthermore, recent evidence found physicians using ChatGPT performed no better than those without AI access, despite ChatGPT alone outperforming physicians on the same cases.

Sam: Because they were only engaging with the AI output superficially. They weren't really thinking critically about it. I can see how that becomes a massive problem for medical residents and trainees who are still learning how to think through a diagnosis.

Maya: Exactly. Documentation is where clinical reasoning crystallizes from observations into coherent assessments. The author warns of skill degradation. To evaluate this, the framework proposes longitudinal trainee cohort studies. You compare programs recently adopting AI scribes with matched programs delaying implementation.

Sam: How do you test if a resident is actually losing their ability to formulate a clinical narrative?

Maya: Through periodic AI-off probes. You literally withdraw the AI assistance and test them using standardized cases to see if their diagnostic-reasoning performance and independently generated documentation quality decline when the crutch is removed.

Sam: That makes total sense. Now, let us tackle the fourth domain, which is equity. The document points out that if we do not evaluate this carefully, these tools might work really well for some patients and terribly for others.

Maya: Right. Algorithmic bias in speech recognition is well documented. The author cites a flagship analysis by Koenecke et al that reported substantial racial disparities in word error rates across the 5 major commercial automatic speech recognition systems.

Sam: And the scribe pipeline relies entirely on that automatic speech recognition as its first step. If it mishears a patient with a nonstandard accent, that error cascades into the final note.

Maya: Exactly. The downstream note reflects compounded effects from speech recognition, speaker diarization, and summarization. Most published AI scribe evaluations have examined primarily English-speaking patients in straightforward encounters. Evidence is largely absent for performance in multilingual or interpreter-mediated encounters.

Sam: The document mentions Asian health care contexts specifically, right? How does the AI handle something like that?

Maya: The author notes that in contexts like Singapore, multilingual encounters involving Mandarin, Cantonese, Tamil, Malay, and English code-switching represent normative practice. The AI has to accurately attribute speaker identity among multiple family members, distinguish patient speech from an interpreter, and handle mid-sentence language switching.

Sam: That sounds incredibly complex for an algorithm. If it fails there, the doctor has to spend so much time revising the note that the efficiency benefit just vanishes for those specific populations.

Maya: Precisely. That is why the framework demands stratified equity analyses. Implementing organizations serving diverse populations need to report word error rates, speaker attribution accuracy, and clinical concept omission rates stratified by patient primary language, interpreter status, and demographic characteristics.

Sam: How do they get a large enough sample to actually measure that?

Maya: The minimum specification is deliberate oversampling, targeting at least 30% non-English encounters where the catchment area supports it, rather than just sampling proportionate to frequency in organizational caseloads. Then, blinded expert raters must audit the AI-generated documentation against verbatim transcripts and pre-AI manual notes.

Sam: Okay, so we have 4 major domains: communication quality, narrative preservation, cognitive engagement for trainees, and equity. That is a lot for a hospital to take on. Does the author offer practical guidance for a clinic manager or a doctor just trying to survive their shift tomorrow?

Maya: Yes, there are lighter-weight monitoring tactics for everyday practice while formal research matures. The first is periodic side-by-side comparison. Clinicians can sample a small proportion of their own encounters, specifically targeting 1 in 20 notes, and write a brief independent version before reviewing the AI output to catch systematic omissions.

Sam: That is smart. Just write 1 in 20 yourself to keep your own skills sharp and calibrate the AI. What else?

Maya: Targeted spot-checking for encounters with high information density or psychosocial weight. Things like disclosures of significant diagnoses or complex shared decision-making involving family members. These are the encounters where omissions matter most and where AI performance is least studied.

Sam: And for smaller practices that cannot run massive statistical analyses?

Maya: They should participate in pooled sentinel surveillance networks convened by specialty societies. Member practices contribute anonymized samples of AI notes for centralized review to generate system-level evidence.

Sam: What about the regulators? It feels like relying entirely on hospitals to voluntarily run complex equity and narrative audits is wishful thinking. Does the FDA or anyone else step in here?

Maya: The viewpoint argues that current regulatory frameworks treating ambient AI scribes primarily as administrative tools may be inadequate. Documentation preserves diagnostic reasoning and provides the medical record subsequent decisions depend on. The author suggests vendors should be required to publish stratified predeployment performance reporting.

Sam: So the vendors would have to show their error rates broken down by demographics and languages before they deploy, rather than just claiming high accuracy overall.

Maya: Correct. Along with postmarket monitoring reporting revision burden and signal-level performance changes following model updates. Regulatory instruments like the United States Food and Drug Administration software-as-a-medical-device pathways, the European Union AI Act, and Singapore's Health Sciences Authority guidance could enforce these expectations at the vendor level.

Sam: It makes total sense. We cannot let the desperate need to solve the burnout crisis blind us to the downstream effects on the medical record and patient safety. Saving time is the first step, but it cannot be the only metric of success.

Maya: To recap: Ambient AI scribes are reducing documentation time, but the evidence base has limitations. Health leaders must expand evaluation beyond efficiency to include validated communication measures like CAHPS, narrative audits like PDQI-9, AI-off probes for trainees, and equity analyses oversampling multilingual encounters.

Sam: The source document is linked in the show notes, and a reminder that this is not medical advice. Thanks for listening, and see you next time.