20 November 2026 · 19 min

Who Gets Sued When an AI Scribe Hallucinates? The Hidden Liability of Automated Medical Notes

Generative AI & LLMsRegulation & PolicySafety, Ethics & Liability

AI scribes are rapidly entering hospitals to save clinicians from documentation burnout, but who is legally responsible when the software hallucinates a diagnosis or drops a critical negation? This episode unpacks a PLOS Digital Health analysis of 74 official regulatory documents governing AI scribes. We explore the drafting fiction that shifts liability onto doctors, the biometric privacy risks of voice data, and why hospital leaders must rethink protected review time before deploying ambient tools.

Key points

Source: Regulating the drafting fiction: A qualitative content analysis of official guidance and regulator documents on AI scribe adoption and use in healthcare - PLOS Digital Health, 2026

This spot is available. Reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsor the show

This episode is an AI-generated conversation summarising a public document; the hosts' voices are synthetic. It is for information only and is not medical advice. Always refer to the original source.

Your company here. This podcast is looking for its first sponsors: reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsorship options and rates →

Transcript

Maya: When an AI scribe listens to a consultation and drafts a note, regulators are quietly pushing all the legal risk for its mistakes straight onto the clinician. Today, we are unpacking a document called Regulating the drafting fiction: A qualitative content analysis of official guidance and regulator documents on AI scribe adoption and use in healthcare, published in PLOS Digital Health.

Sam: And before we dive in, a quick reminder that our voices are AI-generated, and this is a summary of a public document. But Maya, that hook is exactly what every doctor and hospital leader is worried about right now. Regulators pushing the risk onto clinicians?

Maya: Exactly. The study introduces a concept called the drafting fiction. It is the single most interesting finding in the paper, and it completely reframes how we should think about AI scribes on the ward and in the boardroom. The author, Samuel Oluwatobi Atiku, argues that while the technology is moving fast, the governance is highly scattered, and it all relies on treating the AI output as a mere draft that the clinician has to authenticate.

Sam: That makes sense on the surface. You want the doctor to check the work. But how did the author actually figure out this regulatory landscape? Because it sounds like a maze.

Maya: It is, and the methodology of this paper shows exactly why this is so complex. The author conducted a purposive, staged grey-literature search. They initially screened 100 candidates. From that 100, they retained 38 and excluded 62.

Sam: Why did they throw out 62 documents? Were they just irrelevant?

Maya: Not irrelevant, but they lacked regulatory authority. For example, 20 of the excluded documents were privacy notices published by individual general practices, like High Glades Medical Centre and Claremont Surgery. Those describe a local product, they do not issue governance. Another 13 came from membership bodies like the Royal College of General Practitioners, the British Medical Association, and the Royal Australasian College of Physicians. They advise members, but they hold no regulatory authority.

Sam: That makes sense. Who else got excluded?

Maya: They also excluded 8 academic journal publications from places like npj Digital Medicine and JMIR, 7 from charities or consultancies, 5 from universities like McGill and Duke, and 3 slide decks.

Sam: Okay, so they were extremely strict about only including official, authoritative documents of record. What happened after they got those initial 38?

Maya: They did supplementary searching and source chasing to find under-represented domains like privacy and medical-device regulation, adding 39 further documents. That brought the combined set to 77 files. During final reconciliation, they removed 3. 1 was a French-language document with no official English version, and 2 were duplicate copies. So they landed on a final analytic corpus of exactly 74 unique English-language official documents.

Sam: 74 authoritative documents. How did they categorize them?

Maya: They found 13 direct AI-scribe documents, which explicitly addressed AI scribes or ambient scribing. The other 61 were adjacent governance documents. Those govern mechanisms central to safe use, like data protection or professional duties, even if they do not explicitly name AI scribes. The author mapped all of this across 16 governance dimensions.

Sam: What kind of dimensions are we talking about?

Maya: Things like patient transparency and consent, draft status and clinician verification, professional accountability, organizational governance, vendor assurance, privacy and clinical safety, medical-device classification, human oversight, monitoring and incident reporting, equity and training, and retention and limits on use. These 16 dimensions generated 17 deductive codes for the analysis.

Sam: That is an incredibly wide net. How did they process all that text?

Maya: With a very strict re-audit. After the initial coding pass, they had 714 positive document-level assignments. But they went back and re-read every single one against the source PDF. They removed 344 assignments that were just generic principle statements or keyword hits, leaving exactly 370 retained positive assignments. Every single one of those 370 has a located supporting passage.

Sam: That is an incredibly rigorous process. Let us get into the findings. What exactly is this drafting fiction you mentioned at the start?

Maya: The drafting fiction is an author-developed analytic construct. It is not terminology used by the issuing organizations themselves. It describes a recurrent cross-document pattern where AI-generated clinical text remains provisional until clinician review and authentication. Different organizations attach completely different consequences to that review.

Sam: So it is a draft, but different regulators care about that draft for different reasons?

Maya: Exactly. Patient leaflets frame the draft as reassurance. A leaflet from Royal Devon tells patients that clinicians will always review the notes to keep the record accurate. North East London states that the AI scribe does not add anything directly to the medical record. So for the patient, the doctor checking it is a safety promise.

Sam: And how do privacy regulators view it?

Maya: Ontario uses it as a record-control rule. They require AI-generated personal health information to be reviewed for accuracy before entry into an electronic medical record or electronic health record. They even require technical functionality to flag content that has not yet been reviewed and to prevent disclosure until verification occurs. Shared Health Manitoba adds a traceability rule, directing that AI-generated notes should be labeled with words such as Generated by the named scribe system.

Sam: So privacy and system admins want a hard boundary before the data gets saved. Where does the liability come in?

Maya: That comes from the adjacent professional guidance. The Medical Council of New Zealand states that AI is not a substitute for clinical judgement. The College of Physicians and Surgeons of Manitoba requires registrants to verify and validate generated content before it enters the record. The Australasian Institute of Digital Health takes a very strict route, stating the AI scribe is not responsible for the notes and the clinician must confirm that the information is true, accurate, and complete.

Sam: Wow. So the patient leaflet says, do not worry, the doctor checks it. The privacy regulator says, do not save it until the doctor checks it. And the professional regulator says, if there is a mistake in it, it is the doctor's fault because they checked it.

Maya: Precisely. The drafting fiction shifts the risk away from the system and straight onto the downstream clinician. It treats clinician review as the magical step where a probabilistic machine output suddenly becomes an accountable professional record.

Sam: But doctors are already drowning in documentation. That is why they want the AI scribes in the first place! Are these documents giving doctors protected time to actually do this true, accurate, and complete review?

Maya: That is the core problem. The author points out that verification is treated as the central safeguard in direct AI-scribe guidance, but the conditions for that review are totally underspecified. Few direct documents state how long careful review should take or how clinics should preserve that time. The Australasian Institute of Digital Health even notes that some practitioners speak slower and clearer so that the scribe captures information accurately.

Sam: Wait, really? So the technology is supposed to save time, but doctors are altering their own speech patterns during the consultation just to accommodate the machine?

Maya: Yes. And then at the end of the visit, they have to check the output. And we are not just talking about catching typos. The Australian Commission on Safety and Quality in Health Care warns that ambient AI scribes can fabricate diagnosis, and they can omit or add treatment steps. They specifically point out that AI can confuse words such as medication and mediation.

Sam: Confusing medication and mediation? That could completely derail a patient's care plan.

Maya: Exactly. And a board proposal from Hywel Dda University Health Board narrows the risk down to a subtle but dangerous documentation error. They warn of reversals such as does and does not.

Sam: Missing a not changes the entire meaning. Does the patient have a history of heart disease, or does the patient not have a history of heart disease. That is terrifying.

Maya: It is. Ontario warns that hallucinated errors may propagate if they are not corrected through human oversight. And NHS England warns about a behavioral risk called overreliance or automation bias. The College of Physicians and Surgeons of Manitoba similarly warns about complacency. When the machine is usually right, your brain stops looking for the one time it drops a does not.

Sam: But surely once a doctor uses a specific software for a few months, they learn its quirks, right? It becomes predictable?

Maya: You would think so, but a testing report from the South West Healthcare Centre of Digital Excellence shatters that idea. They tested ambient voice technology software packages and found errors in summaries across all of them. More importantly, they observed that supplier accuracy was not consistent over time because underlying algorithms are regularly updated. The verification target is constantly moving.

Sam: So the model behavior can change after an update, and suddenly it is making new types of errors, and the doctor is still legally on the hook to catch every single one of them.

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: Yes. The study argues that guidance cannot make verification the central safeguard while treating the time, attention, and stress of verification as negligible. If organizations do not design appointments and workloads around that checking, the draft label does less to govern the AI and more to just make clinicians answer for errors the system helped create.

Sam: Let us pivot to privacy. If an AI is sitting in the room listening to a highly sensitive medical conversation, how are regulators protecting that data?

Maya: NHS England and NHS Scotland frame privacy assessment as a deployment gate. NHS England links ambient scribing to DCB0160 documentation, a safety case, a hazard log, monitoring frameworks, and a Data Protection Impact Assessment. NHS Scotland requires a manufacturer safety assessment and a System Security Plan alongside MHRA approval. Saskatchewan explicitly asks whether a privacy impact assessment has been completed.

Sam: What about data retention? How long does the AI company get to keep the transcript?

Maya: It varies. Manitoba tells adopters to configure retention to 30 days. NHS Scotland states that patient data from clinical sessions should be automatically deleted unless retention is legally or operationally required. Royal Devon reassures patients that information is processed in a fully secure environment and automatically deleted after documentation is complete.

Sam: That sounds fairly standard. Keep it secure, delete it when you are done.

Maya: But Ontario brings up a risk that patient leaflets cannot govern on their own. Voice data. They state that it may be virtually impossible to de-identify voice recordings because voiceprint and speaker-recognition technologies can infer identity and personal characteristics.

Sam: Wait. Because my voice itself is a biometric identifier. So even if they scrub my name from the transcript, they still have my voiceprint.

Maya: Exactly. The spoken consultation is biometric and inferential data. The issue is whether voice, transcript, metadata, or derived features can be linked to identity, reused in model improvement, profiled in future systems, or combined with other data. Saskatchewan explicitly warns about function creep where vendor capability changes over time, and they warn about contract terms that permit secondary use or disclosure to third parties.

Sam: So how do hospitals protect themselves from that function creep? Just aggressive vendor contracts?

Maya: Contracts help narrow product-level uncertainty, but they have limits. Manitoba introduces a coordination mechanism, noting that Canada Health Infoway released a list of 9 approved AI scribe vendors after a competitive pre-qualification process. They strongly advise providers to use endorsed tools reviewed for privacy and security.

Sam: Using one of the 9 pre-qualified vendors seems like a smart shortcut for a hospital.

Maya: It is, but Ontario limits that comfort. They state that even when a vendor-of-record programme is used, every custodian must still exercise its own due diligence. Saskatchewan warns that many Canadian AI scribe contracts are drafted for federal privacy law rather than Saskatchewan's specific provincial requirements. But the bigger problem is that contractual control cannot substitute for encounter-level reliability. A contract can demand security, but it cannot prove that a generated note for a given patient captured a negated symptom correctly.

Sam: Right. The contract protects the hospital from a data breach, but it does not protect the doctor from a hallucinated diagnosis in a specific patient's chart.

Maya: Exactly. Product assurance does not guarantee the reliability of a particular note.

Sam: Are these AI tools considered medical devices? I mean, they are generating clinical documentation.

Maya: This is where we see a fascinating functional boundary. NHS England states that products that solely generate easily verified text transcriptions are likely not medical devices. However, generative AI used for summarisation is high functionality and likely qualifies as a medical device.

Sam: Because summarization requires the AI to interpret data and make clinical choices about what to include or exclude.

Maya: Exactly. NHS Scotland takes a firmer adoption stance, stating that ambient voice technology solutions that generate summarisation should be at least Class I, and often Class II. Adjacent device regulators like the FDA, Health Canada, and the Medicines and Healthcare products Regulatory Agency place machine-learning software inside quality management and lifecycle control.

Sam: What does lifecycle control mean in this context?

Maya: It means you have to plan for how the software changes after it is deployed. The FDA has guidance on predetermined change control plans, which formalize how future AI-enabled device changes may be anticipated through a modification protocol. Hywel Dda gives a very blunt warning about this: contracts signed before classification is confirmed could make tools unusable until certified. If your transcription tool suddenly starts offering coding suggestions or structured summaries, its classification might change.

Sam: Let us go back to the patient in the room. What happens if I just do not want the AI listening to my appointment? What is the consent process?

Maya: Transparency is a strong point of convergence. NHS England asks adopters to be transparent about how information is used and shared. The Australian Commission takes a pragmatic route, allowing disclosure through posters in waiting room, registration forms, and booking information. But there is a huge difference between notice and consent. Notice tells patients the AI is there. Consent gives them a moment of decision.

Sam: Do they actually get a moment of decision?

Maya: The College of Physicians and Surgeons of Alberta states that implied consent is not recommended for AI tools in patient care. The Medical Council of New Zealand requires informed consent where AI is used to record a consultation. Health PEI even provides a patient-facing script that simply says, you can say no, and adds that refusal will not affect care. Manitoba states that patients may refuse, withdraw, or modify consent over time.

Sam: But if a patient says no, what actually happens in the clinic? Does the doctor just have to scramble to take notes the old-fashioned way?

Maya: That is the central operational gap. Ontario requires patients who withhold or withdraw consent to receive the same or similar level of care. The Australian Commission asks organizations to manage non-consent consistently across clinicians. Saskatchewan directly asks whether staff are available for manual notes if consent is withheld.

Sam: Wait, if the whole point of buying the AI scribe was to fit more patients into the schedule, and a patient says no, the clinic probably does not have staff sitting around to take manual notes.

Maya: Exactly. The author concludes that the procedural power of consent remains operationally thin because the labor of refusal is not fully allocated. If the organization does not resource an alternative pathway, the burden of maintaining a non-scribe pathway falls entirely on the clinician.

Sam: If the burden is on the clinician, are they at least getting trained on how to use these tools safely? What happens if they spot a recurring error?

Maya: Training is recognized mostly as an organizational prerequisite. Ontario's scribe-specific checklist includes training among the elements of an AI governance framework, alongside human oversight and mechanisms for reporting, notification, inquiry, and complaint.

Sam: Inquiry and complaint mechanisms sound useful. But when exactly should a doctor report an error?

Maya: That is another gray area. Adjacent regulation, like the Netherlands AI Act Guide, requires relevant providers to record information about serious incidents and report those incidents to the AI Office. But the direct AI-scribe corpus rarely specifies when a missed negation, a hallucination, or a recurrent correction pattern crosses the line into becoming a reportable incident or near miss.

Sam: We have talked about transcription errors, but what about systemic biases? We know voice recognition struggles with certain accents. Is that addressed in these 74 documents?

Maya: Yes, but again, as an underspecified human control assumption. NHS England asks adopters to establish whether products work for patients with different accents, and to monitor whether any groups hesitate to seek services because of ambient scribing. Ontario notes limitations in multilingual conversations and with speakers who have limited English. The College of Physicians and Surgeons of Alberta adds that equitable use requires monitoring outcomes over time.

Sam: It sounds like they are just telling doctors to keep an eye on it. Do they actually provide a quantitative threshold? Like, if the error rate for a specific accent hits a certain percentage, you have to stop using the tool?

Maya: No. That is one of the author's strongest critiques. None of these documents sets a quantitative threshold at which performance inequality should pause or restrict use. Equity is treated as a broad value rather than a documented safety threshold. The author suggests future guidance should require a pause rule where one group has a materially higher missed-negation or correction rate than the local baseline. Until then, the residual labor of noticing and fixing those equity gaps remains entirely on the clinician.

Sam: It is a critical lesson for any hospital leader. You cannot just buy the software and assume the human will perfectly absorb all the residual risk for free. To recap, this PLOS Digital Health analysis of 74 official documents reveals that regulators use a drafting fiction to govern AI scribes. By treating AI output as a provisional draft, they push the liability for hallucinations, biases, and missed negations onto the verifying clinician, without allocating the protected time or operational support needed to actually keep patients safe.

Maya: Spot on. As a reminder, the source document is linked in our show notes. And a final note, this podcast provides a summary of a public document for informational purposes only and is not medical advice. Thanks for tuning in.