Who Takes the Blame When AI Fails? The UK's Plan to Rewrite the Medical Device Rulebook

Sam: Did you know that just under 1 in 10 people are now using large language model chatbots as the source of information they most often use for health and care purposes? That is a massive shift happening right under our noses.

Maya: It really is. And the UK government has noticed. Welcome to AI in Medicine - Smart Summaries. I am Maya, here with Sam. Just a quick reminder right at the top: our voices are AI-generated, and this podcast is a summary of a publicly available document.

Sam: Today we are looking at the Call for Evidence: National Commission into the Regulation of AI in Healthcare. It was published on 18 December 2025 by the UK Department of Health and Social Care.

Maya: This is a really critical document for anyone building, buying, or using clinical AI. The government is trying to figure out how to update their rulebook to handle everything from symptom-checking chatbots to highly autonomous systems. And they are asking the public, industry, and clinicians for the answers.

Sam: They are not just asking nicely, they have a massive ambition here. The document states that the 10 Year Health Plan for England and the Life Sciences Sector Plan both commit to the NHS becoming the most AI-enabled healthcare system in the world.

Maya: That is a huge claim. To back it up, they need a new regulatory framework. The document notes that if this is addressed properly, AI could make care safer, faster, and more personalised. It could boost NHS productivity and accelerate UK growth in the health technology sector.

Sam: But the operative phrase there is if addressed properly. The text acknowledges several new regulatory challenges. To tackle them, they have set up this National Commission. Maya, who is actually running this group?

Maya: The Commission is chaired by Professor Alastair Denniston on behalf of the MHRA, which is the UK medicines and medical devices regulator. He heads up the UK centre of excellence in regulatory science in AI and digital health. And crucially, the Deputy Chair is the Patient Safety Commissioner, Professor Henrietta Hughes.

Sam: Having the Patient Safety Commissioner as the deputy really sends a signal. This isn't just about speeding up tech procurement, it is fundamentally about keeping patients safe while moving fast.

Maya: Exactly. They are supported by several working groups filled with experts from global tech companies, leading clinicians, and international regulators. Plus, there is a research and engagement programme supported by the Health Foundation to make sure patient and service user voices stay central.

Sam: And the clock is ticking on this. This call for evidence is basically their way of crowdsourcing the hardest questions in AI regulation. They want to publish their recommendations via the MHRA in 2026. The deadline to submit evidence is 23.59 pm on 2nd February 2026.

Maya: So if you are a health-system leader, a software developer, or a clinician, you have a very narrow window to shape the rules you will be living under for the foreseeable future.

Sam: Let us get into the meat of what they are actually regulating. The document lists how AI is currently being used, starting from outside formal healthcare, which is where that 1 in 10 statistic comes from about people relying on chatbots. But then it moves into the hospital.

Maya: Right. It mentions administrative uses, like automated systems with conversational interfaces that invite patients to screening and schedule appointments via messaging. Then it highlights Ambient Voice Technologies.

Sam: The tools that listen to the consultation and write the notes.

Maya: Yes. The text notes these reduce the note-taking burden on health professionals. It also flags a newer class of patient assistant services that give patients a record of the conversation and advice based on it.

Sam: Which is fascinating, because normally the doctor holds the record. Now the patient has an AI summarising the clinical encounter directly for them.

Maya: Beyond that, the document covers AI for screening, like detecting cancer, diagnosis, treatment decisions, and delivering therapies. Some are narrow, just doing a specific task. Others use large language models for broader applications.

Sam: And this leads to a phrase that I think every boardroom needs to pay attention to. The document explicitly brings up agentic AI. Maya, how do they define that?

Maya: The document explains that this means designing AI to coordinate multiple other specialist AI systems to handle complex tasks. That is a massive leap from a basic image scanner that just flags a potential tumour.

Sam: An AI that manages other algorithms to complete tasks. No wonder the regulators are losing sleep. How on earth does the current system handle something like that?

Maya: Well, that is the core problem. The existing framework is rooted in the Medical Devices Regulations 2002. If software has a medical purpose, it qualifies as a medical device. For low-risk devices, manufacturers self-declare conformity. For medium-risk and high-risk devices, Approved Bodies in Great Britain carry out independent assessments.

Sam: Wait, 2002? The smartphone didn't even exist then, let alone agentic AI.

Maya: Exactly. Regulators review clinical evidence, risk management, manufacturing, and labelling. Then the manufacturer registers with the MHRA. Interestingly, because of market access rules, Northern Ireland still follows EU regulations and requires a CE certificate and marking.

Sam: So we have a framework from 2002 trying to regulate autonomous AI agents. But the document mentions some very recent updates to the rules, right? Specifically around what happens after the product is sold.

Maya: Yes, post-market surveillance. In June 2025, the MHRA introduced new regulations for Great Britain. This is where it gets highly operational for tech companies. They laid out specific requirements for manufacturers.

Sam: Let us walk through those. Because if you are a vendor, this is your compliance checklist.

Maya: First, you must have a post-market surveillance system in place, produce a plan, and actually follow it. You also have to report serious incidents to the MHRA, investigate them, and report your conclusions.

Sam: Standard stuff so far. Find a problem, tell the regulator, investigate it.

Maya: Right. You also have to undertake preventative and corrective actions, including field safety corrective actions, and issue field safety notices to customers.

Sam: Field safety corrective actions for software. I suppose that means pushing an emergency update or pulling the algorithm offline if it starts hallucinating medical advice.

Maya: Exactly. Another tricky requirement for AI is conducting trend reporting of significant increases in incidents that do not require reporting. If your AI slowly degrades in accuracy over time, you have to catch that trend.

Sam: That requires constant, rigorous monitoring by the vendor. What else is there?

Maya: There are also rules to produce a safety report or periodic safety update depending on the device risk class, and to meet all documentation and information provision requirements to support these processes.

Sam: That is a heavy burden for a lean tech startup. But if we want safe AI, it seems non-negotiable. The document also talks about a tangle of data laws you have to navigate before you even get to the medical device part.

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: Yes, they list several different acts. The Public Records Act 1958 regarding how officials access data. The Freedom of Information Act 2000 for public access. The Health and Social Care Act 2008 for maintaining accurate patient records.

Sam: And I assume the data protection ones?

Maya: Correct. The Data Protection Act 2018 for processing personal information. And the brand new Data Use and Access Act 2025, which specifically introduces changes around using data for scientific purposes.

Sam: So companies and hospitals have to work together across all those laws to complete data sharing agreements and data protection impact assessments. It is a legal minefield.

Maya: Which is exactly why the MHRA wants evidence. They explicitly ask how to manage the overlap between multiple sources of assurance. You have the regulation of medical devices, you have the oversight of trained healthcare professionals, and you have the regulation of health and care services.

Sam: These domains operate entirely differently. A software bug is a device issue. A doctor missing a red flag is a professional oversight issue. A poorly staffed ward is a service regulation issue. But with AI, they all collide at once.

Maya: That collision is the focus of Section 3, which is the actual survey. They give respondents multiple choice questions and 500 words of free text space to solve these exact problems.

Sam: Let us look at Question 1. They are not holding back. They ask which statement best describes your view on the current UK framework. The options are: No change, Minor adjustments, Significant reform, Complete overhaul, or Unsure.

Maya: It is quite rare for a government body to openly offer Complete overhaul as an option for a framework they currently manage. It shows they know the 2002 regulations might just be fundamentally unfit for modern AI.

Sam: And Question 3 asks about the framework's impact on innovation. Options range from Too restrictive, stifling innovation, to Too loose, risking patient safety.

Maya: But the hardest part of the survey, the part that will keep hospital lawyers and tech executives awake, is Question 8. It asks about liability in the event of an adverse patient outcome.

Sam: This is the classic AI dilemma. They present highly specific scenarios for where liability should lie.

Maya: The first scenario is when the AI tool gives the correct answer, but is incorrectly overridden by the healthcare professional.

Sam: So the machine was right, the human doctor ignored it, and the patient was harmed. Does the doctor take all the blame for ignoring the system? Or does the hospital take the blame for not training the doctor to trust the system?

Maya: Exactly. And then a second scenario: when the AI tool gives the incorrect answer and the healthcare professional follows it. The text says they incorrectly choose to trust the AI.

Sam: The machine hallucinates, the doctor trusts the machine, the patient is harmed. Who gets sued? The tech vendor who built the flawed AI? Or the doctor who failed to spot the error?

Maya: The document doesn't have the answer. They are literally asking the healthcare ecosystem to tell them who should take the blame. In Question 6, they ask if existing laws like medical negligence and product liability can handle these disputes, if gaps exist, or if current laws are just unfit for AI.

Sam: If you are a vendor selling an AI diagnostic tool to the NHS right now, how you answer that question could dictate your insurance premiums and legal risk.

Maya: Absolutely. They also ask in Question 5 how to manage post-market surveillance for very specific edge cases. For instance, foundation models, continuous learning AI that updates itself, and AI used for purposes beyond its original intended use.

Sam: Oh, off-label AI use. A hospital buys an AI to summarise notes, and a clever administrator starts using it to triage patient severity based on those notes. The vendor never tested it for triage.

Maya: Precisely. The MHRA wants to know how information should be shared between healthcare providers and manufacturers to catch that kind of off-label use during post-market surveillance. They also ask about AI systems developed by a single institution for in-house use only.

Sam: Homegrown algorithms. If a university hospital builds its own agentic AI to coordinate its ICU beds, do they have to regulate themselves as a medical device manufacturer? The compliance cost could kill academic innovation.

Maya: Which loops back to Question 4. The MHRA explicitly asks for ideas on ensuring public and patient safety whilst minimising the cost of complying with regulations in terms of time and resource.

Sam: They want it safe, fast, and trusted, but also cheap to comply with. That is the holy grail of regulation. It is a very tough needle to thread.

Maya: And they acknowledge previous efforts to thread it. The document references the Software and AI as a Medical Device Change Programme that the MHRA published in 2023. That programme aimed to provide a high degree of protection while making the UK a globally recognised home of responsible innovation.

Sam: This new Commission seems like the sequel to that 2023 programme, but with much higher stakes given the explosion of LLMs and agentic AI. So, who exactly are they begging to fill out this survey?

Maya: They want to hear from a wide range. The public, patients, and charities. Industry, which includes AI developers of any size, trade associations, and approved bodies. Healthcare leaders, including NHS and independent sector professionals, practice managers, and procurement staff. And finally, UK and international regulators.

Sam: It is interesting that they specifically call out procurement staff. Those are the people who actually buy the software. If procurement doesn't trust the regulatory stamp, the NHS won't buy the tool.

Maya: Exactly. They need practical feedback. You can respond via an online system, or by email to Engagement at mhra.gov.uk, or even by post to their Canary Wharf office.

Sam: And there is a whole section on data protection for the survey itself. If you respond as an individual, they ask for optional demographic data like your sex as registered at birth and ethnic group.

Maya: Yes, they state their legal basis for processing that special category data is Substantial Public Interest for the purpose of Equality of Opportunity or Treatment. They will publish a summary of responses on GOV.UK, listing names of organisations, but keeping individual people's details confidential.

Sam: Let us bring this back to the ward and the boardroom. Maya, you have been on the clinical side and the analysis side. What is the single most memorable takeaway from this call for evidence?

Maya: The most memorable takeaway is that the UK government is openly admitting that the era of regulating AI like a static piece of hardware is over. By asking the industry to define liability when a doctor incorrectly overrides a correct AI, they are acknowledging that AI is no longer just a tool, it is an active participant in clinical decision-making. The rules of liability have to reflect that reality.

Sam: To recap, this National Commission is gathering evidence to advise the MHRA on developing a new regulatory framework. They will make recommendations on how to handle continuous learning and agentic AI, manage post-market surveillance, and figure out who is liable when AI and humans disagree. Their recommendations will be published in 2026.

Maya: The source document is linked in the show notes so you can read the questions and submit your own response. As always, this is not medical advice. Thanks for listening to AI in Medicine - Smart Summaries.