Who Holds the Bag When Medical AI Fails? The UK's New Blueprint

Maya: Did you know that 65% of stakeholders surveyed believe our current way of monitoring medical devices after they are deployed is completely insufficient for artificial intelligence? Welcome to AI in Medicine - Smart Summaries. I am Maya, and today we are unpacking a massive shift in how the United Kingdom plans to govern medical AI.

Sam: And I am Sam. Today we are digging into the National Commission into the Regulation of AI in Healthcare, a major report published by the UK Government in September 2026. A quick reminder upfront that our voices are AI-generated, and this episode is a summary of a public document.

Maya: This massive blueprint was put together by an independent expert advisory body established by the Medicines and Healthcare products Regulatory Agency, or MHRA. The core argument here is that our current regulatory approaches were designed for static products that you can reliably assess at a single point in time.

Sam: Right, which makes total sense for a traditional medical device, like a pacemaker or a surgical scalpel. You test it, you prove it works, and it basically stays the same. But AI-enabled products iterate rapidly. They perform differently depending on the clinical setting, and they can experience performance drift over time. The rules we have just do not fit the technology we are building.

Maya: Exactly, and the sector is really feeling that friction. The report notes that two-thirds of industry respondents and half of healthcare providers view the current framework as restricting innovation. The Commission, chaired by Professor Alastair Denniston and Professor Henrietta Hughes, has laid out three main pillars for reform. These are proportionate lifecycle regulation, system-wide responsibility, and enhanced trust and transparency.

Sam: Let us start with that first pillar, proportionate lifecycle regulation. What does that actually mean for a medtech developer trying to get an AI tool into the National Health Service today? Where are the current roadblocks?

Maya: Practically, it starts with getting clarity on qualification and classification. Right now, a lot of software and AI products operate in a grey area. The report asks if tools meant for administrative purposes, general wellbeing apps, or low risk clinical decision support really need to be regulated as medical devices. The MHRA needs to systematically update the definition of a medical device to make it crystal clear.

Sam: And the document points out a major flaw in how devices are classified right now. Most software and AI-enabled devices are currently self-declared as Class I. That means the manufacturer essentially grades their own homework, which leads to a limited understanding of the actual risks and benefits to patients.

Maya: Precisely. The Commission is calling for a classification system that genuinely reflects clinical risks and patient benefits. And they want to take a tailored, function-based approach. This is huge for developers building on top of general-purpose AI models. If a company builds an ambient voice technology product that includes transcription, which is not a medical device, alongside an advanced decision support function, which is a medical device, the regulator should only focus its oversight on that specific medical function.

Sam: That is a massive relief for companies. It would be entirely disproportionate and inefficient to regulate an entire general-purpose platform just because one downstream feature acts as a medical device. It levels the playing field between large corporations and smaller developers.

Maya: It definitely does, but it brings up a really thorny issue. If you are a medical device manufacturer building a clinical tool on top of a massive general-purpose foundation model built by a different company, how do you prove to the regulator that the underlying model is safe? You do not have access to all of their proprietary technical data.

Sam: That is one of the most interesting solutions in this blueprint. The Commission is proposing an opt-in Master File approach for these general-purpose platforms. It builds on similar systems used by the United States Food and Drug Administration, Health Canada, and Japan's Pharmaceutical and Medical Devices Agency.

Maya: So essentially, the upstream developer of the foundation model can confidentially submit technical specifications, benchmarks, and internal guardrails directly to the MHRA via this Master File. That creates a secure bridge. The regulatory agency gets the confidence it needs to grant market authorisation, but the foundation model developer does not have to hand over commercially sensitive trade secrets to the downstream device manufacturer.

Sam: Exactly. But the Commission also stresses that this requires immense transparency regarding product dependencies. At the time of regulatory submission, and during procurement, manufacturers have to transparently report their reliance on these underlying general-purpose models, including any continuity plans. The health system needs to monitor the cumulative risk exposure of having multiple critical medical devices relying on a very small number of external foundation models.

Maya: That is a serious sovereignty and resilience risk. If a single underlying model goes down, or changes its architecture, it could ripple across dozens of clinical tools in hospitals across the UK. Which brings us to how we handle changes and updates to these tools. We mentioned earlier that AI learns and iterates. How does the MHRA plan to regulate something that is constantly changing?

Sam: Through something called Predetermined Change Control Plans, or PCCPs. Historically, if you changed a medical device, you might need a whole new regulatory review. With a PCCP, the manufacturer and the regulator agree in advance on the boundaries within which the AI can operate and adapt.

Maya: This allows for site-specific or sub-population tuning, and even evidence-driven use expansion, without grinding the innovation process to a halt. The document references the MHRA's AI Airlock regulatory sandbox programme, which really tested these ideas. That pilot worked with four companies and generated 40 recommendations to enhance patient safety.

Sam: One of their key findings from the AI Airlock was that you cannot assess the significance of a change based on technical modifications alone. It is highly contextual. It depends on the clinical function, user interaction, automation bias, and the specific deployment environment. A technical tweak in a radiology sorting tool is very different from a tweak in an autonomous diagnostic tool.

Maya: Exactly, which is why the Commission recommends expanding regulatory sandboxes into real-world clinical settings. They mention the new London Region I sandbox, a collaboration between the MHRA, NHS England, and London Health Innovation Networks, which is supporting up to 10 AI-enabled medical devices to generate real-world evidence. Sandboxes are highly effective across industries.

Sam: They point out that the Financial Conduct Authority operates a sandbox that has worked with over 700 businesses and accelerated time to market by 40% on average. Accelerating time to market is critical, but it has to be balanced with robust evidence. The report suggests introducing staged authorisations to manage this balance.

Maya: With a staged authorisation, a device could be deployed within a tightly controlled scope, based on initial evidence, and then expanded once it meets prespecified performance thresholds in the real world. They also highlight the new National Healthtech Access Programme, which is already accelerating access to things like AI devices for prostate and breast cancer detection on histopathology slides.

Sam: Staged authorisations shift the burden of proof from a single pre-market hurdle to continuous lifecycle evidence generation. But for that to work, post-market surveillance has to be vastly improved. As we mentioned at the top, 65% of respondents disagreed or strongly disagreed that current post-market surveillance approaches are sufficient.

Maya: We need to move from passive reporting of incidents to proactive, regular reporting on overall device performance. But how do they actually plan to achieve that? Because tracking software that is embedded in complex clinical pathways sounds like a logistical nightmare. If a patient has a bad outcome, how do you even know which specific version of an AI algorithm was involved?

Sam: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Maya: And now, back to the document.

Sam: That is where traceability comes in. The MHRA wants to introduce Unique Device Identifiers, or UDIs, that follow the device throughout its lifecycle. This UDI, along with specific version control information, needs to be routinely recorded in the electronic patient health record. That way, healthcare professionals and providers can trace and audit the outcomes of patients whose care involved a specific algorithm.

Maya: That creates a massive data trail, which is great for safety, but it could overwhelm manufacturers and the regulator with reporting requirements. The document addresses this by calling for the automation of low-burden reporting. They want to integrate AI into existing reporting mechanisms, like the Yellow Card scheme, so that manufacturers can automatically report minor events without drowning the MHRA in manual paperwork.

Sam: Yes, and they want to make this safety data much more transparent to the public. They propose creating a database or tool, building on the MHRA's interactive Drug Analysis Profiles, or iDAPs, that would allow anyone to search for adverse incidents related to specific AI devices. They point to the FDA's MAUDE Database as a strong example of allowing the public to search by manufacturer, device name, and timeframe.

Maya: Transparency is great, but transparency without consequences lacks teeth. The Commission is very clear that the MHRA needs enhanced enforcement mechanisms. If a manufacturer identifies performance issues and fails to take corrective action, there must be clear consequences. They explicitly suggest financial penalties or fines to penalise manufacturers that fail to comply with legal requirements and place patients at risk.

Sam: This ties directly into the concept of health equity, which the Commission highlights as a critical aspect of safety. If an AI device works perfectly for one demographic but underperforms for another, that is a fundamental safety failure. The MHRA is urged to provide specific guidance on how manufacturers must monitor subgroups continuously to ensure equitable care outcomes across age, gender, sex, race, ethnicity, and geography.

Maya: That leads us perfectly into the second major pillar of the report, system-wide responsibility and safe management. A recurring theme in our conversations about AI is the fear among clinicians that they will become liability sinks. If an AI system makes a subtly flawed recommendation, and the doctor follows it because they are overworked or experiencing automation bias, who gets sued?

Sam: It is a massive concern. Under the current negligence framework, claims are disproportionately likely to be brought against healthcare professionals and healthcare providers, because they have the clearest duty of care. The report explicitly acknowledges this tension, noting that clinicians should not be asked to carry accountability for risks they cannot plausibly mitigate.

Maya: The UK Jurisdiction Taskforce recently issued a Legal Statement on Liability for AI Harms, concluding that the common law of England and Wales is highly flexible and capable of accommodating these novel technologies. But the perception of uncertainty alone is enough to discourage responsible innovation. So, how does the blueprint propose we divide up this responsibility so doctors do not feel like they are left holding the bag?

Sam: They want to mandate that pre-market submissions explicitly outline the operational conditions needed for safe deployment. The manufacturer has to clearly specify in their risk management files exactly what controls must exist in the user environment. That might include minimum cybersecurity controls, specific training outcomes for professional users, or defined processes for monitoring performance drift.

Maya: And to formalise that, they recommend a contract to deploy and use between the manufacturer and the healthcare provider. Before the AI is ever turned on, this contract explicitly allocates who is responsible for delivering which risk controls. No risk control should be left unaccounted for.

Sam: Right. If the manufacturer needs post-market data, the hospital contractually agrees to provide it. If the hospital needs onboarding support, the manufacturer contractually agrees to supply it. Which brings us to the concept of AI readiness. You cannot just drop a highly sophisticated AI agent into a hospital with poor digital maturity and expect it to go well.

Maya: Absolutely not. The Department of Health and Social Care, alongside devolved health departments, needs to co-develop an AI readiness toolbox. Providers must be able to self-assess their decision-making readiness and their implementation readiness before they procure anything. The report notes this toolbox should build on existing governance processes, mentioning standards like DCB129 and DCB160 in England.

Sam: They also call for a national learning function or observatory. They want to centralise the lessons learned from local implementations so that every trust does not have to make the same painful mistakes in a silo. And we cannot ignore the workforce component. Healthcare providers have a strict obligation to ensure their staff are trained on these specific technologies.

Maya: The General Medical Council has already published a learning resource on applying standards in the context of AI, but we need scalable, continuous professional development. The report mentions the NHS Digital Academy's Clinical AI Fellowships and the AI Ambassador Network, which already has more than 14,000 members from NHS England, as a great foundation for building this baseline literacy.

Sam: I want to circle back to a specific type of technology the report touches on, direct-to-consumer devices. Think wearables and consumer-facing health apps. If there is no clinician acting as an intermediary, the dynamic completely changes.

Maya: It does, and the MHRA is told to provide clear regulatory expectations for these direct-to-consumer applications. Because there is no healthcare professional to act as a safeguard, the manufacturer bears a much heavier burden for user-centred design and human factors. The device interface and all supporting materials must be suitable for a general user to interpret safely without any medical background.

Sam: On the flip side, that direct relationship allows manufacturers to gather near-real-time performance data and push rapid updates directly to the user. It is a massive opportunity for shifting towards preventative care, provided the human factors testing is rigorous. They suggest using model cards and dynamic labelling to give users up-to-date transparency about the tool's performance and limitations.

Maya: Another major vulnerability mentioned in the text is cybersecurity. As these devices become increasingly interconnected and integrated into clinical workflows, the risk of cyber-attacks skyrockets. The document explicitly mentions data poisoning, where training data is deliberately corrupted, and malware infiltration. The MHRA is urged to issue specific guidance so that safeguards are embedded into the operating model from the very design phase.

Sam: There are also complex jurisdictional issues here. The report points out that the Windsor Framework creates unique challenges for Northern Ireland. If a software product is not classed as a medical device in Great Britain but is regulated as one in the European Union, a manufacturer might just skip getting a CE marking entirely. That could leave Northern Ireland without access to emerging technologies.

Maya: Which is why international harmonisation is such a huge focus of this document. The MHRA recently held a consultation on international reliance routes, and 59% of respondents agreed that software-based medical devices should be added to these pathways. The UK wants to introduce recognition pathways with trusted international regulators, allowing fast-tracked access to tools developed abroad, provided it is in the best interest of patients.

Sam: This whole document really paints a picture of a regulatory system trying to evolve from a tollbooth into a continuous partnership. They are asking manufacturers, hospitals, clinicians, and regulators to essentially lock arms across the entire lifecycle of an AI product. But if I am a hospital leader, I am looking at this and thinking about the immense resource burden of monitoring, reporting, and managing these new contracts to deploy and use.

Maya: That is the central tension of the entire initiative. The Commission acknowledges that successful implementation requires regulatory bodies to be appropriately resourced and equipped with the necessary capability and capacity. The same goes for the healthcare providers. If we want AI to reduce the risk of clinical burnout and handle routine administrative tasks, we have to invest upfront in the digital maturity, the UDI tracking, and the AI readiness toolboxes required to manage them safely.

Sam: If I had to pick one memorable takeaway from this report, it is the fundamental shift in how we view accountability. Trust cannot be assumed; it must be earned. And we earn it by moving away from blaming the end-user clinician for system failures, and instead enforcing strict, shared responsibilities across the entire lifecycle of the product. The AI needs to do what AI is good at, and humans need to be empowered to do what they are good at.

Maya: To give a crisp recap of our discussion, the UK is proposing a lifecycle approach to AI regulation with shared system accountability, moving away from static approvals toward real-world monitoring. A crisp reminder that the source document is linked in the show notes, and that this is not medical advice. Thanks for listening!