When AI Changes Overnight: How NICE is Rewriting the Rules for Evaluating Frontier AI

Maya: How do you evaluate a medical technology when the system completely changes the day after you approve it? That is the massive question facing health technology assessment right now.

Sam: Today we are unpacking a report called 'FRONTIER ARTIFICIAL INTELLIGENCE AND HEALTH TECHNOLOGY ASSESSMENT', published by the Decision Support Unit at the University of Sheffield.

Maya: And before we get into the details, a quick reminder that our voices are AI-generated, and this episode is a summary of a publicly available document.

Sam: This report was funded by NICE, the National Institute for Health and Care Excellence. They are the gatekeepers for what the NHS pays for in the UK. And they are looking at this coming wave of AI and realising their current rulebook is going to be seriously stretched. Maya, as someone who looks at health AI from both a clinical and business angle, what makes this report stand out?

Maya: It stands out because the authors actually reviewed 37 documents to figure out where the strain is happening. They looked at 17 guidance and policy documents, 18 peer-reviewed studies, and 2 preprints. What they found is that we are moving away from fixed, narrow algorithms to what they call frontier AI.

Sam: So, let us define that. What makes an AI a frontier AI in this context? Because it is not just about the size of the model, right?

Maya: Right. The report defines frontier AI as technologies where the effects, the risks, and the costs depend heavily on how they are configured, integrated, and governed in practice. And crucially, they change in meaningful ways after deployment. They behave like an evolving service rather than a fixed medical device.

Sam: The report gives a really concrete example of an AI copilot embedded in electronic patient record systems. Companies like Epic Systems are already integrating large language models into their software to draft notes, pull up patient histories, and coordinate workflows.

Maya: Exactly. But from a boardroom or assessment perspective, how do you evaluate that? If the copilot gets a remote update, or if the hospital tweaks the data it can access, the system you evaluated in a trial is not the same system running on the ward a month later. The real-world impact depends on what data it accesses, how outputs are framed, and whether the system can trigger downstream actions.

Sam: Which leads to what the authors call intervention instability. Change is routine, not exceptional. And that causes evidence decay. I think that is one of the most memorable takeaways from this document. The idea that evidence decays.

Maya: Evidence decay is a massive headache for payers. Usually, if you run a gold-standard trial on a new drug, that evidence holds up. For frontier AI, the question is not whether the evidence was once valid, but whether it remains valid after the model, the prompts, or the safety guardrails have been updated.

Sam: Wait, really? So the manufacturer could just push a software update that alters the prompt templates, and suddenly the safety profile shifts? And they might not even release it as a brand new product version?

Maya: Yes, they call this version ambiguity. Material changes in system behaviour can happen through updates to the model, the data sources, the prompts, the guardrails, or the workflow integration, without a clearly identifiable new version in the traditional medical device sense. It makes it incredibly hard to detect when evidence is no longer aligned with real-world use.

Sam: That is wild. But before we get deeper into the assessment nightmare, let us look at what is actually coming down the pipeline. The authors reference a horizon scan watch list from Canada's Drug Agency that identifies AI technologies likely to diffuse in the next 5 years.

Maya: The Canadian list gives a great view of where the market is heading. They highlight AI for notetaking, like AI scribes that generate transcripts and referral letters. They mention tools like PhenoPad and Tali. They also look at AI for clinical training, citing tools like OpenEvidence and ChatGPT.

Sam: And it goes beyond admin and training. They mention disease detection and diagnosis, like ASIST-TBI for screening head-injury CT scans, and LumeNeuro for retinal imaging. And for treatment, they point to Kaia Health for musculoskeletal pain and Wysa, an AI mental health chatbot.

Maya: They even mention Valence Labs' LOWE, an orchestrated workflow engine for drug discovery, and remote monitoring tools like Coughy for sound analysis, and AlayaCare. Interestingly, for AlayaCare, a Canadian clinical study reported reductions in emergency visits and hospitalisations in cohorts during a three-month implementation.

Sam: So these tools are here, or arriving very soon. But the report says a central challenge for assessing them is poor specification and transparency gaps. What does that mean for a health system looking to buy one of these tools?

Maya: It means the technology is often not described in enough detail to allow confident interpretation of the evidence. High-level product descriptions fail to capture the configuration choices and workflow roles that actually shape safety and effectiveness. If you do not have a structured specification, you cannot establish what is being assessed.

Sam: And there is another fascinating challenge called the accuracy-to-impact gap. Just because an AI model scores high on a technical accuracy test in a lab does not mean it improves patient care. The report notes that apparent technical gains might be offset by downstream harms or costs.

Maya: Exactly. The report brings in the concept of sociotechnical risk. The benefits and harms of AI depend on how clinicians interpret outputs, how uncertainty is communicated, and how oversight is designed. You have to treat the human-AI system as the intervention, rather than the AI component alone.

Sam: Because humans are messy. If an AI generates a really confident but incorrect summary, a rushed doctor might just sign off on it. The report mentions an AI system called Med-PaLM that supports expert-level medical reasoning. Evidence that focuses only on the model answers risks overstating the benefit if it ignores how clinicians act under time pressure.

Maya: Right. And to study this in the real world, the report references a study of a multi-agent, large language model-based clinical system evaluated across 500 real-world virtual urgent-care encounters. Even when the AI recommendations broadly match those made by clinicians, health technology assessment bodies still have to examine the risk of rare but severe errors.

Sam: This brings up a huge issue for me. How do you even choose a comparator? If an AI system is completely redesigning a care pathway, redistributing tasks, and changing who does what, you cannot just compare it to standard care in a clean one-to-one substitution.

Maya: You cannot. And that is why comparator choice is a substantive design choice, not just a technical detail. The report says comparator selection strongly conditions estimates of value and cost-effectiveness. Frontier AI acts as a disruptive technology, so you have to consider system-wide effects, not just local performance improvements.

Sam: And what about transferability? Let us say an AI tool looks amazing in a massive, highly funded research hospital. Does that mean it will work in a small, underfunded regional clinic?

Maya: The report explicitly states that transferability and generalisability cannot be assumed. Performance depends heavily on local data quality, digital infrastructure, staffing, training, and governance. Treating transferability as implicit risks overstating value or underestimating risk across diverse settings.

Sam: This sounds like an absolute nightmare for health economists. If the value depends on how the hospital is run, how do you build an economic model for NICE? The report lists a bunch of specific health economic challenges for frontier AI.

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: The economic challenges are severe. First, you have unstable cost drivers. Frontier AI is often priced through subscriptions or per-user fees, so costs depend on usage, which can increase as adoption grows. Then you have lifecycle-based cost structures. Maintaining safe use requires ongoing monitoring, managing cybersecurity risks, auditing outputs, and retraining staff.

Sam: Right, you do not just buy the software and walk away. The implementation and adoption themselves are cost-bearing interventions. If you do not include the cost of governance and capacity building in the economic model, you are faking your cost-effectiveness.

Maya: Exactly. And then there is induced demand. Frontier AI can increase downstream activity by identifying more cases earlier or lowering clinical thresholds for further investigation. That increases demand in services that might already face capacity constraints.

Sam: And what about time savings? Medtech companies love to say their AI saves doctors 2 hours a day. That must be a huge cost saving for the health system.

Maya: Not necessarily. The report warns about linking staff time savings to outcomes and costs. Time savings might just be redistributed to other tasks rather than converted into direct financial savings. If it does not lead to measurable improvements like reduced waiting times or lower staff burnout, those efficiency gains might not translate into cash-releasing savings.

Sam: Wow, so the economic models have to account for dynamic and structural uncertainty, plus equity and affordability as distributional issues. A tool might be cost-effective for a digitally mature trust but completely unaffordable for a smaller site.

Maya: Yes. And the report identifies a few more cross-cutting issues that make deployment difficult. There are huge concerns around data sovereignty, cross-border services, and jurisdictional governance. If a frontier AI relies on cloud services that process data outside the NHS's immediate control, that affects safe deployment.

Sam: They also mention environmental sustainability, noting that the energy use and data storage for large AI models can influence affordability and procurement constraints. Plus, there is the whole issue of liability, accountability, and responsibility across the value chain. If the AI messes up, whose fault is it? The developer? The cloud provider? The hospital? The doctor?

Maya: The diffusion of responsibility complicates governance directly. And finally, there is the reliability of the evidence trail. Because a single frontier AI system can be used for multiple clinical purposes with little or no technical change, evidence generated for one use might not reliably apply to another.

Sam: Okay, so the problems are massive. But this report is not just a list of complaints. The authors outline 12 specific recommendations for NICE to actually solve this. And they grouped them into three broad tiers of relative feasibility.

Maya: Let us start with Tier 1. These are high feasibility, near-term actions that fit into what NICE already does. The first is strengthening economic evidence to reflect lifecycle use and uncertainty for high-update frontier AI. This means economic analyses must reflect how costs and effects change with updates and ongoing governance.

Sam: Tier 1 also includes treating the human-AI system as the intervention, rather than just the AI component. And strengthening transparency requirements. NICE may require standardized disclosures setting out the intended purpose, misuse risks, limitations, and how uncertainty is communicated.

Maya: They also recommend strengthening comparator expectations and enforcing explicit comparator justification at scoping. Because frontier AI reshapes pathways, picking standard care as a baseline has to be deeply justified, not just assumed.

Sam: Moving to Tier 2, which has moderate feasibility and needs some targeted development. The biggest one here is binding NICE guidance to a specific version, configuration, and scope of use, with predefined review triggers.

Maya: This addresses evidence decay directly. If you bind the recommendation to a specific setup, you can make it clear when changes are significant enough to require review, rather than assuming the recommendation applies indefinitely.

Sam: Tier 2 also suggests requiring structured transferability analysis and UK-relevant real-world evidence planning. You have to prove how the evidence will translate into diverse NHS settings and how it will be monitored.

Maya: And for high-update frontier AI, they suggest using proportionate conditional recommendation pathways and maintained guidance. Basically, linking NHS access to ongoing evidence generation and clear stop criteria.

Sam: They also recommend adding a short frontier AI checklist to NICE guidance to ensure consistent reporting. This would require manufacturers to clearly state the regulatory and assurance arrangements they have in place, without forcing NICE to become a cybersecurity auditor.

Maya: Exactly. Now for Tier 3, the lower feasibility, longer-term recommendations. These are the heavy lifts. They recommend specifying precisely what the intervention is that is to be evaluated. This means a structured specification defining the AI system-in-use, its components, orchestration layer, prompt templates, and integration points.

Sam: Wait, they want the actual prompt templates submitted for assessment? That makes sense, seeing how a different prompt could completely change the medical output. What else is in Tier 3?

Maya: They recommend requiring Predetermined Change Control Plans, or PCCPs. Since frontier AI systems are expected to change after deployment, the appraisal should include a plan for how those changes will be managed, validated, and communicated.

Sam: That is basically asking the manufacturer to predict their own software update schedule and prove how they will keep it safe. That is a huge new hurdle for medtech companies.

Maya: It is. Another Tier 3 recommendation is taking account of the real work and costs of putting frontier AI into practice. Even if an AI works perfectly, there is significant effort and expense to buy it, connect it to NHS systems, keep it secure, and maintain assurance. NICE should reflect these practical burdens in costs and feasibility.

Sam: And the final Tier 3 recommendation is strengthening NICE's internal capacity to assess frontier AI and standardising multidisciplinary assessment processes. They need clinical, technical, organisational, and behavioural experts in the room to avoid variability in decisions.

Maya: Taken together, these 12 recommendations show exactly how health technology assessment has to evolve. For medtech companies, the days of submitting a static clinical trial for an AI product and getting lifelong reimbursement are over. You are going to have to prove you can manage the lifecycle.

Sam: And for health system leaders, this report is a warning that buying AI isn't like buying an MRI machine. You are buying a continuously evolving service, and you have to budget for the ongoing governance and cybersecurity burden that comes with it.

Maya: Exactly. To recap, frontier AI creates unprecedented challenges for HTA because it changes after deployment, leading to evidence decay. The University of Sheffield report outlines a roadmap for NICE to adapt, focusing on binding guidance to specific configurations, demanding clear change control plans, and treating the human-AI interaction as the actual intervention.

Sam: A fascinating look at the future of medical regulation. As always, you can find a link to the full source document in the show notes. And a quick reminder that this is not medical advice.

Maya: Thanks for joining us on this deep dive into the evolving world of AI evaluation. We will see you next time on AI in Medicine - Smart Summaries.