NICE's New AI Rulebook: Validating AI-Generated Evidence for Healthcare

Sam: If a pharma or medtech company uses AI to help write their drug dossier, the UK's top health tech assessor just published the exact rulebook they have to follow to get it approved.

Maya: Today we are looking at the AI methods for evidence framework, a corporate document published by the National Institute for Health and Care Excellence on 9 November 2026.

Sam: And just a quick reminder for our listeners, our voices are AI-generated, and this episode is a summary of a public document.

Maya: This framework is incredibly important for anyone in the boardroom of a life sciences company, or anyone reading clinical guidelines on the ward. It dictates how artificial intelligence can be used to actually generate the evidence that NICE relies on.

Sam: Right, so just to be clear, we are not talking about regulating an AI medical device, like an algorithm that detects tumours. We are talking about using AI to do the background research, right?

Maya: Exactly. The development or assessment of health technology products that incorporate AI is outside the scope of this framework. This is entirely about using AI in evidence activities, like systematic literature reviews, data extraction, and health economic evaluations.

Sam: Which is a massive bottleneck for companies. Evidence synthesis takes a huge amount of human hours. Obviously, developers want to use large language models to speed that up. But NICE is laying down some serious guardrails here.

Maya: They are. NICE structures this around 4 types of principles. Foundational principles, which cover data compliance and responsible use. Governance principles, for accountability and human oversight. Assurance principles, ensuring validation and reproducibility. And Trustworthiness principles, focusing on explainability and transparency.

Sam: Let's start with Governance. If an AI writes part of a clinical dossier and gets a fact wrong, who actually takes the fall?

Maya: The human does. The framework explicitly splits accountability into answerability and auditability. Answerability means humans must take responsibility for their entire involvement across the AI design and implementation. There can be no responsibility gaps.

Sam: And auditability?

Maya: That demands that the results of the work are traceable from start to finish. If a reviewer is looking at your submission, they need to be able to trace how the AI systems were used, what decisions were made, and who approved them.

Sam: So you can't just blame the black box. Speaking of human involvement, how does NICE classify the way humans and AI work together? I imagine it's not a one-size-fits-all situation.

Maya: It isn't. The document outlines 3 levels of automation. First is support, where AI just assists, maybe by highlighting text, but humans make all the decisions. Second is semi-automation, where AI completes tasks alongside a human, but final decisions need human adjudication.

Sam: And the third level?

Maya: Full automation. That is when the AI completes an end-to-end task and makes decisions on its own. But even then, NICE requires what they call a human-at-the-helm oversight.

Sam: Wait, what is the difference between human-at-the-helm and just keeping a human in the loop?

Maya: NICE considers 2 levels of human oversight. Human-in-the-loop is when a human scrutinises all interim and final results. Human-at-the-helm means a human might only scrutinise the final results from the system. But full automation requires incredibly rigorous validation to prove it is safe.

Sam: Let's dig into that validation part, which falls under the Assurance principles. If a company builds an AI tool to screen medical journals, how do they prove to NICE that it actually works?

Maya: Every validation exercise must be carried out on independent datasets. You cannot test the model on the data used to train it. That creates data contamination bias, where the model essentially just memorises the answers.

Sam: Makes sense. But when they are comparing the AI's performance, what are they comparing it against? The old way of doing things with human reviewers?

Maya: Yes, but with a massive caveat. The framework warns against benchmark bias. This happens when the reference standard itself has errors. You cannot just assume human performance is a perfect benchmark.

Sam: Because humans get tired and miss things too.

Maya: Exactly. NICE says that pre-specifying thresholds for a model to achieve, like an 80% true positive rate for screening, may not be appropriate. Instead, you need a robust reference standard, often established by independent review and adjudication by multiple reviewers.

Sam: So you can't just pick an arbitrary percentage and say your AI passed. What metrics do they actually want to see when companies report their validation?

Maya: When possible, they want 95% confidence intervals reported alongside metrics. If that cannot be calculated, they suggest alternatives like bootstrapped estimates or statistical tests.

Sam: Let's apply these rules to the actual pipeline of putting together a NICE submission. The first heavy lift is usually the literature search. Can you just ask a large language model to go find every paper on a specific drug?

Maya: Absolutely not. The framework is very strict here. AI may not be used as the sole strategy to automatically generate terms and run a search. The evidence shows that when AI runs searches automatically, it has highly variable performance.

Sam: How variable are we talking?

Maya: Some studies demonstrate that true positive and true negative rates can be as low as 0% to 10%.

Sam: Wait, really? 0% to 10%? That is completely useless.

Maya: Which is exactly why NICE prohibits fully automating the literature search. You can use AI to generate and test candidate terms, but a qualified human expert must review the syntax and databases before the final search is executed.

Sam: Okay, so the human runs the search and gets back hundreds or thousands of abstracts. Usually, reviewers spend a lot of time doing title and abstract screening to figure out which ones are actually relevant. Can AI do that part?

Maya: Yes, but the level of automation dictates the risk. Support is low risk. Semi-automation is allowed if an independent human adjudicator resolves disagreements. Full automation is the trickiest.

Sam: Because if the AI fully automates it and gets it wrong, you just permanently lose a vital piece of clinical evidence.

Maya: Precisely. The primary performance metric for title and abstract screening is the false negative rate, or its complement, the true positive rate. A false negative is a relevant record incorrectly excluded.

Sam: So they care much more about not missing good data than they do about accidentally including a few irrelevant papers.

Maya: Exactly. NICE states that automation should only be used for making exclusion decisions when validation gives a high degree of assurance that the excluded records are not relevant. Anything the AI keeps must still be screened by a human reviewer.

Sam: That is a very pragmatic rule. Use the AI as a highly cautious filter to throw out the obvious garbage, but humans still check the rest.

Maya: And to make the AI cautious, the framework suggests prompt engineering techniques. For instance, creating a cautious persona. You literally prompt the model with, you are an experienced but cautious evidence synthesist who over-includes when in doubt.

Sam: I love that. You have to actively tell the LLM not to be arrogant.

Maya: You can also constrain the outputs to reduce hallucination and save compute costs. The framework gives an example of telling the model to respond with single characters: I for include, E for exclude, or ? for when the model is unsure.

Sam: What about using a scoring system?

Maya: That is also recommended. You can ask the model to rate the likelihood of a record being included on a scale from 1 to 5. Then a human sets a pre-defined decision threshold to convert that score into a screening decision.

Sam: Okay, so we've screened the abstracts. Now we have the full text of the actual clinical trial reports. Can AI screen the full text automatically?

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: No. The framework explicitly says that given the limited maturity of evidence and the elevated risk profile, it is not recommended to use AI to fully automate full text screening.

Sam: Because the stakes are higher once you are looking at the definitive inclusion decisions?

Maya: Yes. And full texts are long. Even if you fit the document into the context window, it does not guarantee all relevant information will be used reliably. NICE suggests techniques like retrieval-augmented generation to chunk the text, but the RAG engine itself must be validated.

Sam: So humans are still reading the full texts. What about data extraction? That's where you have to pull the actual statistics, patient demographics, and hazard ratios out of the text and put them into a spreadsheet.

Maya: Fully automated data extraction using AI is also not currently recommended. The published evidence is limited and highly variable. Continuous data are particularly prone to incorrect extraction compared to categorical data.

Sam: So no letting the AI pull the numbers unsupervised. But if a company uses AI for semi-automated extraction, where a human checks it, how does NICE evaluate if the AI is doing a good job?

Maya: NICE requires overall concordance as the primary performance metric. Concordance measures semantic alignment, not just lexical alignment. The extracted element must be factually aligned to the source material.

Sam: Why call out semantic alignment specifically?

Maya: Because traditional text generation metrics are dangerously misleading here. The framework notes that scoring methods like BLEU and recall-oriented understudy for gisting evaluation, or ROUGE, have limited value for structured data extraction.

Sam: Why are they bad?

Maya: Because they do not reflect factual or numerical accuracy. Those metrics can score an AI extraction as very similar to the human extraction even when the numerical value is incorrect, including cases where the direction of effect is reversed.

Sam: That is terrifying. The AI could completely flip the clinical result of a trial, but the traditional algorithm scoring metric would say it did a great job just because the words around the number matched.

Maya: Exactly. NICE gives a great example regarding how you set up your extraction tables to catch this. Imagine you are extracting an odds ratio and a confidence interval.

Sam: Okay, so an odds ratio of 0.55, with a confidence interval of 0.41 to 0.68.

Maya: Right. You can define that as a single element in your table requiring correct extraction to 2 decimal places. If the AI pulls 0.55 and 0.41, but hallucinates the upper bound as 0.64, the entire element is marked discordant.

Sam: Because it got a single piece wrong, the whole cell is a failure.

Maya: Yes. Alternatively, you can assess it as 3 individual elements. The odds ratio, the lower bound, and the upper bound. If it gets the odds ratio and lower bound right to 2 decimal places, but hallucinates the upper bound to 1 decimal place, like 0.70, only that third element is discordant. This changes your overall concordance score drastically.

Sam: So companies have to be incredibly transparent about exactly how they define a correct extraction. It's not enough to say the AI is highly accurate. You have to prove what 'accurate' actually means cell by cell.

Maya: Exactly. And they must report any data hallucinated by the AI, or extracted when it wasn't available in the source, as a false positive.

Sam: This leads perfectly to another massive part of the framework: reproducibility. LLMs are famous for giving you a different answer every time you ask the same question. How does NICE handle that?

Maya: They distinguish between deterministic models and stochastic models. Deterministic models, like a trained random forest, must demonstrate identical outputs given the same inputs and configuration. But LLMs are inherently stochastic processes.

Sam: So strict reproducibility is impossible?

Maya: Strict reproducibility is not always achievable for generative AI. So NICE says verifiability may be more pragmatic. The primary findings should be verifiable and remain consistent across repeated applications. Response stability must be evaluated by repeating representative tasks.

Sam: What about the actual writing of the submission? We've talked about finding the data, but companies also build complex health economic evaluations and write massive technical documents. Can AI build the health economic model?

Maya: For health economic evaluation, the framework is very clear. AI may be used as a support tool only. AI must not be used for semi or full automation. You can use it to help design a model or generate code, but a human must verify and run it.

Sam: And for writing the actual technical documents, like the company evidence submission form or the budget impact analysis?

Maya: AI can support or semi-automate drafting sections or summarising, but all outputs must be reviewed by a human with relevant expertise. Outputs from generative AI must be accompanied with verbatim quotes or references to the source documents.

Sam: Because they are so prone to hallucinating citations.

Maya: Yes. And there is a fascinating behavioural warning here. Users must be trained to recognise the tendency of LLMs to be sycophantic, meaning they tend to reinforce users' assumptions or preferences.

Sam: Ah, so if a pharma company wants their drug to look highly effective, the LLM might just agree with them and subtly spin the summary to please the prompter. That is context bias in action.

Maya: Exactly. Prompting with 'you are a cautious reviewer' produces very different results than 'you have some risk appetite'. Prompts must be reviewed by multiple people and written in a neutral voice.

Sam: To prevent people from engineering an AI persona that just rubber-stamps their own bias.

Maya: Furthermore, to minimise the risk of automation bias, NICE recommends AI is only used to summarise documents created by the user or their organisation, and not by third parties.

Sam: Wait, really? You cannot just feed your competitor's dossier or an independent clinical report into an LLM and ask for a summary?

Maya: They consider it too high a risk for automation bias. When you summarise external documents, it requires careful contextual and scientific interpretation. Humans might place undue trust in the AI's summary of a complex third-party text without reading the original.

Sam: That is a huge limitation on how people naturally use these tools. But it makes perfect sense for ensuring the integrity of a national health technology assessment.

Maya: Integrity is the overarching theme here. We also have to touch on the Assurance principle of fairness and equity. Bias can be introduced through the data inputs. If an AI performs poorly on non-English abstracts, or evidence on rare diseases, it creates a systemic disadvantage.

Sam: How do companies prove their AI isn't doing that?

Maya: They have to assess performance across relevant demographic, clinical, and population subgroups. If the AI performs worse on certain study types, they have to configure deterministic flags to ensure those records are escalated to human experts.

Sam: So bringing it back to the boardroom. If you are leading a market access team at a pharma company, what does this framework actually change for you tomorrow?

Maya: It means you can no longer buy off-the-shelf AI automation and assume it will speed up your NICE submission without severe scrutiny. You have to maintain an internal record of model versions, workflows, and prompts. If the underlying API updates, you might have to re-validate everything.

Sam: AI lifecycle management. The performance degrades or changes, and you are on the hook to catch it.

Maya: Exactly. Non-technical users also share the responsibility to actively monitor model performance over time. You cannot just delegate this to your IT department. The domain experts must oversee the outputs.

Sam: To recap, NICE's new framework demands strict validation, semantic concordance, and meaningful human oversight before AI can generate evidence for healthcare evaluations. As always, you can find the source document linked in our show notes.

Maya: And a quick reminder that we are AI-generated voices, and this podcast is not medical advice. See you next time!