19 November 2026 · 19 min

When Clinical AI Acts on Its Own: Building Defensible Agents in Healthcare

Generative AI & LLMsHealth Systems & OperationsRegulation & PolicySafety, Ethics & Liability

AI in healthcare is shifting from making predictions to taking autonomous actions. This document from IQVIA outlines how health systems and pharma companies must update their validation pipelines to ensure AI agents are defensible, auditable, and safe.

Key points

Source: Validating AI Agents and Software - IQVIA, 2026

This spot is available. Reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsor the show

This episode is an AI-generated conversation summarising a public document; the hosts' voices are synthetic. It is for information only and is not medical advice. Always refer to the original source.

Your company here. This podcast is looking for its first sponsors: reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsorship options and rates →

Transcript

Sam: The biggest shift in healthcare technology right now isn't about AI making better predictions. It is about an emerging class of systems, called agents, that actually take actions. And that completely changes the central governance question from whether a prediction is accurate to whether an action is authorized, appropriate, and reversible.

Maya: We are looking at a document called Validating AI Agents and Software, published by IQVIA on September 30, 2026. Before we dive into the details, a quick reminder that our voices are AI-generated, and this podcast is a summary of a public document.

Sam: This document hits on a massive anxiety I hear from both hospital IT leaders and pharma executives. We spent the last few years getting comfortable with generative AI tools that act like super-smart interns giving us advice. But now, we are entering what IQVIA calls the agentic era. What exactly does that mean on the ground?

Maya: It means software is being built in a fundamentally different way. A decade ago, code was written line by line by human engineers, and then reviewed and tested against specifications. Today, a growing share of software is drafted, refactored, and tested by AI assistants. But the real shift is the move to agents. Agents do not merely produce code or predictions. They take actions.

Sam: Right, actions. So instead of just suggesting a diagnosis or summarizing a chart, they are doing what?

Maya: They call tools. They query databases. They draft and send communications, update patient records, and orchestrate multi-step workflows with very limited human intervention. The authors call this a qualitative shift. It changes the central governance question from whether a prediction is accurate to whether an action is authorized, appropriate, and reversible.

Sam: And that is exactly what keeps a compliance officer awake at night. If an agent is orchestrating multi-step workflows, its behavior isn't fixed, right? It adapts based on what it encounters in the moment.

Maya: Exactly. The very properties that make agents so powerful also make them incredibly hard to govern. Their behavior emerges at runtime rather than being fixed at design time. So traditional testing and code review capture only a tiny fraction of the risk surface. Their reasoning is probabilistic, their execution path is dynamic, and they can even delegate tasks to other agents. That creates accountability chains that traditional software governance never anticipated.

Sam: Which is a huge problem in healthcare and life sciences. If an AI action influences a regulated decision, or a patient interaction, or the release of a pharmaceutical product, you cannot just shrug and say the algorithm was having a weird day. The tolerance for unexplained behavior is basically zero.

Maya: And the expectation of evidence is very high. Customers are converging on two words: validation and trust. They want to know that AI-assisted software does what it is supposed to do, that an agent acts only within its specific remit, and that when it does act, someone can reconstruct exactly what happened and why.

Sam: So regulators must be scrambling to figure this out. How is the regulatory landscape actually handling this shift from predictive models to acting agents?

Maya: According to the document, the regulatory ground has shifted decisively in roughly 24 months. They have moved from saying let us think about this to explicitly saying here is how we will assess it. The bar has officially moved from simply being trustworthy to being provable.

Sam: Provable. That implies a heavy burden of documentation. What specific regulatory moves are we talking about here?

Maya: There are several key milestones. On the medicines side, the EMA finalized its reflection paper on artificial intelligence in the medicinal product lifecycle in September 2024. Then, the FDA released a draft guidance in January 2025 setting out a seven-step, risk-based credibility framework with lifecycle maintenance. Since then, the two agencies have converged on a shared set of AI principles.

Sam: And beyond just the FDA and EMA, what about broader international standards? Are we flying blind, or are there actual frameworks to map to?

Maya: The foundation is now provided by international standards. Specifically, ISO/IEC 42001 for AI management systems and ISO/IEC 42005 for AI system impact assessment. On top of that, there are industry and reporting standards formalizing risk-based, tiered validation. Things like TRIPOD+AI, TRIPOD-LLM, the ISPE GAMP guidance, and EFPIA's work on AI across the medicines lifecycle.

Sam: Okay, so the rules of the road exist. But the paper makes a really interesting distinction between words that companies often use interchangeably. They talk about responsible AI, trustworthy AI, and defensible AI. I feel like most companies just slap the word responsible on their website and call it a day. What is the actual difference here?

Maya: It is a vital distinction for boardroom leaders. Responsible AI is mostly about stating principles and is highly policy focused. Trustworthy AI is more technical, looking at properties like reliability, robustness, and fairness. But defensible AI is the operational standard. It is what actually produces the evidence that a system can withstand scrutiny throughout its entire lifecycle, rather than just passing a one-time review.

Sam: So defensible AI turns good intentions into something an auditor or a clinician can actually examine and accept. How does the paper break down this concept of defensibility?

Maya: It splits defensibility into three faces. First is scientific defensibility, which means having evidence that the system is practically useful and works for its intended use and context. Second is technical defensibility. This involves the safeguards engineered into the system to prevent or manage failure, drift, bias, and attack. And third is ethical defensibility, which guarantees meaningful human integration and oversight to preserve accountability as automation increases.

Sam: I want to dig into how we apply those three faces. If I am running a hospital system, I might have hundreds of different AI tools running simultaneously. Some might just be summarizing medical literature, while others might be autonomously flagging sepsis or managing batch releases in a manufacturing plant. Do I have to apply the same heavy validation pipeline to all of them?

Maya: No, and that brings us to the organizing idea that makes all of this workable, which is proportionality. The operating model is the same for every system, but the bar is strictly calibrated to the use case. The document explicitly says that tiering is use-driven. The inherent risk rises with proximity to patient care, to product quality, and to regulated decisions.

Sam: So the downstream impact is what matters, not necessarily how complex the AI model itself is?

Maya: Exactly. A commercial drafting assistant should trigger promotional-compliance checks and drift monitoring, not a full clinical validation package. But an agent that touches a regulated product release attracts the full weight of evidence and oversight. Proportionality prevents governance from becoming either theater for low-risk uses or a rubber stamp for high-risk ones.

Sam: But wait, doesn't the type of model matter at all? If I use a basic deterministic rules engine versus a cutting-edge generative foundation model, surely the generative model is inherently riskier?

Maya: The model exposure does modulate the risk, but it does not define the tier on its own. The paper suggests tiering first by use, and then adjusting for exposure. Yes, a generative model in a patient-facing context warrants escalation because it can amplify the risk. But when we talk about agents, the decisive variable that amplifies everything is autonomy.

Sam: Because autonomy moves a system from just giving advice to actually executing a task. That makes sense. It takes the human buffer out of the loop.

Maya: Precisely. The document points out that in the 2026 revision of the most widely used catalogue of language-model application risks, excessive agency rose to become one of the top three concerns. This was the largest upward movement on the list. So, an agent might be built on a generative model, but its risk tier is ultimately driven by what it is allowed to do.

Sam: Can you walk through a concrete example from the paper? How does a single AI capability change risk tiers based on its autonomy and context?

Maya: The authors give a great illustration. Imagine an off-the-shelf assistant built on a foundation model. If you use it internally to prepare meetings, summarize documents, and track actions, it operates on non-regulated data with advisory outputs. That sits at the lowest tier. The generative exposure alone does not escalate it.

Sam: Okay, so what happens if I take that exact same foundational model and give it more power?

Maya: If you wire that same generative capability into a workflow where it executes against a regulated system of record, the autonomy amplifier applies. The system now behaves as a higher-tier deployment and attracts the corresponding validation and oversight. The core capability did not change at all. The use, and therefore the risk, did.

Sam: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Maya: And now, back to the document.

Sam: That is a brilliant way for CIOs to think about it. It forces you to look at the execution boundary. So let's talk about validation. Validating a static machine learning model that just predicts readmission risk is one thing. You test it on a held-out dataset and check its accuracy. But how on earth do you scientifically validate an agent whose behavior emerges dynamically at runtime?

Maya: You have hit on the core scientific challenge. An agent has a dynamic execution path, it uses tools, and any errors it makes can propagate across steps and compound. Therefore, technical evaluation of the underlying model's accuracy and calibration is necessary, but it is no longer sufficient. Evidence must extend to the workflow itself.

Sam: Meaning you have to evaluate the agent actually performing the task in its environment?

Maya: Yes. You have to exercise each key step, assess how errors propagate, and crucially, test the system together with the human who supervises it. A defensible validation program has to move through stages with explicit decision gates, so that progression depends on hard evidence rather than just momentum.

Sam: Alright, let's get into the engineering side. The paper argues that governance cannot just live in policy documents; it has to be embedded into the software so that evidence is produced as a by-product of operation. They outline four specific engineering approaches to build trust. What is the first one?

Maya: The first approach is audit logs and traceability. An agent's trustworthiness is truly only as strong as its audit trail, because we can only govern what we can observe. Traditional application logs were designed for deterministic software. They just record that an interface returned a result. But they tell you nothing about the reasoning that led an agent to call a particular tool.

Sam: So if an agent decides to email a patient, a traditional log just says the email was sent, but it doesn't capture the chain of logic connecting the original intent to that specific action.

Maya: Exactly. Defensible agents require intent-based, identity-linked telemetry. The system must treat the agent as a first-class actor and capture what it did, why it did it, which tools and data it touched, and which model version, prompt, and configuration were active. In regulated environments, this reconstructability is the absolute substrate of data integrity.

Sam: That makes total sense. We need to look inside the black box at runtime. What is the second engineering approach?

Maya: The second is using user interfaces as trust surfaces. Oversight happens at the interface where a person meets the output. If the interface does not communicate uncertainty, surface provenance, or make it easy to inspect what the agent is about to do, human review just becomes a formality.

Sam: We see this all the time in clinical settings. Alert fatigue sets in, and doctors just click approve on whatever the computer says. Automation bias is a huge risk when systems sound highly confident.

Maya: Precisely. The design of the interface is the most direct defense against automation bias. But an interface can only support oversight if the workflow leaves room for it. The authors call this preserving cognitive space. If you deliver an output in a compressed decision window, or bury it among competing alerts, a human cannot meaningfully evaluate it.

Sam: And if the default setting makes overriding the AI slower or harder than just clicking accept, you are quietly pushing the user toward passive confirmation. It has to be just as easy to question the system as it is to accept it. Okay, what is the third engineering approach?

Maya: The third is control by design through guardrails, policy-as-code, and tool-gating. For agents, content filters that moderate text are simply insufficient when the risk is an actual action, like writing to a system of record. Defensible design constrains the execution path using least-privilege access, meaning the agent can only reach what its specific task requires.

Sam: So you gate the tools. You don't let it touch the main database unless it absolutely needs to for that specific micro-task. It is about containing the blast radius if the agent hallucinates a plan.

Maya: Exactly. Containment matters as much as prevention. In multi-agent settings, a single fault can propagate across workflows faster than stepwise human checks can intervene. So you need policy-as-code to prevent prohibited actions before they happen, explicit action boundaries, reversibility for consequential steps, and a highly reliable kill switch.

Sam: A kill switch feels mandatory when software is acting on its own. Let's look at the fourth approach, which sounds a bit like science fiction. They suggest using agents to validate agents. How does that work without creating an infinite loop of AI marking its own homework?

Maya: It is a fair question, but it comes down to scale. The volume of software that agents can produce would completely overwhelm manual validation. Human compliance teams would become a massive bottleneck. So, validation itself must be partly automated to accelerate the work.

Sam: So the agents do the heavy lifting of gathering the evidence, running regression tests, and doing adversarial probing like prompt-injection testing?

Maya: Yes, they map controls to regulatory requirements at a scale and speed no manual process can match. But the discipline that keeps this defensible is that humans remain accountable for the judgment. People must set the acceptance criteria, adjudicate the edge cases, and accept the residual risk. Agents do the labor of producing and organizing evidence, but humans make the final call.

Sam: That division of labor is crucial. The paper says you can't just slap human in the loop on a product and assume it is safe. If the human reviewer lacks the knowledge, time, or authority to intervene, their presence is just providing the appearance of control, not the substance of it.

Maya: Which brings us to the core of ethical defensibility: continuous assurance and human oversight. The paper points out that oversight is not binary. There are graded positions. You can have a human in the loop, where someone approves each output before it takes effect. You can have a human on the loop, monitoring an autonomous system and intervening by exception. Or a human in command, who simply governs the parameters of operation.

Sam: But for any of those positions to actually mean anything, the humans involved have to actually understand the system they are supervising. And this ties into a fascinating legal requirement the document mentions. What is happening with the EU AI Act?

Maya: This is a major quiet obligation that has reframed trust as a workforce question. Article 4 of the EU AI Act, which is the AI literacy provision, has applied since February 2025. It reaches every provider and deployer of AI systems, regardless of the risk tier.

Sam: Every provider and deployer? So a hospital system using an AI tool has a legal mandate to ensure their staff understands it?

Maya: Yes. Organizations are required to take measures so that staff, and people acting on their behalf, have a level of understanding proportionate to their role, the use case, and the people affected. Gartner suggests AI literacy is no longer an HR initiative, but a market-access requirement. If you cannot demonstrate readiness, you risk losing opportunities before AI value is ever realized.

Sam: That is huge. It means you can't just hand a clinician a model card and wish them luck. Transparency artifacts only help if they are paired with actual training and onboarding. If a reviewer doesn't know where the system is most likely to fail, they are just choosing between blind trust and blanket rejection.

Maya: Exactly. And oversight requires decisional authority. The organization has to support people who question, override, or escalate an agent's output. That divergence shouldn't be treated as deviance. Justified disagreement that is documented is what separates real oversight from a symbolic sign-off. Override rates and discordance should be treated as safety signals, not compliance failures.

Sam: Which implies you need an oversight view tracking those metrics: override frequency, review latency, and performance drift across subgroups. Because, as the paper notes, harm is rarely distributed evenly.

Maya: Right. Continuous assurance means maintaining trust over the lifecycle, not just certifying it once. It requires explicit residual-risk acceptance before progression, defined revalidation triggers, and a solid plan for retiring a system when its risks can no longer be managed.

Sam: This has been an incredibly dense but necessary look at the future of healthcare software. To recap, as we move into the agentic era where AI takes autonomous actions, organizations have to shift from just building trustworthy AI to building defensible AI. That means proportionate risk tiering based on use, validating entire workflows instead of just models, and engineering trust directly into the execution path with audit logs, tool-gating, and smart UI.

Maya: And the most memorable takeaway from IQVIA here is that the organizations that pull ahead will not be the ones with the most autonomous agents. They will be the ones that can show their work, whose agents carry a real audit trail, and whose people understand what they are supervising. It is about having defensibility built in rather than bolted on afterwards.

Sam: If you are responsible for buying, building, or regulating these tools, you need to read the full document. We have linked Validating AI Agents and Software in the show notes. And as always, a reminder that we are AI analysts, not your doctors, and this podcast is not medical advice. Thanks for listening.