Who Goes to Jail When AI Fails? Compliance in the Gen AI Era

Sam: Generative AI is supposed to unlock massive economic value for the life sciences industry, but right now, only 5% of organizations have actually realized those benefits.

Maya: That massive gap is exactly what we are unpacking today. We are looking at a white paper published by IQVIA titled AI in Life Science Quality Management and Regulatory Affairs: Global Compliance and Strategic Readiness.

Sam: And just a quick reminder up front for our listeners, our voices are AI-generated, and this podcast is a summary of a publicly available document.

Maya: So, we are talking about Quality Assurance and Regulatory Affairs, or QA/RA. For anyone working in a health system or a medtech company, these teams make sure a product is actually safe to use on a patient, and that it legally stays on the market.

Sam: Right, these are the guardrails of the entire industry. And the document points out that AI has actually been evolving for nearly 70 years, going from basic pattern recognition to advanced machine learning, and now agentic AI that can make autonomous decisions.

Maya: Exactly. Historically, you had to be an engineer or computer scientist to really understand this stuff. But now, touchscreen technology and high-computational-powered devices are accessible to the public. It has become completely mainstream.

Sam: And it is everywhere in life sciences now. The paper lists diagnostics, clinical decision support, medical imaging, digital therapeutics, drug discovery, manufacturing, shipping, and business analytics. But if it is so widespread, why are QA and RA professionals specifically so hesitant to fully embrace it?

Maya: To answer that, IQVIA conducted a poll prior to a webinar they hosted with legal and regulatory experts. They asked healthcare companies what their top concerns were when implementing AI in healthcare QA/RA. The results are fascinating because of what is at the bottom of the list.

Sam: Let's hear the numbers. What did the poll find?

Maya: The lowest concern, by a wide margin, was cost. Only 4% of respondents cited cost as their top concern. The bigger issues were organizational readiness at 20%, and readiness of data at 32%.

Sam: Wait, really? Cost is only 4%? Usually, when we talk about deploying enterprise software or buying new technology in healthcare, budget is the initial thing people fight over in the boardroom.

Maya: It is surprisingly low, but it makes sense when you see the top concern. A massive 44% of respondents said their top concern was regulation.

Sam: Okay, 44% of the respondents are terrified of the regulatory side. Why does regulation completely dwarf cost and data readiness?

Maya: Alex Denoon, a Partner and Head of Life Sciences and Regulatory at Bristows LLP, gave a very blunt explanation in the document. He said it does not surprise him that regulation is the highest priority because, in his words, it is the only criteria that could result in a go-to-jail point.

Sam: A go-to-jail point. That is definitely a way to focus a boardroom's attention.

Maya: Exactly. Healthcare organizations are naturally risk-averse. If your data is messy, your project might fail. If your organizational readiness is poor, you waste money. But if you violate healthcare regulations, the consequences are incredibly severe.

Sam: And Chris Hart, another expert in the paper from Foley Hoag LLP, pointed out that if this were a strictly U.S. audience, that 44% might actually be a little lower, because the U.S. federal government has recently taken a more hands-off approach. But globally, regulation is the giant roadblock.

Maya: Right. Mike King from IQVIA also chimed in on this, saying that if you take a customer-centric approach to applying AI, you literally have to start with the regulations. You cannot build a solution and then try to make it compliant afterward.

Sam: So, let's talk about what these QA and RA teams actually do, and where AI fits in. Because QA/RA sounds very administrative, but it directly impacts patient safety, right?

Maya: Absolutely. These teams use QMS, which is Quality Management System, and RIM, which is Regulatory Information Management solutions. AI can step in to handle massive amounts of data search and extraction.

Sam: Give me an example. What kind of data are they searching?

Maya: Think about post-market surveillance. Once a drug or a medical device is out in the world, companies have to monitor it continuously. AI can mine structured and unstructured data across government recall databases, adverse event reports, and technical service records.

Sam: So instead of a human manually reading through countless clinical notes or complaint logs, the AI is doing the heavy lifting to spot a trend?

Maya: Yes, it provides powerful deep analytics. The goal, as the paper perfectly puts it, is to identify potential deviations in product performance or safety profiles before small sparks become large wildfires.

Sam: That is a great visual. Catching a tiny signal that a specific batch of pacemakers is draining batteries too fast, or a drug is causing a rare side effect, before it harms patients.

Maya: Exactly. And beyond just searching data, AI is now being used for content generation in these heavily regulated spaces. It is creating generative content for human review.

Sam: Wait, AI is writing regulatory submissions?

Maya: It is drafting them, yes. The document lists several areas where generative AI is creating content for QMS and RIM processes. This includes global regulatory submissions, product registration activities, adverse event reports, and recall or field action notifications.

Sam: That is wild. I can see why the regulatory compliance anxiety is so high. If an AI hallucinates a data point in a global regulatory submission, the fallout could be disastrous.

Maya: Which is exactly why the document stresses that human oversight is absolutely essential. AI is also drafting product labeling, packaging, marketing literature, and CAPA summaries. CAPA stands for corrective and preventive action. It is also doing audit summaries.

Sam: So if you are a QA professional, AI takes the pressure off by acting as a highly efficient drafting tool and data miner. It accelerates process efficiency and improves the quality of performance.

Maya: Right, and this frees up QA/RA professionals to dedicate more time to actual product quality, patient safety, building relationships with regulators, and market access activities. It drives commercial performance alongside patient-centric solutions.

Sam: Okay, let's dig into the motivations here. Why is there such an enormous amount of interest in adoption right now? Obviously, efficiency is part of it.

Maya: Chris Hart breaks down the motivations. Initially, it is about doing your job more efficiently to drive the bottom line and get the most out of a workforce. Another motivation is doing the job better. And finally, pure competition.

Sam: Pure competition meaning fear of missing out?

Maya: Essentially. He notes that if you do not adopt generative AI tools in some fashion, you are going to be left out as a competitive player in the marketplace.

Sam: Alex Denoon backed that up too. He said he works in the life sciences sector exclusively, and every single large client he has is already well and truly invested in this space. They are either currently deploying, about to deploy, or investigating where to best deploy generative AI.

Maya: He actually said he doesn't think there is a single problem that is too complex, too trivial, too simple, or too multifaceted that AI cannot help with.

Sam: That is a huge endorsement from a regulatory lawyer. But he also said the particular difficulties relate to how you direct it, govern it, control it, and the guardrails you put around it.

Maya: And those guardrails have to navigate a very complex regulatory landscape. The paper makes a critical distinction between vertical regulations and horizontal regulations.

Sam: I noticed that. How do they define the difference between vertical and horizontal in this context?

Maya: Vertical regulations are the specific rules that apply to your exact industry. So for life sciences, this means the regulations governing medical devices, in vitro diagnostics, pharmaceuticals, and biologics.

Sam: So that is the FDA approval process, clinical trial standards, things that only apply to healthcare.

Maya: Exactly. Horizontal regulations, on the other hand, step broadly across all industries. This includes data protection, environmental health and safety, and now, AI.

Sam: Because an AI regulation applies to a bank just as much as it applies to a pharma company. So a life sciences company now has to cross-reference their vertical medical regulations with these new horizontal AI laws.

Maya: And the horizontal AI laws are moving incredibly fast. The document highlights the global AI regulatory landscape, noting that significant developments have occurred globally in just the last few years.

Sam: The big framework is Europe, right? The EU AI Act.

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: Yes, the EU AI Act came into force in August 2024. Then you have the UK, which published a pro-innovation white paper on AI regulation in March 2023, and hosted the initial AI Safety Summit in November 2023.

Sam: What about the U.S.? Chris Hart mentioned earlier that the federal government has a hands-off approach compared to Europe.

Maya: Hart says there has been a real shift at the federal level from a focus on safety to a focus on innovation and competitiveness. However, products that make guarantees about safety, data security, or accuracy are still subject to consumer protection laws enforced by federal agencies.

Sam: So if an AI tool promises to be accurate in a clinical setting and it fails, the federal agencies can still go after them for violating those representations.

Maya: Exactly. But Hart points out that the really interesting question is what state regulation looks like. States are making various attempts to regulate AI across industries, focusing on transparency, governance, registration, or consumer rights. It is very much a work in progress.

Sam: That sounds like a compliance nightmare for a global medtech company. You have the strict EU AI Act on a global scale, and then a patchwork of U.S. state laws.

Maya: It is complex, but Alex Denoon offers a silver lining. He says Europe loves to regulate, so if you are compliant with their astonishingly broad frameworks like GDPR and the EU AI Act, you get a huge chunk of the world, because many countries piggyback off European frameworks.

Sam: Oh, that makes sense. If you build your AI to survive the toughest European audits, it will probably be readily usable in other jurisdictions.

Maya: Exactly. The European frameworks do actually deliver a degree of certainty, even if they are rigorous. But the innovation of AI within life sciences seems to be evolving at a higher rate than regulatory bodies can keep up with.

Sam: Which brings us back to risk. If the technology is moving faster than the law, how do companies protect themselves from the risks associated with generative AI? The paper explicitly mentions hallucinations, bias, and outdated information.

Maya: Chris Hart emphasizes the need for protection against algorithmic bias. You have to figure out how to control for bias in the training inputs as well as in the use of outputs.

Sam: Because if the training data only represents a narrow demographic, the AI's clinical insights or safety trend analysis will be fundamentally skewed.

Maya: Right. Hart says this has a governance consequence. You have to constantly assess these tools, assess the outputs coming from them, how they are being used, and in what ways they may create prejudicial bias.

Sam: But how do you manually monitor countless AI-generated adverse event reports or CAPA summaries every day? That defeats the purpose of the efficiency gains.

Maya: This is a fascinating point in the document. Alex Denoon says that AI tools are actually the best at monitoring whether another AI tool is going awry.

Sam: Wait, using AI to babysit AI?

Maya: Yes. He says they are much better than humans at monitoring them. All of his very large clients use multiple different tools all day, every day, to cross-interrogate the tools they are using to triangulate back.

Sam: That is incredible. You basically have an AI model generating the regulatory dossier, and an entirely separate AI model acting as the auditor to flag hallucinations or biases.

Maya: It is a brilliant strategy, but it requires a very mature organizational setup. And remember, 32% of companies said readiness of data was their top concern. The paper states that companies are beset by data quality, access, and bias challenges.

Sam: Right, if your foundational data is messy, having multiple AI models cross-interrogating each other just creates a very fast, very efficient mess.

Maya: Exactly. Effective AI solutions require well-prepared and curated data. The organization needs a level of data literacy and AI literacy to drive sustained, long-term benefits.

Sam: So how does a company actually implement this without falling apart? Does the paper offer a framework for adoption?

Maya: It does. They recommend a layered AI-implementation strategy. It starts with leading with the customer problem statement and solutioning outwards, choosing the right tool for the job.

Sam: Okay, what are the layers?

Maya: Initially, define the problem. Next, evaluate organizational readiness. Then, understand the applicable vertical and horizontal regulations and standards, making sure to accommodate for global variance.

Sam: That covers the compliance piece we were just talking about.

Maya: Right. Then, solution outwards with those variables in mind, considering the cost-effectiveness and choosing the right type of AI, if any AI is actually suitable for the original problem.

Sam: I like that they add if any. Sometimes a simple database query is better than a massive language model.

Maya: Exactly. After that is to assess data readiness, looking specifically at congruence, volume, specificity, and quality. And finally, support the whole thing with core quality methodology and continuous improvement approaches.

Sam: And the key pillars supporting this entire framework?

Maya: Transparency, Governance, and Trustability. You need clear governance structures, documenting exactly where and how AI is used, and having accountable individuals or teams overseeing the AI activities.

Sam: You need a human to take the blame if something goes wrong, essentially.

Maya: Essentially, yes. You also need detailed documentation of the rationale behind AI deployments and decisions. This is crucial for regulatory compliance and for defending decisions if they are challenged.

Sam: So if a company follows this layered strategy, where should they actually start? Because overhauling a global regulatory submission process sounds terrifying for an initial pilot project.

Maya: Alex Denoon has a very clear answer for that. He says the clients who get the biggest bang for their buck already are the ones using AI in the QA/RA space to help with advertising and claims management.

Sam: Advertising and claims management? That makes sense. Reviewing marketing materials to make sure they do not make illegal medical claims is incredibly time-consuming, but the risk of physical harm to a patient is lower than an AI adjusting a manufacturing process.

Maya: Exactly. Denoon says you generate a lot of benefits and a lot of productivity very quickly in that area. It allows teams to track and monitor the impact of internal and external communications across a range of social media and electronic channels.

Sam: That is a perfect quick win. Build trust in the AI with marketing compliance before you let it draft your adverse event reports.

Maya: Mike King echoes that sentiment. His advice is to focus clearly on a few key use cases that offer the most value for your business. Define the problem statement, deliver the value, and then reinvest the benefits back into new projects.

Sam: Do not try to boil the ocean.

Maya: Right. King reminds us that AI is just a single tool, albeit a brilliant one. The true deliverable is always safe and effective, global patient care.

Sam: And what is the final warning from the legal side as companies embark on this?

Maya: Chris Hart says we should expect the regulatory environment to shift in unexpected ways. Some jurisdictions will be very aggressive, while others will want to be innovative. His core advice is not to assume that tomorrow is going to look like today.

Sam: That is a great place to land. So, let's recap the core takeaways. AI in life sciences is no longer an emerging tech; it is mainstream. But 44% of companies see regulation as their biggest hurdle.

Maya: To get over that hurdle, organizations need to master both vertical medical regulations and horizontal AI regulations like the EU AI Act. They must ensure data readiness, establish transparency and governance, and carefully select high-value use cases to start, like advertising and claims management.

Sam: And perhaps the most memorable takeaway: when you need to monitor an AI tool in a highly regulated space, the best tool for the job is often another AI tool.

Maya: For anyone leading a QA or RA team, or working for a company like IQVIA which has 88,000 employees in over 100 countries helping to navigate this, the message is clear: connect data, technology, and expertise to realize AI's potential safely.

Sam: As always, the source document is linked in the show notes, and remember, this is not medical advice.