22 November 2026 · 20 min
Will the FDA Clear Your AI? Managing the Lifecycle of Generative AI in Health Care
We explore the FDA's executive summary on how generative AI models will be regulated throughout their entire product lifecycle. Listeners will learn how the agency differentiates generative AI from traditional machine learning, the challenges of bounding intended use when algorithms can hallucinate, and what evidence device makers will need to provide both before and after an AI tool hits the market.
Key points
- The FDA has yet to authorize any unlocked AI-enabled device, signaling a cautious approach to continual machine learning.
- Generative AI introduces novel regulatory challenges because its open-ended outputs make it difficult to strictly bound a device's intended use.
- The FDA draws a parallel between incorporating third-party foundation models and the use of off-the-shelf software, meaning the device manufacturer ultimately bears responsibility for safety.
- Premarket submissions for generative AI will likely require highly specific details, such as model parameters like temperature and top-p, as well as prompting strategies.
- Postmarket monitoring is critical; manufacturers must track real-world performance to catch data drift and hallucinations.
Source: Total Product Lifecycle Considerations for Generative AI-Enabled Devices - U.S. Food and Drug Administration, 2024
This spot is available. Reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsor the show
This episode is an AI-generated conversation summarising a public document; the hosts' voices are synthetic. It is for information only and is not medical advice. Always refer to the original source.
Transcript
Sam: Here is a reality check for the health tech world: the agency has yet to authorize any unlocked AI-enabled device. That is a massive line in the sand for the future of generative artificial intelligence in medicine.
Maya: Today we are unpacking the document "Total Product Lifecycle Considerations for Generative AI-Enabled Devices", which was published by the U.S. Food and Drug Administration. It is an executive summary prepared for a Digital Health Advisory Committee meeting.
Sam: And just a quick reminder right at the top, our voices are AI-generated, and this episode is a summary of a publicly available document.
Maya: This document exists because the interest in generative AI tools across the health care sector has expanded rapidly, but there remain open questions on the approach to regulating these products. The FDA has long promoted a total product life cycle approach, often referred to as TPLC, to oversee medical devices. But generative AI brings entirely new complexities.
Sam: Let us start with the basics. How does the FDA actually differentiate generative AI from the artificial intelligence we have been seeing in health care for the last decade?
Maya: Before generative AI, developers were already incorporating artificial intelligence and machine learning models into medical devices. The document points out that those non-generative models are used for specific tasks like image segmentation, classification, biomarker extraction, and risk prediction. They are generally considered reproducible, even if they are not fully transparent.
Sam: Right, so traditional AI is mostly about finding patterns to make accurate predictions. It looks at a scan and highlights a potential tumor based on patterns it has seen before.
Maya: Exactly. Generative AI, on the other hand, refers to the class of models that mimic the structure and characteristics of input data to generate derived synthetic content. This can include images, videos, audio, text, and digital content. Unlike older models, generative AI is meant to create new data that resembles what it learned from.
Sam: And that is where the regulatory headache begins, right? Because these generative models are usually built on top of foundation models, which are trained on datasets so massive that human developers cannot possibly know everything inside them during development.
Maya: That is a critical point in the FDA's analysis. Foundation models are typically not created for an individual product, nor are they generally intended for use as a device as that term is defined in section 201(h) of the Federal Food, Drug, and Cosmetic Act. They derive highly complex relationships between different words, pixels, or elements of language found in the development data. So their output reflects probabilistic predictions.
Sam: Wait, really? So the foundation model itself is usually not a medical device, but an application developer comes along, adapts that single foundation model for a specific health care application, and suddenly it might fall under FDA jurisdiction?
Maya: Yes. The agency uses the term "GenAI-enabled device" to refer to a device in which generative AI methods or models are integral to the device's output or functionality. And to figure out if a product meets that definition, the FDA first looks at the intended use. Intended use is defined as the general purpose of the product or its function.
Sam: And intended use is exactly what generative AI makes so slippery. The document brings up the risk of hallucinations. The glossary also refers to this as confabulation. It defines it as the generation of erroneous or false content to meet the programmed objective of fulfilling a user's prompt. How does the FDA view that risk in a clinical setting?
Maya: The FDA notes that hallucinations can appear to be authentic outputs. In health care applications where highly accurate, truthful information is critical, this presents a significant challenge. When insufficiently controlled, the ability of generative AI to tackle diverse, new, and complex tasks contributes to uncertainty around the limits of a device's output. That makes it difficult to confirm the bounds of a device's intended use.
Sam: That makes total sense. If I build an AI that is only supposed to draft a clinical summary, but a doctor types in a weird prompt and the AI spits out a diagnostic recommendation, it just jumped the fence. It started acting outside its intended use.
Maya: The document gives a very clear example of this. Consider a mobile application intended to let a user enter an herb and a drug they want to take concurrently, and the app provides information about known interactions from the literature. Under the policy for device software functions, that is generally software for which the FDA intends to exercise enforcement discretion, meaning they pose lower risk to the public and the FDA does not intend to enforce requirements under the FD&C Act.
Sam: Okay, so a standard search app for drug interactions gets a pass from strict enforcement.
Maya: Right. But if that exact same product is enabled with generative AI, and it is not sufficiently controlled, it could generate output beyond its intended use. It might direct a patient to an alternative treatment, or it might generate erroneous or false content regarding the interactions. The FDA says it is important to consider whether implementing generative AI introduces potential uncertainty or risk into a product that would otherwise have been considered low risk.
Sam: So just adding a large language model to a basic tool could suddenly force you under the FDA's microscope. That is a massive takeaway for developers and hospital IT buyers. You cannot just sprinkle generative AI onto a low-risk tool and expect it to stay low-risk.
Maya: Exactly. The FDA explicitly states that certain software functions are excluded from the device definition by section 520(o) of the FD&C Act, like software intended for administrative support of a health care facility or for encouraging a healthy lifestyle. But if the functionality could pose a risk to a patient's safety if it fails to function as intended, it becomes a device that is the focus of FDA regulatory oversight.
Sam: There are some broad applications mentioned here too. They talk about clinical documentation, obtaining insurance pre-authorization, and facilitating medical training with realistic patient simulations. Some of those sound like they might not be devices at all.
Maya: That is correct. Products intended for health care professionals to use as educational tools for medical training are likely not devices. However, the document also mentions research describing the utility of generative AI models in generating a clinical diagnosis, or the potential to diagnose and treat mental health conditions through chatbots. Those applications may indeed meet the definition of a device.
Sam: And for the ones that do meet the definition, how does the FDA actually classify them? They mention three classes, right?
Maya: Yes, devices are assigned to Class I, Class II, or Class III, with increasing levels of regulatory control. They may be subject to premarket review under a few pathways: Premarket Approval, or PMA, when the device is high risk; a De Novo Classification Request for low to moderate risk with no legally marketed predicate; or Premarket Notification, commonly known as 510(k), when the device is low to moderate risk and there is a legally marketed predicate device.
Sam: The 510(k) pathway seems like it would be a nightmare for generative AI. To get cleared under a 510(k), you have to prove your device is substantially equivalent to something already on the market.
Maya: That is exactly the challenge the document highlights. Section 513(i) of the FD&C Act requires a comparison between the intended uses and technological characteristics of the predicate device and the subject device. The FDA notes that if a non-generative AI device is used as a predicate for a generative AI device, it is likely the generative AI device could be found to have different technological characteristics, and that those characteristics raise different questions of safety and effectiveness.
Sam: Because the underlying technology is so fundamentally different. You are moving from a locked pattern-matching algorithm to a system that generates novel text or images probabilistically. So how does a company actually prove this new technology is safe?
Maya: The FDA frames this around valid scientific evidence across the total product lifecycle. But a major hurdle is that device manufacturers often do not build the foundation models they rely on. The FDA compares this to incorporating off-the-shelf software into medical devices. They reference their guidance titled "Off-The-Shelf Software Use in Medical Devices".
Sam: And what does that guidance say about using third-party code?
Maya: It says that the medical device manufacturer using off-the-shelf software generally gives up software life cycle control, but still bears the responsibility for the continued safe and effective performance of the medical device. The FDA warns that in the context of generative AI, an analogous lack of software lifecycle control over an incorporated foundation model raises challenges.
Sam: That is a huge liability warning for startups. You might be licensing a foundation model from a massive tech company, but if they change their model's overall data composition, it could impact your device's intended use or introduce performance bias. And the FDA says the device manufacturer is the one left holding the bag.
Maya: Precisely. And it brings us back to that crucial distinction in the document: locked versus unlocked models. The glossary defines a locked model as one that provides the same output each time the same input is applied and does not change with use. An unlocked model, which involves continual machine learning, adapts its performance by incorporating new data or experiences over time.
Sam: And as we mentioned at the start, the FDA states clearly that they have yet to authorize any unlocked AI-enabled device. That means if a third-party foundation model is incorporated in an unlocked state, it raises new and significant questions for the FDA review.
Maya: To get any of these devices authorized, manufacturers are going to need a robust premarket evidence package. The FDA notes that the level of detail needed regarding the underlying device design and performance testing may be greater. They want information on the foundation model's attributes, architecture, training methods, and datasets.
Sam: Which, again, is not trivial if you do not own the foundation model. Most of those big models do not openly disclose their billions of parameters. How specific is the FDA getting about the design details they want to see in a submission?
Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.
Sam: And now, back to the document.
Maya: Very specific. The document lists examples of design details they may expect. This includes design specifications like attention mechanisms and model merging. It includes model parameters like temperature, top-K, and top-p. And it includes built-in prompting strategies like in-context learning or zero-shot prompting, as well as the prompting capabilities available to the end user.
Sam: Wow. So if you are a regulatory affairs professional listening to this, you need your engineering team to pull exact model parameters like temperature and top-p for your submission. You cannot just say, 'we use a large language model and it works.' You have to map out exactly how the model is constrained.
Maya: Exactly. Because generative AI models have open-ended inputs, the FDA says it is unreasonable to evaluate every conceivable input the model might encounter during deployment. They need to understand the guardrails. They also highlight the need for new evaluation methodologies. For instance, current quantitative approaches may need to be complemented with new qualitative metrics.
Sam: In the panel questions section, they actually ask about specific performance metrics for multimodal models, right? Because evaluating a chatbot requires different math than evaluating an image scanner.
Maya: Yes, they note that metrics will typically vary with the device's intended use. For generative text, they mention metrics like perplexity and quantitative comparison to reference text. For generative images, they list Frechet Inception Distance and Structural Similarity Index Measure. For generative audio, they suggest Log-Spectral Distance and Perceptual Evaluation of Speech Quality.
Sam: Those are highly specialized data science metrics. It shows the FDA is staffing up on deep technical expertise. They are not just looking at clinical outcomes; they want to see the underlying mathematical stability of the generated synthetic content.
Maya: And that stability has to be maintained over time, which brings us to the total product lifecycle approach. The FDA outlines an AI Lifecycle framework that builds on modern Software Development Lifecycles, which have been used since the 1960s. The AI Lifecycle includes seven main phases.
Sam: Let us walk through those phases. It sounds like they are laying out a literal blueprint for how a medical AI company should operate.
Maya: The first phase is Planning and Design, where goals, scope, and architecture are defined. Next is Data Collection and Management, to ensure the quality and relevance of the data. The third is Model Building and Tuning, which includes selecting algorithms and feature engineering. Fourth is Verification and Validation, to assess accuracy and reliability.
Sam: And then we get to the real world. Phase five is Model Deployment. What comes after that?
Maya: Phase six is Operation and Monitoring, which involves troubleshooting, bug fixing, and making necessary updates. The final phase is Real-World Performance Evaluation, assessing aspects like accuracy, efficiency, scalability, robustness, and user satisfaction in real-world conditions.
Sam: That Real-World Performance Evaluation phase seems like the make-or-break step for generative AI. Once this tool is deployed in a hospital, how do you ensure it does not start hallucinating because the patient population shifted?
Maya: The FDA identifies that exact problem in the glossary. They call it data drift, which refers to the change in the input data distribution a deployed model receives over time. This can happen due to variations between medical practices, or changes in patient demographics and disease trends. To combat this, the FDA says robust postmarket monitoring strategies are critical.
Sam: What kind of postmarket strategies are we talking about? Just waiting for a doctor to complain?
Maya: Far more proactive than that. All medical devices are subject to general controls, which include medical device reporting under 21 CFR Part 803, reports of corrections and removals under 21 CFR Part 806, establishment registration and device listing under 21 CFR Part 807, and quality system regulation under 21 CFR Part 820.
Sam: Those are the baseline rules for anyone making a medical device.
Maya: Right. But for certain Class II devices, the FDA may require special controls unique to generative AI devices. This could include mandatory postmarket monitoring of device safety and performance, or specific notification requirements if the device is not performing as intended.
Sam: The panel questions section brings up another massive postmarket challenge. The FDA asks what strategies can be implemented to monitor a generative AI device across multiple sites, ensuring consistency and addressing potential regional biases.
Maya: Yes, and they also ask about a multi-layer application design. Specifically, they want to know what methods and metrics can be utilized to effectively monitor and evaluate a device that queries external consumer-grade AI services that are not themselves medical devices.
Sam: That is wild. So an app clearing the FDA might secretly be passing data to a general-purpose chatbot API in the background. No wonder the agency wants to know how to monitor that. If the API provider updates their consumer model, the medical device could suddenly behave completely differently.
Maya: Exactly. The agency states that continuous, robust generation of evidence throughout the premarket and postmarket setting as part of a learning health care system is important. It is the only way to continuously refine our understanding of how these devices are used and how they perform.
Sam: It is also worth noting how much emphasis the FDA puts on transparency for the end user. The document talks about how a generative AI model can use free text or multimodal inputs, which is very different from the structured data formats used by non-generative models.
Maya: That flexibility is a benefit, but also a risk. The FDA says it is important to ensure transparency to the end user. This might include clear information on the device design, how the device was tested, the level of autonomy, and if or how it incorporates human-in-the-loop controls. Users need to understand how interacting with the device through prompt engineering impacts the output.
Sam: That is a huge point for clinicians. A doctor needs to know that if they phrase a question slightly differently, they might get a completely different clinical recommendation. The technology is non-deterministic. It does not just spit out the same answer every time like a calculator.
Maya: Which is why the FDA insists on understanding the fine-tuning process. The steps taken to fine-tune a foundation model into a specific adaptation are what ultimately govern the device's intended use. They also mention watermarking in the glossary, defined as embedding information into outputs created by AI for the purposes of verifying the authenticity of the output or its provenance.
Sam: Let us step back and look at what this means for the broader market. For hospital leaders and people in the boardroom buying this software, what is the practical takeaway from this document?
Maya: For buyers, the takeaway is that you need to ask vendors hard questions about their postmarket monitoring. Do not just ask if the tool is FDA cleared. Ask how they are monitoring for data drift in your specific patient population. Ask if their foundation model is locked or unlocked, and what happens to their FDA clearance if the third-party model provider updates the underlying system.
Sam: That is brilliant. If an AI vendor licenses a model from a big tech company, and that tech company silently updates its weights, the clinical performance in your hospital could change overnight. The FDA is clearly putting the burden of tracking that on the medical device manufacturer.
Maya: And for developers and companies building these tools, the message is that the FDA's regulatory oversight is risk-based, taking into consideration the product's intended use and technological characteristics. If you cannot bound your intended use because your chatbot is too creative, you are going to have a very difficult time getting authorized.
Sam: You really have to constrain the model. Use prompt engineering, use quality control on inputs and outputs, maybe use topic-specific foundation models rather than general ones. Whatever it takes to prove to the FDA that your device will do what you say it does, and only what you say it does.
Maya: Precisely. The FDA notes that using generative AI to tackle diverse, new, and complex tasks may contribute to uncertainty. As with all medical devices, applying the right tool for each task is essential. Sometimes, generative AI might not be the best technology for a specific intended use if it introduces too much risk of erroneous content.
Sam: That is a refreshing bit of common sense from the regulator. Just because you can use a large language model does not mean you should, especially if a simpler, locked algorithm can do the job without the risk of hallucination.
Maya: The document ultimately shows that the FDA is actively working to bridge the gap between their traditional lifecycle approach and the rapid pace of generative AI development. They are asking experts for input on performance evaluation, risk management, and postmarket monitoring strategies to ensure these devices remain safe and effective.
Sam: To wrap up: if your generative AI tool acts like a medical device, the FDA intends to regulate it like one. Manufacturers have to take responsibility for third-party foundation models, prove safety with very specific model parameters, and monitor relentlessly for data drift in the real world.
Maya: You can find a link to the FDA's source document in our show notes. And as a reminder, this podcast is for informational purposes only and is not medical advice. Thanks for listening!