Credentialing the AI Doctor: The FDA's Blueprint for Generative AI

Sam: The single most fascinating idea here is that regulators might stop trying to test medical software line by line, and instead start credentialing generative AI the same way we evaluate human doctors.

Maya: Today we are looking at the Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback, published by the U.S. Food and Drug Administration.

Sam: And before we dive in, a quick reminder that our voices are AI-generated, and this is a summary of a public document.

Maya: This paper is a big deal for anyone building or buying healthcare technology. The Center for Devices and Radiological Health, or CDRH, wants to figure out how to handle medical devices that use generative AI. And they are very clear upfront: this is just for discussion. It is not draft guidance, it is not final guidance, and it is not a policy change. They are asking the industry for early input.

Sam: Right, they want to start a conversation about risk assessment, premarket evaluation, and postmarket monitoring. And the reason they need a totally new approach is because generative AI breaks all the old rules of software. Traditional software takes a fixed input and gives a fixed output. Generative AI accepts open-ended inputs, does multiple subtasks, and produces variable outputs to similar inputs.

Maya: Exactly. The FDA points out that these tools can offer more personalized outputs and handle complex decision-making. But those same flexible characteristics introduce unique risks. The document specifically mentions confabulations (or hallucinations) that look completely authentic to the user. There is also uncertainty about the bounds of the device's intended use, and the risk that performance might degrade across different test environments and real-world applications.

Sam: And a lot of these tools are built on third-party foundation models. The FDA notes that these third parties offer varying levels of transparency about their training data and architecture, which makes it really hard to figure out if an error is coming from the medical device itself or the underlying model.

Maya: Which is exactly what they discussed in November 2024 at a public advisory committee meeting of the Digital Health Advisory Committee. The executive summary for that meeting highlighted how hard it is to apply a risk-based approach to classifying these devices, and how difficult it is to figure out what constitutes valid scientific evidence to prove they are safe and effective.

Sam: Because you cannot possibly test every conceivable input a large language model might encounter. It is impossible.

Maya: Right. So to tackle this, the FDA is floating a two-axis framework to help assess the risk of a software function. Picture a graph. The horizontal axis represents the activity performed by the function, increasing in independence from left to right. The vertical axis represents the consequences, or the severity of harm if you rely on an incorrect output. Risk increases from the bottom left to the top right.

Sam: Let us map that out. What sits on the left side of the activity axis?

Maya: On the far left, you have functions that provide simple, non-directive information. The paper uses the example of a risk score for a future cardiovascular event. It just gives you information. Further to the right, you get into action-directing functions. This might be a strongly worded patient-facing recommendation about seeking emergency care.

Sam: But the FDA points out that action-directing is not a simple yes or no binary, right? It is a continuum.

Maya: It is a massive continuum. An informational function could just say that lisinopril dosages are sometimes increased when blood pressure stays high. That is general information. Or it could say that in situations like yours, clinicians often increase the dosage. Or it could explicitly say, I recommend increasing the lisinopril dosage. Or, at the extreme end, it gives a specific instruction to increase the lisinopril from 10 mg to 20 mg daily.

Sam: And the FDA is wondering if the directiveness depends on the context, not just whether the AI uses words like should or recommend. They also wonder if slapping a talk to your doctor or I am not a medical professional warning on the screen actually makes the tool any less directive.

Maya: Which leads us to the vertical axis: consequences. Even if a tool is highly directive and tells you to do something, the risk might be low if the consequence is minor. Directing a patient to put over-the-counter hydrocortisone cream on poison ivy is very different from directing a patient to change their basal insulin dose.

Sam: And things get really tricky when we look at who is looking at the screen. The FDA breaks down patient-facing versus clinician-facing tools.

Maya: Yes, clinicians are trained to recognize limitations and integrate AI output with other clinical data. Patients might not be able to identify an incorrect output. They might delay needed care or try inappropriate self-treatment. So patient-facing tools might shift higher on the consequences axis. But the FDA also acknowledges the flip side. They want feedback on how to balance this without falling into medical parentalism that underestimates what patients are capable of.

Sam: They also draw a line between generalist doctors and specialists. If an AI gives specialist-level advice to a generalist, it could be amazing for access to care. But if that generalist lacks the expertise to independently evaluate a complex specialist output, that elevates the risk.

Maya: There are other fascinating risk modifiers the FDA is asking about. Multi-turn conversations, for instance. A chatbot might start out giving non-directive information, but over the course of an exchange, it migrates into giving action-directing advice. You have to assess the risk across realistic conversational trajectories.

Sam: And then there is care escalation, like an AI deciding if a kid with a sore throat needs the emergency room. The FDA says we have to look at both directions of error. Under-escalation delays treatment. Over-escalation causes anxiety, wastes hospital resources, and erodes patient trust.

Maya: Which brings us to measurement and signal processing functions. These might just give non-directive info, but the user cannot independently verify how the AI got its answer. The FDA suggests this might push the risk higher on the consequence axis. They reference the clinical decision support exclusion under section 520(o)(1)(E) as being consistent with this idea.

Sam: So if the risk framework tells us how dangerous a tool could be, how does the FDA actually propose evaluating it before it hits the market? This is where that residency analogy comes in.

Maya: Right. Because exhaustive software testing will not work, the FDA is considering a competency-based approach. The paper cites authors who argue that generative AI is too broad and opaque for current frameworks, and suggest a model based on medical training, licensure exams, supervised practice, and periodic reevaluation.

Sam: But with the caveat that when a human doctor messes up, they face professional, legal, and reputational consequences. We need analogous mechanisms for flawed AI.

Maya: Exactly. So the FDA's proposed competency-based approach has two main parts: device benchmarking and clinical confirmation. Benchmarking evaluates whether the device demonstrates the clinical knowledge, safety behavior, and communication skills needed for its intended use. And this applies to the final user-facing device, not just the foundation model standing on its own.

Sam: The document breaks benchmarking down into specific elements. Let us look at safety first.

Maya: The safety elements include safety-critical recognition and escalation. Does the device escalate appropriately when it sees a critical clinical state? It also includes scope maintenance and boundary adherence. Does the AI know when a request is outside its intended use and refuse it? Both under-refusal, like diagnosing when it should not, and over-refusal, declining valid requests, are considered failures.

Sam: They suggest testing this with adversarial prompting, emotional-manipulation scenarios, and seeing if a multi-turn conversation drifts out of scope.

Maya: Another safety element is calibration, uncertainty communication, and clinical deferral. If the AI presents outdated or contested information with false confidence, that is a safety failure. It needs to know when to defer to a human.

Sam: After safety comes clinical proficiency. What does that entail?

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: Clinical knowledge and task fidelity means having accurate, current knowledge for the intended scope, including contraindications and population-specific considerations like pediatric or geriatric patients, or those with renal or hepatic impairment.

Sam: They also want to test information gathering and analysis. Can the AI handle incomplete or ambiguous information without jumping to premature diagnostic closure?

Maya: Yes, and quantitative and measurement analysis. Can it handle weight-based dosing or unit conversions? Can it spot transcription errors or unit confusion when a patient reports something conversationally? And finally, communication quality. Is it avoiding coercive or emotionally manipulative language, and is it appropriate for the user's health literacy?

Sam: The last big benchmarking category is generalizability. Will it work consistently if a patient uses non-standard dialects, accents, or colloquialisms? The FDA mentions that variation in non-safety-critical phrasing might be okay, but variation in safety-critical behaviors like refusal or diagnostic conclusions is a failure.

Maya: To pull all this off, sponsors might use synthetic data generation and simulation tools like virtual patient avatars to automate the testing. But the FDA acknowledges that publicly available benchmarks have issues with data contamination, saturation, and a lack of real-world representativeness. So sponsors might need to build specialized datasets.

Sam: And the FDA states that any scoring rubrics used in testing need to be grounded in established clinical guidelines, and expert adjudicators need to be structurally independent from the device sponsor. Even if the expert adjudicator is another large language model, those independence considerations still apply.

Maya: That is a fascinating detail. But no matter how good the benchmarking is, it is not enough. That brings us to part two: clinical confirmation. The FDA anticipates benchmarking may not fully establish how the device performs in real clinical use with real workflows.

Sam: But they say clinical confirmation might not require a prospective clinical study in every case, right?

Maya: Correct. They outline a spectrum of approaches based on risk. The lowest tier might be retrospective evaluation on real patient inputs. Next is shadow deployment, where the AI runs in a live clinical workflow, but its outputs are hidden from clinicians and do not affect care. We just record what it would have done.

Sam: Then you could use standardized patient interactions with trained actors. Or clinician adjudication of real cases, where independent clinicians review the AI's outputs. And at the highest risk levels, you would need a prospective clinical study, maybe even a randomized controlled clinical trial.

Maya: A huge open question the FDA has for the industry here is about comparators. When you evaluate an open-ended output, there usually is not a single correct answer. So what is the standard? Do you compare the AI to the consensus of a panel of qualified clinicians reflecting the standard of care? Or do you compare it to a median clinician in practice?

Sam: They even ask if performance could be assessed against what would happen in the absence of the device. Like comparing it to unaided clinical judgment, delayed specialist review, or no intervention at all.

Maya: They are also thinking about the role of independent third parties. The FDA points to existing programs like the Accreditation Scheme for Conformity Assessment, or ASCA, and the Medical Device Development Tool program, or MDDT. They are asking if independent third parties could maintain sequestered evaluation datasets, act as expert adjudicators, or even issue certificates that the FDA could consider in its review.

Sam: Which sounds great, but they also ask how to ensure that third-party participation does not limit competition or prevent innovation.

Maya: Exactly. Let us shift to the postmarket phase. The paper acknowledges that generative AI devices may undergo continuous adjustments following deployment. Because of this, the FDA is considering whether it is appropriate to accept greater premarket uncertainty by relying more heavily on postmarket monitoring.

Sam: That is a massive shift. Approving a device with a bit more uncertainty upfront because you are going to watch it like a hawk in the real world. How do they propose doing that?

Maya: They suggest approaches like periodic device benchmarking, where the sponsor reassesses the device against those pre-specified thresholds on a defined cadence, or after a triggering event like a model update. They also suggest periodic sample-based clinician review and performance degradation monitoring to catch things like drift.

Sam: And they mention that machine-based supervisory agents might facilitate some of this postmarket monitoring.

Maya: Yes, but they are asking what considerations are needed to ensure the reliability of the supervisory agent itself. The FDA also stresses shared ecosystem responsibility. Clinicians, healthcare institutions, payers, and professional societies all have roles to play in deployment and monitoring.

Sam: But they ask how to structure those roles without diffusing manufacturer accountability. When a model changes post-deployment, who is responsible? Some changes are active updates by the sponsor. Some are passive, continuous learning. And some are unplanned changes because a third-party foundation model got updated.

Maya: To handle changes, they suggest a Predetermined Change Control Plan, or PCCP. This could allow certain modifications without a new premarket submission. If a device was benchmarked before authorization, it could be re-benchmarked against the same capabilities after a modification.

Sam: But what if the change is initiated by a giant tech company updating the underlying foundation model? The medical device manufacturer might have no control over that.

Maya: That is a critical point. The FDA is floating the idea of Foundation Model MAFs, leveraging the existing Device Master File program. Foundation model developers could voluntarily submit structured model cards or system cards directly to the FDA. These files would be held confidentially. Device sponsors could then reference these master files, with permission, in their premarket submissions.

Sam: So the big tech companies would not have to publish all their safety secrets publicly, but the FDA review teams could look at the master file to better understand the base model. But the FDA explicitly notes that a Foundation Model MAF is not an authorization of the model itself. The device sponsor is still responsible for proving their specific device is safe.

Maya: Exactly. They are asking the industry if voluntary Foundation Model MAFs are practical, and given that participation is voluntary, what would incentivize developers to do it.

Sam: Finally, the paper touches on agentic AI systems. These are systems that autonomously plan and execute multi-step tasks, use external tools, or take actions across a sequence of steps. The FDA notes they are already being deployed for care coordination and clinical documentation. Some of that might not be regulated by the FDA, but if an agentic system controls another medical device, it absolutely falls under their oversight.

Maya: And the FDA points out that agentic architectures introduce elevated risk because of autonomous multi-step action and reduced opportunity for human review. They are looking for feedback on additional competencies for these systems, like graceful handling of tool failures and resistance to prompt injection.

Sam: It is clear the FDA recognizes that old frameworks simply were not designed for these novel characteristics. This is a massive request for feedback from everyone in the healthcare ecosystem.

Maya: To recap, the FDA is proposing a total product life cycle approach for generative AI devices. The key highlights are a two-axis framework to evaluate risk based on independence and consequences, and a competency-based premarket evaluation that uses extensive benchmarking and clinical confirmation, rather than just testing lines of code.

Sam: They are also looking at heavier postmarket monitoring to offset some premarket uncertainty, and exploring voluntary master files for foundation models to help regulators understand the underlying tech.

Maya: You can find a link to the source document in the show notes. It is a must-read if you are building or buying in this space, as the agency is actively seeking comments.

Sam: And as always, a quick reminder that this podcast is not medical advice. Thanks for listening!