Why Generative AI Trials Fail the Reproducibility Test

Maya: In a brand new systematic review of clinical trials testing generative AI, researchers found something genuinely alarming. Exactly 0% of the reviewed studies reported how they handled poor-quality or missing input data.

Sam: It is a massive blind spot for anyone trying to figure out if these tools actually work safely. Today we are digging into a paper titled Reporting Gaps and Challenges in Randomized Trials of Generative AI for Medical Use: Insights From a Systematic Review, published in the Journal of medical Internet research.

Maya: And before we go any further, we need to let you know that our voices are AI-generated, and this episode is a summary of that public document.

Sam: So let us set the stage here. We all know generative AI has exploded in medicine. The authors of this paper wanted to know if the randomized controlled trials testing these tools are actually being reported transparently.

Maya: Right. Randomized controlled trials, or RCTs, are the gold standard for clinical evidence. To make sure an RCT is reliable and transparent, researchers are supposed to follow reporting guidelines. For AI, the benchmark guideline is called the CONSORT-AI extension, which came out in 2020.

Sam: But 2020 feels like ancient history in AI years. That guideline was built for conventional AI, right? Like a model that looks at an X-ray and flags a tumor. It was not built for a chatbot.

Maya: Exactly. Generative AI fundamentally differs from conventional AI. So the researchers wanted to audit item-level adherence to CONSORT-AI among RCTs evaluating generative AI for medical purposes. They wanted to see if the gaps in reporting are just because researchers are ignoring the rules, or if there are structural mismatches where the checklist just fails to capture how generative AI actually works.

Sam: And the scope of this review is pretty massive. They started with 16,485 retrieved records and ended up including 74 reports from 73 unique RCTs. That covers a total of 24,150 human participants.

Maya: The median sample size was 100, with an interquartile range from 50.0 to 247.0. And looking at the tech, it is overwhelmingly text-based. Large language models accounted for 98.6% of the interventions. And specifically, 60.8% used GPT-family models.

Sam: It really shows how dominant those OpenAI models are right now. The publication timeline they tracked is also a perfect snapshot of the hype cycle. They saw 2 reports in 2023, then 8 in 2024. Then a huge spike with 39 reports in 2025, which was 52.7% of the total, and that momentum just kept going into early 2026 with another 25 reports.

Maya: It is moving incredibly fast. And geographically, half of these reports, exactly 50.0%, were conducted in Asia. North America followed with 31.1%, and Europe had 18.9%.

Sam: Okay, so what are all these trials actually testing? Who is using the generative AI in these studies?

Maya: The most common primary intended use was providing medical information and education to laypeople, which made up 44.6% of the reports. After that, you had mental health support at 18.9%, clinical decision support also at 18.9%, and administrative support or clinical documentation at 16.2%.

Sam: So a huge chunk of these tools are being put directly into the hands of patients and the public. In fact, laypeople and patients were the primary end users in 51.4% of the reports, while clinicians were targeted in 31.1%.

Maya: That split between laypeople and clinicians becomes incredibly important when we look at the reporting failures. The researchers rated each item on the CONSORT-AI checklist as reported, partially reported, not reported, or not applicable. And the results are a wild mix of high compliance on basic stuff and total failure on the technical details.

Sam: Let us start with the good news. What did trial authors actually get right?

Maya: They were great at general metadata. For instance, 98.6% stated the intended use of the AI intervention in the title or abstract. Specifying the output of the intervention hit 94.59%. Stating the inclusion and exclusion criteria at the participant level was at 89.19%.

Sam: Okay, so they know how to report a standard clinical trial. But once we get into the AI-specific mechanics, the numbers just fall off a cliff. You mentioned earlier that 0% reported on handling poor-quality input data. How is that even possible?

Maya: That is Item 5(iii) on the checklist. It asks how poor-quality or unavailable input data were assessed and handled. In a conventional AI trial, say one analyzing retinal scans, poor quality means a blurry image or a missing file. You can set a resolution threshold in advance and filter it out automatically.

Sam: Right, the data is locked and loaded before the trial even starts.

Maya: Exactly. But in 73 of the 74 reports in this review, the intervention was a language model receiving text generated during the trial itself, usually by the user. When a user types a query, poor quality means ambiguity, a lack of context, or omitting key medical terms. The model cannot screen that out before it processes it.

Sam: So every single patient is basically creating a completely different, unstandardized input on the fly. And the authors point out this variability is not just a minor detail. They cite evidence that language models vary in accuracy by 8% to 50% based on how a query is optimized.

Maya: And accuracy changes by 5% to 18% just based on the order in which the same information is presented. The way a patient phrases a question fundamentally changes the medical advice they get back. Yet trials are not reporting how they handle this variability.

Sam: This ties directly into input data eligibility, which is Item 4a(ii). That item was only reported in 4.1% of the trials. And input acquisition and selection, Item 5(ii), was only reported in 12.2%.

Maya: And it is fascinating to see who actually got credit for reporting that 12.2%. The review found that full reporting was credited exclusively to trials that defined explicit data gating procedures.

Sam: What does that look like in practice?

Maya: It means they either forced users to use structured input templates, or they had a human-in-the-loop setup where a clinician filtered and verified the patient data before submitting it to the AI. But if a trial just gave a user an open chat box with unconstrained free text, they were categorized as nonreporting.

Sam: Because the researchers basically abandoned oversight. They delegated the data structuring entirely to the user. But the paper notes that even in a free-text chat, researchers still have control at the backend through prompt engineering. Things like system prompts.

Maya: System prompts are huge. They set the boundaries for the model. They dictate the reading level, enforce consistency, and tell the model what not to say. But only 20 reports, which is 27.0%, explicitly reported the system prompt they used.

Sam: If you are a hospital executive trying to buy a solution, or a clinician trying to trust a paper, and they do not tell you the system prompt, you literally cannot reproduce the intervention. It is like reading a drug trial that refuses to tell you the dosage.

Maya: The authors found a statistical link there, too. Reporting of a system prompt was associated with higher adherence to Item 5(ii) after adjustment. So researchers who understand the importance of backend prompt engineering are generally better at reporting their input selection protocols.

Sam: Let us talk about another massive missing piece: inference settings. The paper gives this wild example of how temperature settings change everything.

Maya: Yes, they mentioned a simulation study using GPT-4o on emergency medicine cases. When they set the temperature to 0.0%, the leading diagnostic accuracy was 100%. But when they turned the temperature up to 1.0, accuracy dropped to 89.4%.

Sam: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Maya: And now, back to the document.

Sam: And the scatter of answers exploded. The mean number of distinct diagnoses generated went from 4.5 at a temperature of 0.0%, all the way up to 26.25 at 1.0.

Maya: Exactly. 2 trials evaluating the same model but using different inference settings and system prompts are not evaluating the same intervention at all. Yet the current CONSORT-AI checklist does not even ask for system prompts or inference settings. They are completely absent from the standards.

Sam: That is what they mean by structural mismatches. The guideline is asking the wrong questions for this technology. But even on the questions it does ask, like what version of the model you used, researchers are dropping the ball. Algorithm versioning was only reported in 23.0% of reports.

Maya: This is a critical failure for generative AI. In conventional AI, the model is locked. But off-the-shelf generative systems are continuously updated with new training data and weights, often without public changelogs.

Sam: Right, just saying you used GPT-4 is meaningless. There is gpt-4-0314, gpt-4-0613, gpt-4-turbo, and gpt-4o. They all have completely different performance profiles. If a paper just says GPT-4, they might get a checkmark for stating the version under the current rules, but they have not actually provided the information you need to reproduce the trial.

Maya: And it is not just the version. It is how you access it. The paper notes that using a web interface can yield different results than using an API, because the web interface might have hidden system prompts or web browsing tools built in. An API gives you a controlled environment.

Sam: Which brings up a huge boardroom issue: privacy. This is where the unavailable data requirement of Item 5(iii) takes on a whole new meaning for generative AI.

Maya: Exactly. For a language model, unavailable data can mean inputs that legally should not be submitted, like protected health information. The web-based interface of ChatGPT is not HIPAA-compliant. Patient information must never be entered there.

Sam: To be compliant, you need a cloud environment that offers a Business Associate Agreement, like the Azure OpenAI Service. The review looked at 21 reports from the United States, plus trials under GDPR in Europe. But none of them completely reported what interface arrangement governed the data or what measures prevented participants from submitting inadmissible content.

Maya: Only 8 reports even partially addressed it. Trial reports should state what constitutes an inadmissible input and the contractual arrangement under which data were processed. The fact that this is essentially missing from the literature should be a major red flag for any health system evaluating these tools.

Sam: Let us look at how the AI is actually integrated into the trial setting, which is Item 4b. That was only reported in 28.4% of reports. And there was a really interesting split based on how the AI was deployed.

Maya: There was. Half the trials, 50.0%, relied on off-the-shelf commercial applications. And for those trials, the full reporting rate for setting integration was significantly lower, just 13.5%. Compare that to customized interventions, which hit 37.5%, or bespoke models, which were at 100%.

Sam: It seems like researchers using ChatGPT just assume everyone knows how it works, so they do not bother writing down the basic operational prerequisites, like target access devices or internet requirements. Familiarity breeds sloppy reporting.

Maya: That same kind of oversight showed up in how outputs contributed to decision-making, which is Item 5(vi). Overall, it was reported in 51.5% of trials. But the subgroup analysis here is the largest difference observed in the entire review.

Sam: This blew my mind. When the trial was aimed at clinicians, 100% of the reports, 23 out of 23, fully reported how the output contributed to decision-making. But when the intervention was aimed at laypeople, the full reporting rate was only 15.8%.

Maya: It is totally backward. Clinicians already have baseline expertise. Laypeople bring no baseline medical expertise, meaning the interpretive burden generative AI places on them is much heavier. Trials targeting the public desperately need to state what instructions participants were given for interpreting these outputs.

Sam: And we see the exact same trend with human-AI interaction while handling input data, Item 5(iv). Overall reporting was just 33.8%. But it was higher for clinician-facing tools like administrative documentation, at 58.3%, and plummeted for tools aimed at laypeople, like mental health support, at just 7.1%.

Maya: Trials of laypeople-facing conversational tools are simply not reporting the user instructions and interaction safeguards required to operate these interventions safely.

Sam: What about analyzing when things go wrong? Item 19 asks for an analysis of performance errors. Where did that land?

Maya: Only 36.5% of reports adequately described the results of any performance error analysis and how errors were identified. With generative hallucination being such a known risk, the fact that so many clinical trials are failing to properly report their error analysis is deeply concerning.

Sam: And if you want to look at the data yourself to check for errors, good luck. Item 25 is about open science and data sharing. Only 41.9% of reports fully stated whether and how the intervention or its code could be accessed.

Maya: If subsequent researchers cannot access the exact intervention evaluated, including the underlying prompts and parameters, they cannot validate it. And for off-the-shelf commercial models, the artifact is controlled by a third party. It might not even exist in the form that was tested anymore.

Sam: So if you look at all these gaps, only 9 of the 74 reports, which is 12.2%, explicitly cited using the CONSORT-AI guideline at all. Does citing the guideline actually fix the problem?

Maya: Only partially. The studies that explicitly declared adherence to CONSORT-AI did show higher compliance for data sharing and detailing human-AI interaction. Those are deficiencies of nonadherence. If we just enforce the existing standards better, researchers will report those items.

Sam: But enforcing the checklist will not fix the structural mismatches. You can force someone to state the algorithm version, but as we discussed, for an LLM, that still does not make it reproducible without the system prompt, inference settings, and access modalities.

Maya: Precisely. The authors argue that we need a tailored extension to current reporting standards. They point to other frameworks like TRIPOD-LLM and MI-CLAIM-GEN, which do ask for things like the base model, fine-tuning procedure, system prompts, and structured privacy reporting. But those frameworks were not built specifically for randomized trials.

Sam: So the solution is a GenAI-specific trial guideline, built through a structured consensus process with trialists, regulators, and developers.

Maya: Yes, because until we have that, clinical decisions and regulatory approvals are being based on evidence that is neither reproducible nor transparent. That said, we should note the limitations of this review. The authors restricted their search to peer-reviewed, English-language studies explicitly identified as RCTs.

Sam: Right, so they might have missed evaluations published in other languages or preprints. They also used nonvalidated signaling questions to rate the checklist items. And they acknowledge that these trials were mostly evaluating text-based models. As multimodal AI enters clinical trials, it is going to create a whole new set of reporting headaches.

Maya: For clinicians and health system leaders, the takeaway is clear: do not take randomized controlled trial evidence for generative AI at face value. If the vendor cannot tell you their system prompt, their exact inference settings, and how they handled user input variability, you do not have reproducible evidence. You have an anecdote.

Sam: It is like trying to measure water with a ruler. The tools we use to evaluate evidence have to match the mechanics of the technology we are testing.

Maya: Well said. To recap, this systematic review of 74 trial reports found severe reporting gaps for generative AI in medicine. Critical drivers of reproducibility like system prompts, inference parameters, and input quality management are being routinely omitted, exposing a mismatch between current trial standards and generative AI mechanics.

Sam: A quick reminder that the source document is linked in the show notes, and that this is not medical advice. Thanks for listening.