17 November 2026 · 15 min
Why One-Time Bias Checks Fail: Building Continuous AI Audit Pipelines in Healthcare
This episode unpacks a scoping review on operationalizing fair AI in healthcare, focusing on why static pre-deployment bias checks are no longer enough. Maya and Sam explore the transition to continuous monitoring, detailing specific bias mitigation strategies, the complexities of regulatory compliance, and the exact tools organizations are using to build ongoing algorithmic audit pipelines.
Key points
- AI Act certification compliance is estimated to cost between 10 to 14% of development costs, highlighting the massive operational burden of AI governance.
- A review of 66 studies reveals that fairness in healthcare AI requires a shift from static, one-time checks to continuous post-deployment monitoring due to risks like fairness drifts.
- Technical bias mitigation spans multiple lifecycle stages, utilizing methods like the Synthetic Minority Oversampling Technique, Generative Adversarial Networks, and adversarial debiasing.
- Only 54% of surveyed WHO Member States reported having regulatory agencies responsible for assessing and approving health AI, exposing a global gap in post-deployment surveillance.
- Effective AI governance demands a multidisciplinary approach, utilizing specific auditing toolkits like IBM's AI Fairness 360 and explainable AI models to maintain safety and compliance.
Source: Operationalizing fair AI in healthcare: a scoping review of bias mitigation, monitoring, auditing, and governance strategies - Frontiers in digital health, 2026 (CC BY)
This spot is available. Reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsor the show
This episode is an AI-generated conversation summarising a public document; the hosts' voices are synthetic. It is for information only and is not medical advice. Always refer to the original source.
Authors: Ana Teresa Rebocho, Sara Silva Marques, João Pedro Santos, Cátia Sousa Pinto · DOI: 10.3389/fdgth.2026.1888744
Transcript
Maya: Did you know that 10 to 14% of development costs for healthcare AI are now estimated just for AI Act certification compliance?
Sam: That is a massive operational overhead. And it explains exactly why we are looking at this specific document today, which is titled Operationalizing fair AI in healthcare: a scoping review of bias mitigation, monitoring, auditing, and governance strategies, published in Frontiers in digital health.
Maya: Before we dive in, we need to let our listeners know that our voices are AI-generated, and this episode is a summary of a public document. But as always, we are unpacking exactly what this means for clinicians on the ward and executives in the boardroom.
Sam: And the stakes here could not be higher. We are talking about how health systems and tech companies actually prove that their algorithms are safe and equitable across different patient populations, not just initially, but continuously.
Maya: Exactly. This document is a scoping review that maps out how the industry is trying to operationalize fairness throughout the AI lifecycle. It looks at bias detection, mitigation strategies, and the governance mechanisms needed to keep these systems in check as regulations tighten.
Sam: I think a good place to start is understanding just how much literature they actually reviewed to draw their conclusions. The authors started with a pretty wide net, right?
Maya: They did. The initial search identified 1030 studies in total. They used two main search queries. The first query brought in 600 studies, and the second query brought in 430.
Sam: And it is interesting to note that for that second query, only 30 studies actually came from the PubMed search. So they had this massive pile of 1030 initial studies. How did they narrow it down?
Maya: First, they removed 132 duplicates. Then they removed 178 records that were either not in English or did not have the full text available. That left them with 720 studies to thoroughly examine.
Sam: From those 720, they retrieved 176 for a full eligibility assessment. But they did not just look at academic papers, right? They also dug into grey and regulatory literature, which is crucial when we are talking about compliance.
Maya: Right. They identified 56 documents from official sources. To break that down, they found 3 from EUR-Lex, 5 from the European Commission, 3 from the WHO, 12 from the EMA, and 33 from the FDA.
Sam: And after assessing all of those official documents, they excluded 45 of them because they did not address the specific AI lifecycle stages or governance mechanisms they were looking for. So that left 11 regulatory documents.
Maya: Exactly. So if you take the 55 studies from the academic databases and add the 11 from the grey literature, you get a final count of 66 documents included in this review.
Sam: Okay, so 66 documents form the core of this paper's findings. How did those 66 studies break down across the different themes they were investigating?
Maya: The authors categorized them, and keep in mind these categories are not mutually exclusive. But 34.8%, which is 23 studies, addressed fairness measures. Then 30.3%, or 20 studies, emphasized continuous monitoring.
Sam: And what about the auditing and governance side of things?
Maya: Both audits and AI governance were each addressed by 25.8% of the studies, meaning 17 studies fell into each of those buckets. Finally, 12.1%, which is 8 studies, specifically addressed data representativeness.
Sam: That breakdown really highlights a core theme of this document. A large portion of the literature focuses on the technical fairness measures, but almost just as much focuses on continuous monitoring. And that is the big shift we need to talk about.
Maya: Yes, it is the realization that fairness is not a box you check once before you deploy an algorithm. The authors point out that as AI models move into real-world healthcare, their dynamic learning nature and sensitivity to contextual changes can lead to what they call fairness drifts.
Sam: I love that term. The document defines fairness drifts as inequitable changes over time in AI system performances across population subgroups. Basically, your model might be fair initially, but drift into bias over time.
Maya: Precisely. And to prevent that, or at least mitigate it, the document outlines strategies across the entire AI lifecycle. Let's start upstream, at the data preparation stage. How do we fix unbalanced or skewed health datasets?
Sam: The paper talks heavily about pre-processing techniques, specifically re-weighting and resampling. Re-weighting lets developers modify the weights assigned to different groups in the training samples. Resampling actually changes the size of the dataset.
Maya: And within resampling, you can either decrease the representation of certain groups, which is under-sampling, or increase it, which is oversampling. The authors note that oversampling is the method most described in the literature.
Sam: They specifically mention the Synthetic Minority Oversampling Technique. It generates synthetic samples by interpolating between minority-class instances to balance the data.
Maya: But the document warns there is a trade-off there. That technique can produce unrealistic data points that do not reflect real-world representation. So developers also turn to synthetic data generation methods that mimic statistical properties without compromising privacy.
Sam: Right, they highlight two main synthetic generation methods: Generative Adversarial Networks and Variational Autoencoders. But they each have their own pros and cons depending on the data type.
Maya: Exactly. Generative Adversarial Networks are usually used for realistic synthetic medical imaging like x-rays or MRIs. But the text notes they require high computational demands and high-quality data to produce good results, and training them is often considered challenging.
Sam: And on the flip side, Variational Autoencoders encode real-world datasets like Electronic Health Records into a lower-dimensional latent representation. They achieve lower realism than the adversarial networks, but the document says they are easier to train and tend to provide greater data samples diversity.
Maya: So that covers the data preparation phase. But the review also dives into bias mitigation during model training and validation. One approach they highlight is Federated learning.
Sam: Federated learning is fascinating. It gives models access to diverse datasets across multiple institutions without actually moving the raw data. The institutions share trained model parameters which are then aggregated to create a global model.
Maya: It is great for privacy, but the authors point out a limitation. Because it is decentralized, you cannot easily harmonize the data centrally. The quality and complexity of real-world data can significantly impact local pre-processing stages, which in turn impacts model performance.
Sam: Another training strategy they mention is adversarial debiasing. This involves setting up two neural networks against each other. One is the predictor that learns to predict the outcome, and the other is an adversary that tries to infer sensitive attributes from the predictor's outputs.
Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.
Sam: And now, back to the document.
Maya: The goal there is to balance predictor accuracy while reducing its reliance on sensitive attributes like race or gender. But again, the document is clear that its effectiveness is highly context-dependent and can introduce trade-offs between fairness and accuracy.
Sam: They also bring up counterfactual fairness, which relies on causal modelling. It asks whether a prediction remains consistent if you alter sensitive attributes, essentially testing if those attributes causally influenced the decision-making process.
Maya: But applying counterfactual fairness requires identifying causality in the decision-making process, which the authors say requires further validation for real-world settings. This brings us to the actual metrics used during validation and testing. How do we actually measure fairness?
Sam: The document lists several frequently reported metrics, but it makes a really important point: these metrics represent distinct, and sometimes competing, notions of fairness.
Maya: They list predictive parity, equality of opportunity, delta accuracy, equalized odds, and demographic parity. And the definitions matter. For instance, equality of opportunity assesses whether false negative rates are comparable across groups.
Sam: Right, whereas predictive parity evaluates whether positive predictions have similar predictive values between protected and privileged groups. And demographic parity examines whether different groups have similar probabilities of receiving a positive model prediction.
Maya: Which means a model could look perfectly fair under one metric, but highly biased under another. The authors stress that the selection of fairness metrics has to be tailored to the specific healthcare application and guided by its clinical context.
Sam: And that is exactly why the review pushes so hard on the idea that fairness is not just a technical challenge. It is a sociotechnical challenge. You cannot just pick a metric and walk away. You need a robust AI governance framework.
Maya: Which brings us to the core argument of this paper. Pre-deployment checks are insufficient. The continuous learning nature of these systems means they require post-deployment surveillance strategies.
Sam: The FDA and the EMA are already emphasizing this. The FDA's recent recommendations propose post-deployment monitoring plans, pushing for ongoing assessments in the post-market setting to support corrective actions.
Maya: The authors suggest that, much like in the pharmaceutical sector, it would be advantageous to establish fairness validation thresholds for model outputs, guided by national competent authorities. But defining those thresholds is difficult because fairness requirements vary depending on the clinical purpose and risk levels.
Sam: So how do organizations actually execute this continuous surveillance? What tools do they use?
Maya: The document points to internal and independent audits. And to make audits possible, they need tools for Explainable AI, or XAI. These tools make opaque models more understandable so auditors can identify bias.
Sam: They specifically name a few of these XAI tools: Shapley Additive Explanations, Local Interpretable Model-agnostic Explanations, and Gradient-weighted Class Activation Mapping.
Maya: Though they do note that progress in incorporating XAI methods into clinical workflows has been limited because it is operationally complex. Beyond XAI, they also mention specific bias assessment toolkits that need to be integrated into real-world workflows.
Sam: Right, they call out IBM's AI Fairness 360 and Google's What-If Tool. Integrating these kinds of dashboards and open-source toolkits allows practitioners to conduct routine oversight without disrupting operations.
Maya: But the document is clear: implementing auditing toolkits involves financial and administrative costs. And this ties back to that statistic we mentioned at the top of the episode. 10 to 14% of development costs are already estimated for AI Act certification compliance.
Sam: That is a huge burden for developers, especially because under the MDR, AI-based medical devices have to get CE certification from notified bodies. And the continuous learning nature of AI creates potential bottlenecks for those regulatory approvals.
Maya: The authors actually suggest that regulatory approval should be time-limited and periodically revisited. They mention that regulatory incentives, like funding opportunities and compliance certifications, could encourage institutions to adopt these ethical practices.
Sam: They give an example of one of these certification initiatives: the IEEE Ethics Certification Program for Autonomous and Intelligent Systems. It is about formalizing trust through a recognized certification.
Maya: But to make any of this work, you need clear governance at both the institutional and national levels. And the document reveals a pretty concerning gap in national governance readiness.
Sam: Yeah, they cite a survey across the WHO European Region. It showed that only 54% of Member States reported having one or more regulatory agencies responsible for assessing and approving AI systems in health.
Maya: And even fewer had agencies tasked with post-deployment monitoring. The findings showed that many Member States are relying on cross-sectoral approaches rather than health-specific strategies.
Sam: The EU AI Act tries to address this by requiring each Member State to designate market surveillance authorities. And if they appoint multiple authorities, they have to designate a Single Point of Contact to liaise with the public and EU counterparts.
Maya: But the document notes it remains unclear how this will be implemented in practice, especially in countries that opt for a single cross-sector authority. The operational model for how inspections will be prioritized is still undefined.
Sam: So if I am a health system leader or a compliance officer listening to this, the message is pretty stark. You cannot wait for the national regulatory bodies to iron out all their operational models. You have to build institutional governance now.
Maya: Exactly. The review recommends a multidisciplinary approach at the institutional level. That means AI researchers, healthcare professionals, policymakers, and regulators all sharing responsibility for setting fairness performance benchmarks and overseeing continuous monitoring.
Sam: And integrating AI governance directly with data governance. You need a team that takes responsibility for both the collection of representative data and the ongoing monitoring for fairness drifts.
Maya: The authors conclude that successful implementation will require interdisciplinary collaboration. It is not just about the technical tools, whether that is oversampling or adversarial debiasing. It is about human oversight, comprehensive documentation, and audit trails.
Sam: Meeting the transparency requirements of the AI Act and GDPR means keeping a clear track of algorithmic decisions so that when a failure or inequitable outcome occurs, you can actually trace it back and identify the responsible parties.
Maya: That is the ultimate takeaway here. Fairness in healthcare AI has moved from being a theoretical, design-stage goal to a continuous, operational necessity. One-time checks are simply not enough to ensure trustworthiness or legal compliance.
Sam: That covers the core of this scoping review. To recap, we unpacked why static bias checks fall short, explored the specific metrics and models organizations are using, and highlighted the urgent need for continuous algorithmic auditing pipelines as regulatory pressures mount.
Maya: As always, a reminder that the source is linked in the show notes, and that this is not medical advice. This podcast is for informational purposes only. Thanks for listening.