When Algorithms and Doctors Agree for the Wrong Reasons in the ICU

Maya: The single most interesting finding in today's document is a warning sign for hospitals. When an AI system and a physician agree on a critical care diagnosis, that agreement does not actually mean they are right. If both are reasoning from the same incomplete data and shared assumptions, concordance might just reinforce a dangerous error.

Sam: Which completely flips how we usually think about getting a second opinion from an algorithm. We tend to assume that identical answers mean certainty, but this document shows why that is a trap. Before we dive into the details, a quick reminder that our voices are AI-generated, and this is a summary of a public document.

Maya: Exactly. We are unpacking a document titled Clinical Reasoning, Pathophysiology, and Artificial Intelligence in Critical Care: A Conceptual Framework for Integrating Decision-Making. It was published in the Journal of clinical medicine in 2026.

Sam: So this document is fundamentally about how doctors think, how disease mechanisms actually work, and how AI fits into that incredibly high-stakes mix, specifically in the intensive care unit. It is essential reading for critical care specialists and hospital AI committees trying to prevent algorithm failure modes in hemodynamically unstable patients.

Maya: Right. The authors start by defining medicine as an interpretive practice conducted under uncertainty. Physicians almost never have complete information, deciding based on ambiguous signs, imperfect tests, and changing probabilities. And diagnostic error is now recognized as a major source of preventable harm, contributing to a substantial burden of death and disability in the United States.

Sam: And the ICU magnifies all of that pressure. We are talking about patients with physiologic instability, multiorgan dysfunction, and nonlinear clinical trajectories where multiple interventions occur simultaneously. A small interpretive failure can alter decisions about mechanical ventilation, vasopressors, antimicrobials, anticoagulation, sedation, fluids, dialysis, or even the limitation of life-sustaining treatment.

Maya: In critical care, reasoning does not stop once a diagnosis is made. It persists through every new arterial blood gas result, culture, hemodynamic shift, and response to therapy. But historically, the fields of clinical reasoning, pathophysiology, and artificial intelligence have been studied in complete isolation.

Sam: Which is exactly why the authors proposed this conceptual framework. To map the field, they conducted a narrative review. They searched PubMed and MEDLINE from inception through July 31, 2026. The first search string, focusing on clinical reasoning and pathophysiology in critical care, found 17 records.

Maya: And their second search string, looking at AI applied to clinical reasoning in critical care, identified 46 records. They also did a supplementary search using an AI-assisted academic search engine called Consensus, which turned up 251 additional records. They screened these to map the field rather than providing an exhaustive census.

Sam: It is fascinating to look at how medical reasoning evolved to this point. The authors trace the history of clinical reasoning in the twentieth century. Until the mid-twentieth century, medical education was built on the anatomoclinical model, which was all about recognizing syndromes, localizing lesions, and correlating signs with anatomical abnormalities.

Maya: Which was insufficient for the probabilistic uncertainty of modern medicine. In the 1940s and 1950s, three movements challenged it: clinical epidemiology, decision theory, and the cognitive revolution.

Sam: Then in the 1970s, Elstein, Shulman, and Sprafka proposed the hypothetico-deductive model, where doctors generate early hypotheses and actively look for data to confirm or reject them. Around the same time, Tversky and Kahneman revealed how human judgment under uncertainty relies on mental shortcuts called heuristics.

Maya: And Pauker and Kassirer proposed the threshold model, weighing pretest probability against intervention risks. Later, evidence-based medicine codified integrating evidence with expertise, which reduced practice variability but introduced an applicability gap—the challenge of translating evidence from selected trial populations to complex individual patients.

Sam: Moving into the 1980s and 1990s, research shifted to descriptive accounts of expertise. Norman and Eva established that expert reasoning relies on highly organized knowledge networks. They called it the illness script, linking predisposing factors to expected findings and typical evolution.

Maya: Since the 2000s, dual-process theory has become the foundational framework, distinguishing fast, intuitive Type 1 reasoning from slow, analytical Type 2 deliberation. In the ICU, you need Type 1 to rapidly spot life-threatening patterns, but you absolutely must have Type 2 analytical oversight to reassess initial hypotheses, integrate discordant data, calculate probabilities, consider alternatives, and mitigate the risk of premature closure.

Sam: Which brings us to how reasoning actually fails at the bedside today. The authors emphasize that a diagnostic error—defined as a failure to establish an accurate and timely explanation of the patient's health problem—is rarely just one isolated mistake. It emerges from a complex interaction among cognitive factors, system factors, and case characteristics.

Maya: These cognitive factors include biases, poor problem representation, and overconfidence. They are exacerbated by system factors like excessive workload, frequent interruptions, inadequate handoffs, and a lack of diagnostic feedback.

Sam: And let's not forget the case-specific factors like atypical clinical presentations, concurrent comorbidities, and accelerated disease evolution. But the cognitive biases are particularly actionable for anyone building clinical decision support.

Maya: Let us break those down. First is premature closure, where a physician accepts a diagnosis before relevant alternatives are considered. Then there is anchoring bias, where the first information received exerts excessive influence. Confirmation bias leads clinicians to preferentially value data supporting the initial hypothesis.

Sam: They also highlight availability bias, which makes recently encountered or emotionally salient diagnoses seem more likely than they are. And search satisficing, which occurs when identifying one abnormality reduces the motivation to identify others. These biases take very concrete forms in the intensive care unit.

Maya: The authors give specific ICU examples. A patient with sepsis and persistent hypoxemia might be misattributed to worsening ARDS, when fluid overload or a pulmonary embolism are actually responsible. Or a hypotensive patient on norepinephrine being attributed to vasoplegic shock, while occult hypovolemia or tamponade are overlooked.

Sam: Another example: an agitated patient might receive additional sedation for presumed hyperactive delirium when pain, withdrawal, or ventilator asynchrony are the primary causes. The error is failing to reconsider the diagnosis when clinical evolution contradicts it.

Maya: To prevent these errors, awareness of biases alone is insufficient. The authors propose a combined approach based on theoretical plausibility. The first strategy is improving problem representation through selective case summarization that incorporates the time course, severity, context, risk factors, pertinent negative findings, and incongruities.

Sam: Instead of just saying a patient has simple pneumonia, you say: an immunosuppressed older adult with acute dyspnea, low-grade fever, hypoxemia disproportionate to the radiographic findings, and elevated D-dimer after prolonged hospitalization. That detailed representation activates completely different diagnostic hypotheses.

Maya: A second strategy is maintaining an active differential diagnosis, not as a ritualistic list, but a hierarchy of hypotheses with probabilities and mechanisms. Ask: what diagnosis must I not miss? What best explains all findings? Which treatment may be masking the presentation?

Sam: A third strategy is incorporating cognitive pauses at critical moments, like unexpected deterioration or handoffs. Ask yourself: Does the current diagnosis still explain the clinical course? Is there any new evidence contradicting it? Is there a high-risk alternative cause that must be ruled out now?

Maya: A fourth strategy is to value pathophysiology as a coherence test, distinguishing spurious associations from mechanistically plausible explanations. And a fifth strategy is to improve the team's environment. Many errors are identified by team members who spot incongruities but do not find space to express them.

Sam: Therefore, the question 'What are we missing?' should be routine in the ICU. This moves us perfectly into why pathophysiology and disease mechanisms are so critical here. Clinical medicine operates in two complementary languages: diagnostic categories, which name diseases, and mechanisms, which explain processes.

Maya: Diagnoses without mechanisms can lead to generic management. Mechanisms without diagnostic categories lose prognostic, epidemiologic, and therapeutic integration. The medical education literature, like the research by Woods, shows that biomedical knowledge is not a dispensable preclinical layer. It is an active component of clinical reasoning that contributes to constructing explanations and reasoning through difficult cases.

Sam: In typical cases, pathophysiology might remain implicit. But in atypical, severe, or contradictory cases, it becomes essential. It improves efficiency by enabling clinicians to select interventions aligned with the predominant mechanism, anticipate adverse effects, recognize paradoxical responses, and individualize treatments.

Maya: For example, in hypoxemia, identifying whether the main problem is an intrapulmonary shunt, hypoventilation, low inspired oxygen fraction, impaired diffusion, or low cardiac output completely changes the strategy. In shock, recognizing whether vasodilation, hypovolemia, pump failure, or obstruction predominates changes the priority among fluids, vasopressors, inotropes, drainage, thrombolysis, pericardiocentesis, or ventilator adjustment.

Sam: The authors point out that mechanistic reasoning is simply unavoidable in the ICU. Take mechanical ventilation. It is not just a treatment for respiratory failure; it alters transpulmonary pressure, venous return, right ventricular afterload, regional distribution of ventilation, and the risk of lung injury. Decisions about PEEP require understanding recruitability, hemodynamics, compliance, and hyperdistention.

Maya: They use acute respiratory distress syndrome, or ARDS, as a prime example of how diagnostic categories must be interpreted mechanistically. The Berlin Definition standardized criteria for timing, imaging, edema origin, and degree of hypoxemia. But it does not replace the analysis of recruitability, compliance, dead space, driving pressure, cardiopulmonary interaction, and the precipitating etiology.

Sam: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Maya: And now, back to the document.

Sam: Two patients might meet ARDS criteria and require distinct strategies for ventilation, fluids, prone positioning, sedation, neuromuscular blockade, infectious investigation, or hemodynamic assessment. The same principle applies to sepsis. Guidelines provide recommendations on early recognition, antimicrobials, source control, resuscitation, and vasopressors, but safe application requires individualized interpretation.

Maya: This ties directly into decision theory and Bayesian reasoning. Bayesian reasoning makes explicit what good clinicians do intuitively: interpreting each datum considering a prior probability and updating it in response to new evidence. The same troponin value carries different implications in a young person with atypical chest pain versus an older adult with septic shock.

Sam: Data do not speak for themselves; they modify a situated hypothesis. In the ICU, almost all variables are continuous, unstable, and context-dependent. The probability of a pulmonary embolism, myocardial infarction, sepsis, hypovolemia, tamponade, ARDS, or ventricular dysfunction should be treated as a continuously revisable estimate, not as a fixed category.

Maya: Which is why the authors warn against the excessively dichotomous reading of evidence, like labeling clinical trials as positive or negative based on p-values. Bedside decisions require estimating graded probabilities of benefit, harm, or futility. This Bayesian thinking drives adaptive platform trials, which can update treatment probabilities as data accrue and allocate patients dynamically to more promising strategies.

Sam: So how does artificial intelligence fit into this entire ecosystem? The authors cite Eric Topol, who described the convergence of human and AI as a central transformation in high-performance medicine, emphasizing that its value depends on improving human care, not merely automating tasks.

Maya: In critical care, the appeal is evident because the ICU produces continuous, multidimensional data. But AI systems face massive challenges operating in environments characterized by high uncertainty, high consequences, and multiple simultaneous objectives. The document groups AI applications into three broad approaches, each with distinct risks.

Sam: First, rule-based clinical decision support systems encode explicit knowledge but are limited by rigidity and maintenance burden. Second, discriminative machine learning models, like logistic regression and deep neural networks, estimate risks from data and excel at prediction, but they do not generate clinical language or explanations.

Maya: Third, generative large language models, or LLMs, produce free-text hypotheses, summaries, and plans. This makes them uniquely interactive but also uniquely susceptible to hallucination—fluent and confident outputs that are factually or clinically incorrect, which is a serious safety concern in high-stakes settings.

Sam: The evidence on these models requires careful interpretation. A 2024 randomized clinical trial found that access to an LLM did not significantly improve physicians' diagnostic reasoning compared with conventional resources. A 2025 trial found GPT-4 assistance improved performance on patient-care tasks, and another study suggested diagnostic gains in complex critical-illness cases.

Maya: However, those trials involved case- and simulation-based tasks, not real patients. Benchmark studies show LLMs achieve high performance in some domains but with inconsistency. And fundamentally, LLMs lack essential metacognition—they do not sufficiently recognize the limits of their own knowledge and provide confident answers when uncertainty should be explicit.

Sam: Therefore, the authors argue that medical AI should be viewed as cognitive support, not as the final epistemic authority. Large models should genuinely support the physician's cognitive activities rather than merely optimize performance on decontextualized tests. Instead of asking 'what is the diagnosis?', it is safer to treat AI as a hypothesis generator and cognitive adversary.

Maya: You should ask the AI: what serious diagnoses could explain these findings? Which data contradict my leading hypothesis? Which pathophysiologic mechanisms could produce refractory hypoxemia in this context? Which adverse effects of current interventions may worsen the patient's condition? You request structure, not a final answer, making the AI a tool for problem representation, not an oracle.

Sam: This leads us to a fascinating case study the authors provide to illustrate how this works in practice. They describe a 74-year-old immunosuppressed woman on day 10 of mechanical ventilation for SARS-CoV-2 pneumonia who develops refractory hypoxemia.

Maya: The intensivist's pre-test estimates included ventilator-associated pneumonia, pulmonary embolism, and fluid overload. An elevated D-dimer and new right ventricular dilation on bedside echocardiography updated the probability of pulmonary embolism above the testing threshold, while progressively rising compliance argued against worsening ARDS.

Sam: When the team consulted an AI tool as a cognitive adversary, its differential hypotheses included less common mechanisms like air embolism and pulmonary tumor embolism. But the clinical team did not accept those suggestions automatically. They applied pathophysiologic cross-checking.

Maya: An air embolism was considered unlikely because there had been no recent central venous catheter manipulation or removal, no circuit disruption, and no thoracic procedure immediately preceding the deterioration. A pulmonary tumor embolism was deemed improbable given the absence of known malignancy and the absence of radiological findings suggestive of tumor emboli.

Sam: So those alternative mechanisms prompted a targeted search for discordant data rather than further invasive investigation. After this review, the team retained their original plan to anticoagulate for a suspected pulmonary embolism.

Maya: And here is the crucial takeaway from that case: the team explicitly documented that the clinician-LLM concordance did not constitute independent confirmation. Both the clinician and the model reasoned from the same clinical information and the same underlying assumptions.

Sam: Wait, really? That is a massive paradigm shift for health system leaders. If the algorithm is just looking at the same flawed or incomplete chart data you are, its agreement does not mean your initial hunch is suddenly bulletproof.

Maya: Exactly. Concordance should increase confidence only insofar as the conclusion survived independent pathophysiologic and probabilistic scrutiny. Because AI introduces new cognitive biases. There is automation bias, where the user accepts a recommendation because it comes from a machine perceived as objective.

Sam: And algorithmic anchoring, where the first output organizes all subsequent reasoning. Plus cognitive complacency, where the tool reduces the effort to think independently. Human-AI interaction studies show that cognitive forcing functions can reduce this overreliance, encouraging critical thinking rather than automatic acceptance.

Maya: To safely deploy these tools, the authors propose operational principles for bedside use. They present this as a framework requiring evaluation rather than a validated protocol. First, the physician should formulate an independent problem representation before consulting AI in high-risk decisions, reserving AI consultation for diagnostic uncertainty or complexity.

Sam: Second, request differential diagnoses rather than a single answer. Third, do not treat clinician-LLM agreement as independent confirmation. Fourth, require explicit uncertainty by demanding that assumptions and missing data be stated, recognizing that confidence statements generated by LLMs are not reliably calibrated expressions of uncertainty.

Maya: Fifth, test the output against pathophysiology, rejecting outputs that are mechanistically implausible or conflict with bedside findings. Sixth, therapeutic recommendations should be cross-checked against reliable sources before implementation. Seventh, any clinician-AI disagreement should be resolved by the clinician, documented, and revisited with the team, reflecting the patient's values and goals of care.

Sam: Eighth, the medical record should make explicit that the final decision was human, contextual, and accountable. And ninth, identifiable data should not be entered into tools that have not been institutionally approved. These bedside principles must be complemented by institutional governance.

Maya: Institutions must validate AI tools locally and assess their calibration before clinical use. Models must be clearly versioned, with monitoring for performance drift over time. Audit trails should record how AI outputs informed decisions, and mechanisms for bias assessment and incident reporting must be in place.

Sam: So how do we train the next generation of physicians for this reality? The authors note that traditional training often separates basic science, clinical medicine, epidemiology, and therapeutic decision-making. We might end up with learners who use AI to obtain quick answers but do not develop independent judgment.

Maya: A modern curriculum must integrate biomedical knowledge, problem representation, and Bayesian thinking. The question 'what is the diagnosis?' should be replaced by: What is the problem representation? Which hypotheses are likely or dangerous? Which test would change management?

Sam: Simulation is a promising strategy, allowing training in real-time reasoning without patient risk. Scenarios involving refractory shock should require action and explanation, with debriefings examining anchoring and pathophysiologic coherence.

Maya: The authors even propose using AI as a Socratic tutor, but with a pedagogical safeguard: think first, consult afterward, and always criticize. Before using AI, the learner should write the leading hypothesis, alternatives, pathophysiologic mechanism, and initial plan, then consult the AI, and critically compare the responses.

Sam: Training must also include AI literacy. Physicians need to understand training data, algorithmic bias, hallucination, calibration, and explainability. They don't need to be computer scientists, but they must understand that a model may be right for the wrong reason, or wrong with great confidence.

Maya: The ultimate goal is to develop situated pathophysiologic reasoning—thinking from the real patient, in real time, under real constraints. It is not to train physicians who reject AI, nor physicians who depend on it, but physicians capable of thinking with AI to broaden hypotheses and reduce cognitive load while maintaining clinical responsibility.

Sam: To wrap up: this document gives us a testable framework for the ICU. AI algorithms falter when they lack dynamic physiological context. Grounding these tools in mechanistic reality, maintaining human accountability, and using AI as a cognitive adversary rather than an oracle is the only safe path forward.

Maya: Exactly. You can find the full source document linked in our show notes for a deeper dive. And as always, a reminder that this podcast is not medical advice.