21 November 2026 · 17 min

Who Catches Cancer Patients in Crisis? AI Chatbots, Distress, and the Safety Gap

OncologyMental HealthEvidence & ResearchSafety, Ethics & Liability

In this episode, we unpack a 2026 scoping review examining conversational AI agents used for psychological support in cancer care. We explore the massive gap between testing AI for user experience and actually equipping it to handle clinical distress. Listeners will learn why fewer than a fifth of these projects formally evaluate safety, and what that means for developers and oncology teams building psycho-oncology tools.

Key points

Source: Mapping Technologies, Support Functions, Evaluation Outcomes, and Safety Governance Related to Conversational AI Agents for Psychological Support of Patients With Cancer: Scoping Review - Journal of medical Internet research, 2026 (CC BY)

This spot is available. Reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsor the show

This episode is an AI-generated conversation summarising a public document; the hosts' voices are synthetic. It is for information only and is not medical advice. Always refer to the original source.

Authors: Zhiying Shen, Shuangjiao Shi, Lian Mao, Chengyuan Li, Jianfei Xie · DOI: 10.2196/105599

Your company here. This podcast is looking for its first sponsors: reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsorship options and rates →

Transcript

Maya: Did you know that when developers build conversational AI to support cancer patients, 70.8% of their reports measure acceptability or user experience, but just 16.7% explicitly report evaluating safety or adverse events?

Sam: That is a staggering gap, especially for vulnerable patients. Welcome to AI in Medicine - Smart Summaries. I should mention right up front that our voices are AI-generated, and today we are providing a summary of a public document.

Maya: Exactly. We are diving into a scoping review mapping technologies, support functions, evaluation outcomes, and safety governance related to conversational AI agents for psychological support of patients with cancer.

Sam: This review tackles a massive global challenge. The document cites the Global Burden of Disease Study 2023, which points to the massive scale of new cancer cases and cancer deaths globally in 2023. When you have that level of disease burden, the psychological toll is immense.

Maya: It absolutely is. The authors point to an umbrella review that reported pooled prevalence estimates of 33.16% for depression and 30.55% for anxiety among patients with cancer. That means roughly a third of this population is dealing with severe, measurable distress.

Sam: And that is exactly why digital interventions, specifically conversational AI agents, are stepping into the spotlight. The document notes that resource limitations, stigma, and organizational barriers severely restrict access to traditional psychosocial assessment and referral. AI offers a potentially scalable lifeline.

Maya: But as a clinician, my immediate question is always, what exactly are we handing to our patients? This scoping review aimed to map out the technologies, the psychological support functions, the evaluation outcomes, and the safety governance of these tools. They searched 13 databases and additional registers through May 5, 2026.

Sam: And what did they find? How many of these AI tools are actually out there being tested or used in the literature?

Maya: They included 24 reports representing 25 study units and 23 underlying projects. They looked at studies involving patients with cancer or cancer survivors, focusing on real or explicitly simulated patient-facing oncology scenarios with interactive text, voice, or multimodal dialogue.

Sam: So out of those 23 underlying projects, what kind of AI are we talking about? Is this all just generic chatbots, or are there different architectures at play?

Maya: It is a real mix, which shows how fragmented the field is. Large language model or generative AI systems accounted for 39.1% of the projects, which is 9 out of 23. Rule-, knowledge-, or retrieval-based text agents made up 26.1%.

Sam: That makes sense. The review points out that this transition from rule-based to generative AI changes how these systems interact, noting that system architecture affects the balance between conversational flexibility and content control. Open-ended generative models might be more conversational, but rule-based systems allow for tighter control over the response pathways.

Maya: Exactly. And the diversity does not stop there. Voice assistants or smart speakers accounted for 13% of projects. Social robots or embodied agents were 4.3%. Another 13% were just concepts or perceptions with no fixed deployed system, and 4.3% was a multimodal monitoring platform and chatbot.

Sam: So the technology is incredibly varied. But what are these AI agents actually programmed to do for the patients? Are they acting as therapists, or just answering basic logistical questions?

Maya: That is where the data gets fascinating. Across the 23 projects, information or education was mapped in 23 of them, so 100%. Symptom or self-management support was mapped in 87%, which is 20 out of 23.

Sam: Okay, so almost all of them are giving out information and helping track symptoms. What about actual psychological support?

Maya: Navigation or resource recommendation was found in 60.9%. Emotional support, which they defined as empathic or validating dialogue, was at 52.2%. Motivation or patient activation was 43.5%.

Sam: That sounds like a lot of tools are trying to handle emotional weight. Did they deliver actual structured therapeutic interventions, like cognitive behavioral therapy?

Maya: That is where the numbers drop. Structured psychological or behavioral components were mapped in just 30.4% of projects, or 7 out of 23. Disclosure or communication facilitation was at 26.1%. But Sam, there is a massive caveat to these numbers that health system leaders need to understand before buying one of these systems.

Sam: What is the caveat?

Maya: Those percentages include delivered, planned, simulated, co-designed, and participant-reported functions. When the authors ran a sensitivity analysis looking exclusively at systems that were actually delivered in an implemented system, the numbers shrank significantly.

Sam: Wait, really? How much did they shrink?

Maya: In the delivered-only analysis, information or education dropped to 56.5%, or 13 projects. And those structured psychological or behavioral interventions? They dropped to just 21.7%, which is only 5 out of 23 projects.

Sam: So the reality on the ground is that very few of these bots are actually delivering structured therapy to patients. As the authors write, information and self-management functions may be easier to anchor to curated content or predefined workflows than sustained psychological interventions. Therapy requires continuity and the ability to respond if the patient deteriorates.

Maya: Exactly. Which brings us to how these 24 reports actually evaluated the AI. If they are mostly testing basic information delivery, what outcomes are they measuring? The answer is heavily skewed toward user experience.

Sam: You mentioned the 70.8% stat at the top of the show. Let's break that down.

Maya: Right. Acceptability, satisfaction, or user experience was the most frequently reported evaluation domain, showing up in 70.8% of the 24 reports. Engagement, use, or adherence was reported in 54.2%.

Sam: Measuring if a patient likes an app or keeps using it is standard for early-stage feasibility studies. And the review notes that feasibility was reported in 25% of reports, and usability in 33.3%. But what about clinical outcomes? Did these bots actually reduce the depression and anxiety we talked about earlier?

Maya: Only a third of the studies looked at that. Psychosocial clinical outcomes were reported or planned in just 33.3%, or 8 out of 24 reports. Behavioral or self-management outcomes were in 37.5%. Symptom, sleep, or quality of life outcomes were at 25%.

Sam: That is a stark contrast. We have a lot of data on whether patients like talking to the AI, but very little on whether it durably improves their mental health. And we cannot forget the technology's own performance. The review states that AI response quality, factuality, hallucination, or contextual error outcomes were reported in 29.2% of reports.

Maya: And for hospital administrators looking at workflow integration, outcomes related to health care professionals or clinical workflows were reported in only 20.8%, which is 5 reports. But the most alarming gap for me, as a clinician, is the safety evaluation.

Sam: This is the 16.7% figure you opened with. Out of 24 reports, only 4 explicitly evaluated safety or adverse events. Why do you think that is? Are developers just assuming the bots are safe?

Maya: I think it comes down to how safety is governed and defined in digital health right now, which this review maps out brilliantly. They looked at safety and governance reporting across the 24 reports, and the split between administrative ethics and operational safety is wild.

Sam: Lay out the numbers for us. How did they measure governance?

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: Ethics-related procedures were reported in 91.7% of reports. So almost everyone is getting their standard institutional approvals. Privacy or data protection was explicitly reported in 54.2%.

Sam: So the baseline paperwork is largely getting done. What about keeping an eye on the AI's actual output?

Maya: Human or expert oversight was reported in 62.5% of reports. Accuracy or hallucination management was at 50%. But when you look at the clinical risk mechanisms, the safety net falls completely apart.

Sam: What do you mean by clinical risk mechanisms? Like what happens if a patient tells the chatbot they are in a severe crisis?

Maya: Exactly. Crisis escalation, referral, or high-risk response protocols were only described in 33.3% of the reports. That is just 8 out of 24. If an oncology patient expresses severe distress to an open-ended generative AI system, only a third of these reports have a documented plan for how to escalate that to a human clinician.

Sam: That is a massive liability gap. The document notes that while traditional privacy and ethics procedures are deeply embedded in standard research processes, the specific operational safeguards needed for open-ended AI conversations are still very new and lack standardization. Treating safety as an empirical outcome rather than an assumed property is desperately needed.

Maya: It absolutely is. And the transparency is lacking too. User-facing AI identity disclosure was reported in just 12.5% of reports, which is 3 out of 24. Explicit role boundaries were in 25%. And explicit statements that the system did not replace professional care were reported in only 20.8%.

Sam: Wait, only 20.8% explicitly stated the AI does not replace a human professional? That feels like day-one legal advice for any digital health company.

Maya: You would think so! And regarding that 62.5% that reported human oversight, we have to be careful not to misinterpret that. The document specifies that oversight encompassed everything from intervention development and content validation, to response review, real-time monitoring, clinical supervision, and referral.

Sam: Ah, I see. So having human oversight might just mean a doctor reviewed the chat logs after the study ended, or helped write the initial prompts. It does not mean a nurse is actively watching a dashboard in real-time ready to intervene.

Maya: Precisely. The authors state that just because 15 out of 24 reports describe some form of oversight, it does not mean there were 15 actively clinically monitored systems. This is why they point to reporting frameworks like DECIDE-AI, SPIRIT-AI, and CONSORT-AI, which demand clear documentation of responsibilities and safety monitoring.

Sam: Let's shift gears to how these tools are being built psychologically. If they are talking to vulnerable patients, are they rooted in established therapeutic theory?

Maya: Not consistently. An explicit theoretical or conceptual framework was reported in only 25% of the reports. Another 25% described therapeutic models or components, like cognitive behavioral therapy or mindfulness, without an explicit theory-guided framework.

Sam: Which leaves 50% that reported no explicit theoretical basis at all. Half of these reports are essentially flying blind when it comes to the psychological mechanisms of action. The review suggests future studies should link support functions to proposed mechanisms, like emotion regulation or reduced avoidance, to clarify how the conversational components are actually intended to work.

Maya: Exactly. You cannot just throw a conversational bot at a patient and hope their anxiety goes down. You have to know the mechanism. Which brings me to another major blind spot identified in this review: population representation and equity.

Sam: Who are they actually testing these tools on? Cancer affects everyone, but digital health tools historically skew toward younger, wealthier, more digitally literate populations.

Maya: The reporting here was deeply inconsistent. Pediatric or adolescent and young adult representation was explicit in 25% of reports. Adults aged 65 years or older were explicitly represented in 29.2%. And patients with advanced or metastatic disease were only reported in 16.7%.

Sam: That is concerning, given that advanced disease often correlates with the highest psychological distress. What about other demographic factors?

Maya: Education, occupation, or income-related variables were reported in 58.3% of reports. Explicit language eligibility was in 66.7%. Digital access or literacy was reported in 45.8%.

Sam: And race or ethnicity?

Maya: Reported in just 16.7%, or 4 out of 24 reports. And here is a number that should wake up anyone building for broad health systems: no report established participant-level rurality. Zero.

Sam: Zero? That is a massive oversight. Patients in rural areas are often the ones who stand to benefit most from remote, scalable AI support because they live far from major psycho-oncology centers. The authors note the central concern is not the demonstrated exclusion of these groups, but their limited ability to judge who can access, use, and benefit from the systems studied.

Maya: Exactly. If you are a hospital system buying an AI tool, you need to know if it works for an older rural patient with limited digital access, not just a younger patient in a city. This review proves we simply do not have that data yet.

Sam: So where does this leave us? The authors lay out some clear implications for psycho-oncology practice and care delivery. How should clinical teams and developers move forward based on this map?

Maya: The findings support a risk- and function-sensitive role for conversational AI. Information, education, navigation, and self-management functions may complement existing care by extending access to standardized support. They can prep a patient between visits. But the complex emotional support requires much stronger evidence of therapeutic benefit, and the review emphasizes that current findings do not establish effectiveness or safety.

Sam: In other words, these tools should be evaluated as an adjunct to professional psycho-oncology care, not a substitute for it. And the professional responsibilities should map directly to the system's functions. Mental health professionals need to guide the therapeutic content, oncologists guide the disease accuracy, nurses guide the symptom monitoring, and the technical teams handle incident management.

Maya: Precisely. If you are in the boardroom funding these startups, or on the ward piloting them, you need to ask for the crisis escalation protocols. You need to ask to see the prospective safety evaluation, not just the user experience surveys.

Sam: Before we wrap, we should note the limitations of this review. The search ended on May 5, 2026, so it may not capture very recent commercial systems. The included studies range from early protocols to full trials, so they do not represent equivalent levels of clinical evidence. And importantly, because they did not do a formal critical appraisal or meta-analysis, this map does not establish actual clinical effectiveness or safety.

Maya: Right, and general-purpose large language models were only included if they were evaluated in defined oncology support contexts. So this review does not cover what happens when a cancer patient just opens a generic chatbot on their phone for unstudied, independent advice.

Sam: Which is a whole other layer of risk. Time for our memorable takeaway. Maya, what is the one thing our listeners should walk away with today?

Maya: The takeaway is that the AI psycho-oncology field is fantastic at proving a chatbot is easy to use, but dangerously behind on proving it is safe during a psychiatric emergency. With crisis escalation protocols reported in just 33.3% of these studies, developers and health systems must fundamentally shift their focus from building engaging bots to building secure, medically integrated safety nets.

Sam: A vital wake-up call for the industry. To recap, we unpacked a scoping review of 23 conversational AI projects in cancer care. We saw that while 100% provided information, structured psychological interventions were much rarer, especially in delivered systems. We explored the glaring gap between user experience testing and rigorous safety governance, and highlighted the missing demographics in the research.

Maya: You can find a link to the full open-access scoping review in our show notes. It is a must-read for anyone deploying digital health tools in high-stakes clinical environments.

Sam: Thanks for joining us on AI in Medicine - Smart Summaries. As always, this podcast provides a summary of public documents for educational purposes and is not medical advice. We will catch you next time.