Drawing the Red Lines: The EU's First Formal Review of Prohibited and High-Risk AI

Sam: Out of 3 791 real-world AI incidents recently analyzed by the European Union, the biggest immediate regulatory threat to health and medtech companies might actually be the gray area surrounding mental health chatbots.

Maya: Today we are unpacking a document published by the European Commission. It is the report adopted pursuant to Article 112(1) of the AI Act on the need to review the list of prohibited AI practices and of high-risk AI systems listed in Annex III.

Sam: And before we dive into how this impacts the boardroom, a quick reminder that our voices are AI-generated, and this podcast is a summary of a publicly available document.

Maya: To set the context here, the AI Act entered into force on 1 August 2024. It is the comprehensive legal framework for artificial intelligence across the European Union. The entire framework relies on a risk-based approach, classifying AI systems into four different risk categories: unacceptable risk, high risk, transparency risk, and minimal to no risk.

Sam: Right, and those top two tiers are where the real compliance hurdles are. The unacceptable risk category is covered by Article 5, which lists specific AI practices that are strictly prohibited. Then you have the high-risk category, covered by Article 6 and Annex III, which lists concrete use cases across eight areas that pose a significant threat to health, safety, or fundamental rights.

Maya: Exactly. But technology moves incredibly fast, and the legislation was built to be flexible. Article 112(1) actually mandates that the Commission assesses whether they need to update these lists once a year. This is the first report doing exactly that.

Sam: And the big question for product teams building these tools is whether the rules are changing right now.

Maya: The short answer is no. The Commission concluded there is no immediate need to amend the lists. But the reasoning is incredibly important. The assessment is heavily limited by time. The rules on prohibited practices only entered into application on 2 February 2025, and the rules on enforcement will only apply as from 2 August 2026.

Sam: Wait, so the enforcement mechanisms are not even fully operational yet? The national competent authorities are still being designated?

Maya: Yes. And the obligations for high-risk systems also apply as from 2 August 2026. The report notes that this timeline is actually under consideration for possible prolongation. A proposed Digital Omnibus on AI might align the timeline to the availability of standards. Once those are available, the rules would start to apply after a 6-month transition period, and in any case no later than 2 December 2027.

Sam: That gives companies a massive window to prepare. But even with that buffer, the Commission still had to conduct this review. How did they actually go about deciding whether the current list is sufficient?

Maya: They used a methodology developed by the AI Office. To amend the prohibited list in Article 5, they look for two cumulative conditions. First, there have to be new or emerging AI practices that undermine EU values or fundamental rights enshrined in the Charter of Fundamental Rights. Second, there must be regulatory gaps persisting despite other applicable EU legislation.

Sam: So if an existing privacy law or consumer protection law handles the harm, the AI Act does not necessarily need to step in and ban it entirely.

Maya: Right. And for adding to the high-risk list in Annex III, the threshold is very specific. They can only add new use cases if the AI systems pose a risk of harm "equivalent to or greater than those already listed".

Sam: That phrase is crucial. The report explicitly says the threshold for amending the list is not merely technological novelty or public concern. It has to be a demonstrable and comparable risk.

Maya: To gather evidence for this, they engaged with the Member States, they ran a multi-stakeholder consultation, and they conducted empirical research by analyzing AI incident databases. Let us look at the Member State feedback first.

Sam: It sounds like they essentially polled the national authorities across the continent.

Maya: They did. During the meetings, the majority of the Member States present stated that they do not see a need to amend the prohibited AI practices right now. In addition, nine Member States submitted written statements via e-mail confirming there is no perceived necessity. Similar responses came from two observers.

Sam: And it was a similar story for the high-risk use cases, right?

Maya: Yes. Seven Member States submitted written statements via e-mail confirming that they do not see any need to amend Annex III at this stage. They all pointed out that there is little concrete data available yet, and they need practical experience first.

Sam: But they did flag some very specific issues for future monitoring, and this is where it gets incredibly relevant for clinicians and digital health vendors. One Member State specifically raised concerns about self-help therapy AI chatbots.

Maya: Exactly. These companions and therapy bots are presented as wellbeing tools, but they process highly sensitive personal data and give emotional guidance that could negatively impact an individual's mental health. The Member State asked if these should be classified as high-risk.

Sam: So how did the Commission view that? It feels like a massive gray area.

Maya: The Commission provided a clear decision tree. If a self-help chatbot fulfills the definition of a medical device, meaning it is intended for the treatment or alleviation of disease, it is subject to the strict requirements of the Medical Device Regulation.

Sam: So that takes it out of the AI Act's direct high-risk classification and into established medical regulation. But what if they just market it as general wellness to avoid that?

Maya: If it acts as a wellness tool but uses manipulative or deceptive techniques that exploit vulnerabilities, it potentially falls under the outright prohibitions in Article 5(a) and (b). If it includes functionalities for emotion recognition, it is classified as high-risk under point 1(c) of Annex III.

Sam: And if it does not do any of those things? If it is just a basic conversational bot?

Maya: Then the Commission says most of these systems would not fall under the current Annex III use cases. But they would still be subject to the disclosure obligations in Article 50, meaning they have to inform the user that they are interacting with an AI.

Sam: That is a fascinating regulatory patchwork for any startup building conversational AI for patients. You have to thread the needle between medical device rules, the ban on manipulation, the high-risk status of emotion recognition, and basic transparency.

Maya: And the Commission noted they will be closely monitoring this space to collect further evidence in the coming years. Another area a Member State flagged was AI-driven personalized learning. But the Commission concluded that self-taught or independent learning AI systems do not in most cases pose risks equivalent to or greater than those in Annex III. There is limited empirical evidence of high risk there, and no incidents were identified.

Sam: What about security threats? Did anyone suggest banning AI that writes malicious code?

Maya: One Member State did suggest assessing AI systems intended to develop or distribute malware for inclusion in the prohibited list. But the Commission considered that those risks seemed sufficiently covered by various EU cybersecurity legislation, though they flagged it for further monitoring.

Sam: Okay, so that was the Member States. What about the broader industry and public? You mentioned a multi-stakeholder consultation.

Maya: Yes, they ran a consultation from 6 June to 18 July 2025. In total, 547 responses were received. A total of 409 respondents indicated they were representing an organization.

Sam: How did that break down? Was it mostly corporate?

Maya: Heavily corporate. The largest group was companies with 150 responses, followed by associations with 93 responses, and 81 respondents who selected 'other organization'. Civil society organizations or associations accounted for 56 responses.

Sam: Did the academic side weigh in at all?

Maya: They did. There were 11 responses from research institutes, 7 responses from universities, 6 responses from think tanks, and 5 responses from consumer organizations. The rest were individuals.

Sam: And how many individuals responded?

Maya: Quick note before we carry on. This spot is open for a sponsor. If your company builds or sells AI for healthcare and wants to reach the clinicians, health-system leaders and industry teams who listen to this show, the link to our sponsorship page is in the show notes.

Sam: And now, back to the document.

Maya: A total of 138 respondents indicated they were acting in an individual capacity. Among them were 35 independent experts, 34 providers of an AI system, 28 from academia, 13 deployers of an AI system, and 14 who just selected 'other'.

Sam: A very thorough snapshot. What did this group actually want changed?

Maya: A significant number considered the current list of prohibited practices sufficient. Slightly more than 10% of respondents proposed adding new prohibitions. A lot of them wanted to broaden the ban on emotion recognition.

Sam: Right now, emotion recognition is prohibited in workplaces and educational institutions under Article 5(1)(f), right?

Maya: Yes, except for medical or safety reasons. In all other areas, it is classified as high-risk. Several respondents wanted it explicitly prohibited in sensitive contexts like migration.

Sam: Did the Commission agree to ban it in migration?

Maya: No. They stated there is a lack of empirical evidence showing that classifying those systems as high-risk is not sufficient to ensure fundamental rights are protected. So no immediate amendment, but it is flagged for further consideration.

Sam: What about dark patterns? I know a lot of health apps and consumer devices worry about the line between engaging user interfaces and outright manipulation.

Maya: Several stakeholders raised the issue of AI systems enabling dark patterns and addictive design. The Commission noted the most harmful instances are already covered by prohibitions in Article 5(a) and (b), as well as the Unfair Commercial Practices Directive and the Digital Services Act. So, no new specific prohibition was added.

Sam: And for the high-risk use cases? Did stakeholders suggest expanding Annex III?

Maya: Around 20% of the respondents suggested adding new use cases or extending current ones. A significant number of suggestions concerned point 5 of Annex III. Stakeholders questioned the lack of explicit references to AI used in public administration for managing access to social welfare benefits and social housing. They also noted that AI systems used in tenant screening are not clearly covered.

Sam: So algorithms acting as gatekeepers to basic living essentials. That makes sense that people want them labeled high-risk.

Maya: They also mentioned essential insurances beyond just life and health insurance, like motor vehicle or disability insurance. And some suggested classifying AI used in debt collection and enforcement as high-risk, noting its growing use by energy, telecom, and financial providers.

Sam: This word 'empirical' keeps coming up. The Commission clearly does not want to legislate based on hypotheticals. They want proof of harm. So how did they measure that real-world harm?

Maya: The third pillar of their methodology was empirical research using AI incident databases. For devising the methodology, they analyzed the AI Risk Repository of the Massachusetts Institute of Technology, or MIT. But the main database they relied on was the OECD AI Incidents and Hazards Monitor.

Sam: How much data do we actually have on AI causing real-world harm?

Maya: The timeframe covered incidents reported from 1 January 2024 to 31 May 2025. In total, 3 791 incidents were reported in the OECD database during this period.

Sam: 3 791 incidents in about a year and a half. How did they process all of that?

Maya: They categorized all 3 791 incidents into six categories. The first was incidents falling outside the scope of the AI Act. The second was incidents likely addressed by Annex I. The third was incidents likely covered by Article 50, which is transparency. The fourth was hypothetical risks.

Sam: And the last two categories?

Maya: The fifth category was incidents potentially addressed by Article 5, meaning prohibited practices. And the sixth was incidents potentially addressed by Annex III, meaning high-risk systems.

Sam: So how did the numbers shake out across those buckets?

Maya: Almost half of all reported incidents fell outside the scope of the AI Act or were sufficiently covered by other EU instruments, like the GDPR. Approximately 10% of the reported incidents were likely to be addressed by Annex I.

Sam: So what portion was left for the deep dive into prohibited or high-risk systems?

Maya: Around 30% of reported incidents were identified by the Commission as relevant for a more detailed analysis. Approximately two thirds of these incidents either fell within the scope of Article 5 or were considered relevant for a prohibition assessment. The remaining one third were likely to fall under the high-risk headings in Annex III.

Sam: When they analyzed that 30%, did they find anything that the current law completely missed?

Maya: They found a few highly concerning things. One of the biggest revelations in this entire report is what the Commission calls a "potential regulatory gap" regarding deepfakes. Specifically, AI systems generating child sexual abuse material, or CSAM, and non-consensual sexually explicit images.

Sam: Wait, really? Generating non-consensual explicit deepfakes is not already a prohibited AI practice under the AI Act?

Maya: Legally, no. The Commission analyzed whether the existing prohibitions in Article 5(1)(a) or Article 5(1)(b) would cover this. Those articles prohibit AI that deploys purposefully manipulative or deceptive techniques, or exploits vulnerabilities due to age or disability, to cause significant harm.

Sam: And generating a deepfake of someone does not fit that legal definition?

Maya: Right. The Commission concluded that these practices do not manipulate children and victims to engage them in harmful behavior. The harm is done to them by the creation of the image, not by manipulating their actions. So it falls outside the exact wording of the prohibition.

Sam: That is a massive limitation of focusing the prohibition specifically on user manipulation rather than content generation. I imagine they are going to fix that?

Maya: Because they cannot just rewrite Article 5 without a formal legislative amendment, they have flagged it for further analysis. They also flagged AI systems facilitating scams and financial fraud, noting it requires a plausible causal link and a threshold of "significant financial harm" to be met before it becomes a prohibited practice.

Sam: What about incidents related to high-risk use cases? Did the data uncover anything surprising there?

Maya: A significant number of incidents involved political disinformation campaigns. These thematically fall under point 8 of Annex III, which covers the administration of justice and democratic processes, but they do not clearly appear in the specific use cases listed.

Sam: But aren't there other laws for political ads and deepfakes?

Maya: Yes, the Commission noted that the Digital Services Act and the Regulation on the Transparency and Targeting of Political Advertising address some of these issues, along with measures in the recently adopted Communication on the European Democracy Shield. They will clarify in upcoming guidelines if these practices should be classified as high-risk under the AI Act.

Sam: It sounds like the Commission is actively avoiding double-regulation. If another EU framework covers the risk, the AI Act stays in its lane.

Maya: Exactly. The report concludes that a more substantive evaluation will only become possible after the prohibitions have applied for at least a year. Regulatory sandboxes established in accordance with the AI Act will serve as a mechanism for regulatory learnings and evidence collection. The next assessment pursuant to Article 112 is due in 2026.

Sam: A crisp recap: The European Commission's first formal review leaves the AI Act's risk tiers unchanged for now, but puts companies on notice that gray areas are under strict observation. A reminder that the source is linked in the show notes, and that this is not medical advice.