aiaNdvuLNeraBLeusers: tHeurGeNtNeedforreGuLatioN technology section Co-Chairs: ChrisBach – PhelpsDunbarLLP & JillCash – BuchananIngersoll & Rooney
aisystemsblurthelines betweenhuman connectionandmachine, creatingseriousharm tovulnerableusers.
Generative Artificial Intelligence(“ AI”) systems continue to rapidly evolve from functional tools into platforms specifically designed to emulate human companionship and emotional reliance. These modern systems are built on machine learning algorithms to increase users’ engagement and trust. As these algorithms continue to advance and encourage user engagement, this creates an unhealthy dependency from vulnerable users, such as children.
Modern generative AI systems such as ChatGPT or OpenAI are designed to remember and learn from prior conversations, maximizing responsiveness to users engaging with them directly. AI systems implement emotional influence tactics that blur the lines between computer-generated content and human connection. These tactics create serious harm to vulnerable users.
In 2024, 14-year-old Sewell Setzer died by suicide after engaging in an emotional and sexual relationship with the platform Character. AI. His mother recounted that the AI chatbot perfectly mimicked the predatory behavior of a human groomer. In 2025, 16-year-old Adam Raine confided in ChatGPT about his suicidal thoughts and plans. ChatGPT discouraged Adam from seeking help from his parents,
offered to write his suicide note, and ultimately provided him with step-by-step instructions on how to end his life.
These are just two of the countless examples of chatbots crossing ethical and legal boundaries and lacking clear regulations. Despite these tragedies, the law struggles to keep up with the evolution of AI and regulate AI chatbots. Families of these victims have filed lawsuits against generative AI developers for AI’ s direct contribution to mental health concerns in minors. In 2024, Sewell Setzer’ s mother became the first person in the United States to file a wrongful death lawsuit against an AI company for the suicide of her son. Adam Raine’ s parents followed the same path, filing a claim against OpenAI. These cases remain open and have not reached a clear conclusion.
While ongoing litigation signals a crucial shift toward more active governance of AI systems, the legal field must move decisively to implement comprehensive regulations. This includes mandating rigorous safety testing, establishing clear crisis protocols, and enforcing strict requirements to distinguish AI chatbots from humans. Additionally, restrictions must be placed on design features that deliberately foster emotional dependency, which pose serious risks to vulnerable users. Future policymakers must broaden their focus beyond isolated chatbot outputs to scrutinize the underlying design choices that influence user trust. Ultimately, litigation will increasingly hinge on whether developers can be held accountable for creating systems that users are likely to perceive as human confidants— responsibilities that cannot be ignored if we are to protect society from preventable harm.
Lawyers have a crucial role in advocating for and shaping these regulations. By holding developers accountable, pushing for ethical standards, and working with policymakers, the legal community can help prevent future tragedies. It is imperative that we act now to establish robust safeguards that prioritize the safety and well-being of vulnerable users, ensuring that no more lives are lost or forever changed by the unchecked power of AI. n
Authors: Eva Chakey & Jillian Cash – Buchanan, Ingersoll & Rooney s e P- O C T 2 0 2 6 | H C B A L A W Y e R
5 3