In January, the venture capital firm Andreessen Horowitz announced that it had backed Slingshot AI, the world’s first foundation model for psychology, bringing the startup’s total capital to $40 million. A few weeks later, the European Union’s AI Act, which includes a ban on manipulative AI systems, came into force.
These two events highlight a troubling contradiction. Even as regulators attempt to protect users from deceptive AI practices, investors are betting that AI chatbots can treat people struggling with mental-health issues – in other words, when they are especially vulnerable to exploitation.
Worse, the way that large language models are currently trained may make them fundamentally incapable of providing such treatment.
The mental-health market is huge, and the use of generative AI is poised to expand significantly. The United States National Institute of Mental Health estimates that one in five US adults has a mental illness.
But more than 122 million people in the US live in an area with a shortage of mental-health providers. This has given rise to a slew of AI chatbots that promise to fill the gap. Wysa, for example, calls itself the “clinical alternative to ChatGPT” and claims to have helped six million people in 95 countries.
The global mental-health crisis demands innovative solutions, and AI will be an essential component. But if AI technologies are to expand access to quality care and promote long-term healing, investors should demand evidence of effective therapeutic outcomes before funding the next chatbot therapist.
But AI chatbots’ behavior is at odds with the delicate balance of empathy and confrontation that evidence-based psychotherapy requires. Mental-health professionals must validate patients’ experiences while challenging the rigid thinking that perpetuates psychological distress. This productive discomfort helps patients examine their assumptions, driving meaningful change.
Consider a patient who avoids social situations, claiming that they prefer solitude instead of acknowledging their social anxiety. A skilled therapist might gently challenge them by asking if something else is informing that preference – perhaps a fear of how others might react. This opens space for self-reflection without attacking the patient’s conception of self.
Current AI models tend to avoid such confrontations. In April, OpenAI rolled back the GPT-4o update because it was “overly flattering or agreeable – often described as sycophantic.”
It reportedly praised one person’s plan to “sell shit on a stick” as “genius” – an obvious example of prioritizing agreeableness over accuracy. Researchers have found that sycophancy is “a general behavior of AI assistants” that likely stems from the way these models are trained, particularly the use of human feedback for fine-tuning.
When human evaluators consistently rate validating responses more favorably than challenging ones, AI assistants learn to echo, rather than question, the user.
In mental-health contexts, this tendency toward agreement may prove problematic, because psychological disorders often involve cognitive distortions that feel true to the individual and thus contribute to their distress.
For example, depressed people tend to feel worthless or hopeless, while anxiety is often associated with catastrophic thinking. An AI chatbot programmed to be agreeable might reinforce these harmful thought patterns by focusing solely on validation, rather than introducing alternative points of view.
As governments grapple with how to regulate AI, mental-health applications present unique challenges. While the EU’s ban on manipulative AI is a good first step, it does not address the subtler problem of current models’ excessive agreeableness.
The US has no comprehensive federal laws or regulations for AI – and judging by President Donald Trump’s AI Action Plan, none will be forthcoming.
This regulatory gap will grow more dangerous as US venture capital firms increasingly pour money into AI tools that provide psychological support, and as these tools scale globally, reaching places where access to mental health care is even more limited.
Addressing AI’s sycophancy problem requires fundamental changes to how these systems are designed and used. Instead of optimizing for user satisfaction, AI chatbots that provide mental health care should be trained to recognize when therapeutic challenge is necessary.
That could mean incorporating therapeutic principles and examples of effective therapeutic interventions into training strategies.
Claims about AI revolutionizing mental health care remain premature. Until it can master the very specialized ability of therapeutic confrontation – sensitively but firmly questioning patients’ assumptions and offering alternative perspectives – it could end up harming those it is meant to help.
Crucially, health professionals and patients must play a central role in developing these tools, given their insights into which therapeutic interactions are helpful and which are harmful.
Meaningful patient involvement in design and deployment would ensure that the models serve end users’ real needs, not what tech leaders assume they want.
The global mental-health crisis demands innovative solutions, and AI will be an essential component. But if AI technologies are to expand access to quality care and promote long-term healing, investors should demand evidence of effective therapeutic outcomes before funding the next chatbot therapist.
Likewise, regulators must explicitly require these technologies’ developers to demonstrate clinical efficacy, not just user satisfaction. And policymakers should pass laws that mandate the inclusion of mental-health professionals and patients in the training of AI models aimed at providing this kind of care.
Claims about AI revolutionizing mental health care remain premature. Until it can master the very specialized ability of therapeutic confrontation – sensitively but firmly questioning patients’ assumptions and offering alternative perspectives – it could end up harming those it is meant to help.
(Marc Augustin, a German board-certified psychiatrist/psychotherapist, is a professor at the Protestant University of Applied Sciences in Bochum, Germany, and a SCIANA fellow)
Copyright: Project Syndicate
Comment