California lawmakers are pushing to stop AI from being used as a full-on therapist as interest in chatbot-based mental healthcare has skyrocketed in recent years.
Under Senate Bill 903, companies would be banned from advertising chatbots as therapy, and AI systems themselves would not be allowed to make therapeutic decisions without review from a licensed professional. The bill would also require patients’ permission for health providers to use AI to record therapy sessions or triage care.
“AI has many upsides, and the possibility to improve the lives of Californians across the state, but only if we ensure it is used responsibly,” state Senator Steve Padilla of San Diego, author of the legislation, said earlier this year when introducing the measure.
“AI algorithms are not fit to take over the job of human therapists, who have skills and training that AI is incapable of replicating,” he added. “We must act and place guardrails that prevent AI algorithms from being deployed in a way that is potentially harmful to patients.”
Critics of the bill say it will stifle legitimate uses of AI and could harm the ability of health providers to use AI tools to direct patients toward treatment.
“At a time when every county in California faces a shortage of behavioral healthcare workers, SB 903 still puts a clinician bottleneck in front of the intake and screening tools that help patients reach care faster,” Robert Boykin, California executive director of the industry group TechNet, told CalMatters this week.
The National Union of Healthcare Workers, which supports the bill, has filed a complaint with California regulators alleging the Kaiser Permanente health system used an automated “e-visit” system without oversight by clinicians to make care recommendations and provide initial evaluations of patients who may be facing anxiety or depression.
The company has said it “does not use AI to diagnose patients, make clinical decisions, or determine medical necessity,” and argues the group has mischaracterized its process.
Data indicates massive interest in using AI for therapy, though health experts warn that chatbot-based care is an area that needs further study and regulation to compare with genuine medical care.
A 2025 study from RAND, the Brown University School of Public Health and Harvard University found that about one in eight adolescents and young adults are turning to chatbots for mental health care.
“There are few standardized benchmarks for evaluating mental health advice offered by AI chatbots, and there is limited transparency about the datasets that are used to train these large language models,” Jonathan Cantor, a senior policy researcher at RAND, told the Brown University website of the research.
A study this year from the American Psychological Association, meanwhile, found that 77 percent of psychologists have spoken with patients who have used AI for support, engagement or other reasons.
In October, OpenAI, maker of the popular ChatGPT, disclosed an estimated 1.2 million per week send messages to the chatbot with “explicit indicators of potential suicidal planning or intent.”
Health experts say the phenomenon has flourish amid difficulties access in-person mental health care.
“Given long waiting lists and difficulty accessing mental health care and therapy in many countries, it’s perhaps not a surprise that increasing numbers [of people] are turning to generative AI for support with their emotional wellbeing,” Hamilton Morrin, a clinical fellow in neuropsychiatry at King’s College London, told Harvard Business Review in June.
The makers of popular AI applications, which can speak with humans in remarkably lifelike ways and resemble human beings with full personalities and consciousness, have faced lawsuits alleging chat tools played a role in youth suicides.
If you are experiencing feelings of distress, or are struggling to cope, you can speak to the Samaritans, in confidence, on 116 123 (UK and ROI) or visit the Samaritans website to find details of your nearest branch.
If you are based in the US, and you or someone you know needs mental health assistance right now, call or text 988, or visit 988lifeline.org to access online chat from the 988 Suicide and Crisis Lifeline. This is a free, confidential crisis hotline that is available to everyone 24 hours a day, seven days a week.
If you are in another country, you can go to www.befrienders.org to find a helpline near you.