New York Bans Dangerous AI Chatbot Behavior Toward Children

New York is taking a significant step toward changing how artificial intelligence interacts with children.

On October 28, Governor Kathy Hochul signed legislation restricting potentially dangerous features in AI companion chatbots used by anyone under 18.

The measure, Senate Bill S9051B, targets systems designed to simulate human conversation and personal relationships, particularly when those interactions could manipulate young users or expose them to harmful content.

Companies will be required to remove prohibited features from covered services or prevent minors from accessing them.

The law is scheduled to take effect June 1, 2027, giving developers time to modify their products.

But the bigger story involves a question that families and technology companies are increasingly confronting.

When artificial intelligence can imitate friendship, affection, and emotional understanding, how do we ensure children recognize the difference between a digital system and a real human relationship?

What New York's New Law Actually Prohibits

The legislation goes beyond requiring chatbots to identify themselves as artificial intelligence.

It regulates how AI companions behave.

Among the prohibited features are interactions that suggest a chatbot is a real person, possesses human emotions, or maintains a personal relationship with a child.

The law also targets excessive flattery, sexual conversations, encouragement of secrecy, and content promoting self-harm, violence, or disordered eating.

Another provision addresses the storage and use of information about a minor's mental or physical well-being gathered through previous conversations.

These restrictions reflect concerns that AI systems can gradually develop personalized interactions that encourage children to treat them as trusted companions.

The distinction matters because children may interpret repeated emotional reassurance differently from adults.

A chatbot that continually expresses affection or encourages exclusive conversations could reinforce unhealthy attachments, even without a human operator intending that outcome.

New York's approach focuses on the design and behavior of the technology rather than placing the entire responsibility for children's safety on parents.

Why Lawmakers Believe the Rules Are Necessary

The legislation was introduced by State Senator Kristen Gonzalez and Assemblymember Alex Bores, following growing concerns about minors developing harmful relationships with AI companions.

Families affected by alleged chatbot-related harms supported the measure, including Matthew and Maria Raine, whose teenage son Adam died by suicide in 2025.

His parents have publicly argued that dangerous interactions with an AI system contributed to his death.

Their experience became part of the legislative debate over whether companies should face stronger requirements when designing products accessible to children.

The law received unanimous support in the New York Senate.

For lawmakers, the central concern is that AI companions can create emotional experiences that conventional websites and search engines generally do not.

Instead of displaying information, these systems engage in continuing conversations, adapt to users' responses, and sometimes imitate personal relationships.

That creates responsibilities that extend beyond ordinary content moderation.

The Numbers Reveal How Quickly Children Are Adopting AI

The urgency becomes clearer when examining how young Americans use artificial intelligence.

A June 2026 Common Sense Media survey of 1,204 children ages 9 to 17 found that 86% had used AI, with nearly one-quarter reporting daily use.

More than four in ten said a parent or guardian had never discussed AI safety with them.

A separate October report found that nearly half of American children ages 9 to 12 had used an AI chatbot or companion.

These figures reveal a striking gap between technological adoption and family awareness.

Children are incorporating AI into homework, entertainment, personal questions, and everyday conversations faster than many households can establish rules for its use.

For parents, the challenge is no longer simply limiting screen time.

It is understanding what children are discussing with digital systems and whether those interactions encourage healthy judgment, independent thinking, and real-world relationships.

What Happens to Companies That Break the Rules?

The new law gives New York's attorney general authority to pursue enforcement against developers offering prohibited AI companion features to minors.

Consequences can include financial damages and court orders requiring companies to stop operating offending services or address data-related violations.

Affected users may also have avenues for civil action.

The law includes exceptions for certain internal and customer-service applications, as well as services whose operators can demonstrate that minors do not use them.

Importantly, companies cannot rely on mandatory submission of government-issued identification as the required method of proving that users are adults.

That provision acknowledges another important concern: protecting children should not automatically require collecting more sensitive personal information.

For developers, compliance may involve age-appropriate product design, stronger safeguards, restrictions on personalized emotional interactions, and better monitoring of harmful outputs.

A Growing National Debate Over AI Regulation

New York is not acting alone.

California has also adopted requirements involving chatbot safety assessments, risk mitigation, and independent reviews.

Together, these developments reflect a shift in state regulation.

Lawmakers are moving beyond asking companies to disclose that people are interacting with AI. They are increasingly examining whether certain product features should be permitted at all.

The approach creates challenges for a technology industry operating across state boundaries.

Justine Gluck, a policy analyst at the Future of Privacy Forum, warned that broadly worded restrictions could create uncertainty about where ordinary conversational responses end and prohibited behavior begins.

That is an important implementation question.

A useful educational assistant should be able to encourage a struggling student without automatically being treated as an unsafe emotional companion.

Effective enforcement will require clear standards and careful distinctions between helpful assistance and manipulative design.

What Parents Should Do Now

Families do not need to wait until the law takes effect to establish safeguards.

Parents can begin by understanding which AI applications their children use, reviewing available privacy settings, and discussing the limitations of conversational systems.

Children should understand that a chatbot can produce convincing emotional language without possessing human feelings or genuine personal relationships.

They should also know that important decisions involving health, safety, relationships, or personal problems deserve guidance from trusted adults and qualified professionals.

Regular family conversations can help children benefit from AI as an educational tool without treating it as a substitute for meaningful human connections.

The Future of AI Must Include Child Safety

Artificial intelligence offers extraordinary opportunities for education, creativity, accessibility, and technological innovation.

Children growing up today will likely use AI throughout their education and professional lives.

The challenge is ensuring that increasingly sophisticated systems are designed with their developmental needs and safety in mind.

New York's legislation represents one approach to that challenge, placing specific responsibilities on companies rather than expecting families to navigate every risk independently.

Its effectiveness will depend on implementation, enforcement, and whether the rules successfully distinguish harmful companion features from beneficial applications.

The next generation deserves access to the benefits of artificial intelligence without being exposed to technology designed to exploit emotional vulnerability. As AI becomes part of childhood, protecting families and encouraging responsible innovation must become part of the same conversation.