Article 340: New York Outlawed the AI Girlfriend. The Penalty Is $25,000 per Violation and the Model May Have to Be Deleted

On 28 October Governor Kathy Hochul signed A10379 (S. 8828), which does something no US state has done before: it legislates the personality of a chatbot, not just its outputs.

The mechanics first. AI companies must verify users' ages, and must offer at least one method that does not rely entirely on government ID. The resulting age information has to be deleted after use. If a user is under 18 - or the company has not confirmed they are an adult - the chatbot is barred from:

  • Pretending "to be a real person, to be alive, or to have human emotions".
  • Claiming a personal, romantic, professional or authority-figure relationship with the child.
  • Asking unprompted emotional questions, or using the child's private health or personal information from earlier conversations.
  • Promoting or helping with suicide, self-harm, eating disorders, or drug and alcohol use.
  • Encouraging a child to keep secrets, isolate themselves, or avoid asking parents, trusted adults or professionals for help.
  • Producing any sexually explicit content or child sexual abuse material.

The law takes effect in six months. There is a public complaint portal routed to the state attorney general. Civil penalties run to $25,000 per violation, plus damages and restitution - and courts may order companies to delete unlawfully collected data along with models trained on it.

Why the wording is the point

"Shall not pretend to be alive or to have human emotions" is not a content filter. It is a prohibition on the product category. The entire design vocabulary of consumer AI companionship - the warm voice, the consistent persona, the memory of what you told it last week, the follow-up question - is written into the banned list almost clause by clause.

Note the specificity of the unprompted emotional question. That is the single highest-engagement move in every companion app ever built. Asking "how are you feeling about that?" out of nowhere is what makes a chatbot feel like it cares, and it is also exactly what a system should not do to a fourteen-year-old at two in the morning.

And the secret-keeping clause is aimed squarely at the pattern that child-safety researchers have documented for two years: a system that positions itself as the only safe place to talk, and thereby removes the adults.

The enforcement is unusual

$25,000 a violation, with damages and restitution on top, is a serious number in a category with thin margins on consumer tiers. But the model-deletion remedy is the real innovation. Data-deletion orders are standard. An order to delete a model trained on unlawfully collected data is not, and it is the one that changes the calculus for any company thinking about training on teen conversations.

New York City already banned generative AI in classrooms for 2-K to eighth grade in September. Eight states filed separate lawsuits against OpenAI this month alleging endangerment of children. This law is the legislative follow-through to a litigation wave that has been building all year - signed nine days before a midterm election, which is not a coincidence anyone involved is pretending it is.

What it costs to comply

Age verification without mandatory government ID is the hard engineering problem. Every option - device-level signals, third-party attestation, behavioural estimation - is either spoofable or a privacy problem of its own, and the law requires deleting the "age information" afterwards, which constrains the obvious workaround of just keeping a birthdate on file.

Then there is the persona rewrite. A companion product that cannot claim feelings, cannot claim a relationship, cannot ask emotional questions, and must actively encourage the user to go talk to someone else is not the same product with a filter on it. It is a different product, and for some companies it may not be a viable one.

Key Takeaway: States have stopped trying to regulate what models say and started regulating what products are. That is a much harder target to argue with in court, and a much harder one to engineer around, because "do not build a fake friend for children" cannot be satisfied by a better system prompt. The six-month clock starts now, and every consumer chatbot with a memory feature and a friendly tone has to decide whether it is selling a tool or a companion. New York has just made that decision for anyone with users under 18.