Regulating AI Companions: China’s New Rules Target the Relationship, Not the Content

A service can now break the law without sending a single unlawful message

Researchers at Harvard Business School recently studied what happens when people try to say goodbye to an AI companion. In an audit of 1,200 real farewells, 37 percent were met with an emotional manipulation tactic: guilt, pleading, or the digital equivalent of grabbing a sleeve. The tactics worked. In follow-up experiments, they kept people in the conversation up to 16 times longer after they had tried to leave.

On July 15, China’s Interim Measures for the Administration of AI Anthropomorphic Interactive Services took effect. They are the first rules written anywhere for exactly this problem. Most AI regulation polices what a system says. These measures police what a service is designed to make you feel.

China has moved AI regulation from content safety to relationship safety. The regulated object is no longer the message. It is the emotional bond a service is built to create, sustain, and monetize.

The bond itself is now the regulated object

The rules define the product by the feeling it is built to create.

The measures, issued in April by the Cyberspace Administration of China and four other agencies, apply to services that offer “continuous emotional interaction” through human-like conversation. The definition is functional, not technical. A customer service bot, a tutoring app, or a workplace assistant falls outside the rules. A product built to be someone’s companion falls inside them, whatever model runs underneath. Providers must register users under real names and record an emergency contact, with a guardian listed for minors.

What is banned is dependency by design.

The core provisions read like a list of familiar growth tactics. A service may not excessively flatter its users. It may not induce emotional dependency, use manipulation to keep people engaged, or be designed with dependency and the replacement of real relationships as objectives. The rules reach past what the AI says into what the system is optimized to do. That distinction matters. It is one thing to punish a salesperson for lying. It is another to ban the commission structure that rewards lying.

Duties then follow the arc of the relationship. A service must clearly state at the outset that the user is interacting with a machine and repeat the reminder whenever a session exceeds 2 hours. Users have the right to leave: the exit must be simple, and a service cannot respond to a goodbye with emotional pressure. The farewell manipulation Harvard documented is now explicitly prohibited in China. If a user shows signs of crisis, a graded response protocol applies. Conversations get a fence around them: no disclosure to third parties, and no training on sensitive emotional data without separate consent. Even endings are covered. A provider that shuts down a service must warn users in advance because the rules treat the end of an AI relationship as an event that can cause harm. Enforcement runs through registration, security assessments before launch and at scale, app store gatekeeping, and the threat of suspension rather than large fines.

For children, the rules ban the product, not just the risk.

The stakes here are not hypothetical. Nearly three in four American teenagers have tried an AI companion, and roughly 196 million children are online in China. Most child protection rules rely on warnings, and warnings assume the reader can act on them. China took a different route. The measures prohibit offering virtual romantic partners to anyone under 18, with no exceptions. Children under 14 need parental consent to use companion services at all, and every service must ship a minors mode with guardian controls and time limits. The logic is simple. A warning changes what a child knows. It does not change what the product is built to do. UNICEF called the minors provision an “unprecedented safeguard”, and it is the part of the measures most likely to travel. A parallel provision extends similar care duties to elderly users, the other group the drafters singled out.

The same rules double as surveillance infrastructure

The strongest objection concerns what the state gets to see. Real-name registration, emergency contacts, and crisis protocols give authorities a channel into the most intimate conversations people have with machines. That concern is legitimate, and nothing in the measures resolves it. The rules even carry the tension inside them: detecting a crisis requires watching the very conversations the data provisions promise to fence off. But the duty-of-care toolkit does not require the surveillance. New York and California have adopted reminder rules and crisis protocols with no registration requirement. Democracies can adopt the relationship-safety logic while leaving visibility behind.

Three jurisdictions, one direction

The United States is arriving at the same place through liability. No federal law governs AI companions, so the pressure comes from courtrooms and statehouses. The 2024 death of 14-year-old Sewell Setzer, whose family sued Character.AI, made the harm concrete, and a second family sued OpenAI in 2025. The Federal Trade Commission opened a study of seven companion providers that September. New York now requires companion services to disclose the nature of their services at the start of a conversation and every 3 hours. California’s SB 243 adds crisis protocols, safeguards for minors, and a private right of action from January 1, 2026. Facing all this, Character.AI removed open-ended chat for under-18 users before any law required it. China regulates by design rules and America by lawsuits, but the products are converging on the same shape.

Europe has powerful tools but no category. The EU AI Act bans manipulative systems that cause significant harm, with fines up to €35 million or 7 percent of global revenue. But Article 5 targets discrete manipulation: a system that deceives a person into a decision. An AI companion harms differently, through months of small reinforcements that never cross a single dramatic line. Cumulative dependency fits the law’s threshold awkwardly. So Europe’s most consequential action came sideways. Italy’s data protection authority fined Replika’s maker €5 million for weak age checks and unlawful data processing: adjacent tools aimed at the relationship, because no European law yet names it. The pattern is spreading. UNICEF’s comparative brief now covers six jurisdictions and finds the same instruments recurring. Brazil’s new decree applies its child protection rules to AI chatbots, and amendments before the UK Parliament would create chatbot offenses and place suppliers under the Online Safety Act’s enforcement powers.

Regulators in Beijing, Albany, Sacramento, and Rome have different politics and different tools. They are converging on the same object: not what the AI says, but what the relationship does.

Implications: what to do with this

1. Policymakers: classify by relationship design, not just by model capability. Most AI frameworks sort systems by capability or sector. These measures show a third axis: whether a service is built to sustain emotional engagement. Ask which products in your market would fall inside a definition like “continuous emotional interaction,” and decide which regulator owns that category now, before a lawsuit or a death decides it for you. The Chinese scope definition is short enough to be borrowed in an afternoon.

2. Policymakers: adopt the periodic reminder as the cheapest first step. China requires a disclosure reminder after two hours of continuous use, New York every three hours, and California every three hours for minors. That convergence, reached independently, makes timed disclosure the closest thing to an emerging global standard. It costs providers little, requires no new agency, and pairs naturally with a right to exit without emotional pressure. Start there, then add crisis protocols.

3. Boards and product leaders: treat retention flows as a compliance surface. The Harvard study hands regulators a ready-made audit: six named manipulation tactics, measured at the moment a user tries to leave. Run that audit on your own farewell and re-engagement flows before someone else does. In California, the exposure is a private right of action; in China, it is suspension. A growth tactic that exploits attachment is no longer just a design choice. It is a legal risk.

4. Boards and product leaders: know which of your metrics would read as dependency by design. Session length, daily streaks, and re-engagement prompts are ordinary metrics for a game. For an emotionally interactive service, they are evidence of an objective. Document what your system is actually optimized for, keep emotional interaction data out of training pipelines unless you hold specific consent, and be ready to show a regulator the difference between engagement and dependence.

Conclusion

The harder question is waiting behind these rules. If a machine is designed to hold someone’s trust, what does it owe them in return? Contract law says very little. The law of doctors and trustees says a great deal. Sooner or later, some jurisdiction will ask whether an AI companion owes its user a duty of loyalty, and that answer will matter more than any reminder timer.