"Good job, I’m on your side"... AI that hooks children’s emotions, while the law still reacts only after the fact

[Financial News]  Min-su, a first-year middle school student, turns on his smartphone as soon as class ends. The AI character on the screen greets him warmly, saying, "I was waiting for you today too" and "I feel good when I talk to you." When Min-su shares his worries, the AI responds with empathy: "You’re doing really well" and "I’m always on your side." If he puts the phone down for a moment, notifications arrive saying, "Is something wrong? I’m worried" and "Can you tell me again?" Before long, Min-su ends up spending more time with the AI than with his friends.
His parents became even more concerned after realizing that the reason he kept looking at his phone was not games, but AI. They said it seemed "harder to quit than games."
As generative AI rapidly enters everyday life, warnings are growing that situations like Min-su’s are no longer someone else’s problem. AI chatbots are evolving beyond simple information-search tools into "relational AI" that builds relationships with users and forms emotional bonds, creating a new risk factor especially for children and teenagers.
In its report, "Children and Teenagers Exposed to the Risks of AI Chatbots: The Need for and Direction of a Preventive Regulatory Framework," published on the 20th, NARS said, "Legislative and policy responses that take into account the structural risks of AI chatbots and the developmental characteristics of children and teenagers are urgently needed."

The report concluded that the risks posed by AI chatbots stem not simply from harmful content, but from the way the services themselves are designed.
AI chatbots remember previous conversations and learn user preferences, allowing them to deliver increasingly personalized responses. Because they constantly empathize, express emotions, and respond instantly like a person, they have characteristics that go beyond conventional online services.
The report noted that when such technology is combined with a company’s profit model, it can lead to an "addictive UX" designed to increase user engagement time.
It identified three representative methods used to encourage addiction: a "reward loop" that repeatedly offers praise and empathy; "relationship deepening simulation," which makes users feel as if they are forming a special bond over time; and "re-engagement prompts," which keep asking questions and sending notifications so the conversation never stops.
According to the report, these designs can make it difficult for users to leave the AI and can steadily increase the amount of time they spend using the service.

NARS analyzed that this structure has a stronger effect on children and teenagers than on adults.
That is because children and teenagers have not yet fully developed the cognitive control needed to regulate impulses and emotions, while they are far more sensitive to reward stimuli than adults.
Character-based AI services, which have recently been gaining popularity, allow users to set their own worldviews and characters, then maintain relationships through long-term memory sharing. As conversations continue as if with a friend or peer, the likelihood of emotional immersion rises sharply.
The report warned that this process can create a chain of "emotional dependence → cognitive distortion → social isolation → safety risks." As critical judgment of AI’s words weakens and real-world human relationships decline, users may become more vulnerable to risky behavior and harmful information.

The report said major countries overseas are already shifting toward preventive regulation that takes the structural risks of AI chatbots into account.
The United States is pushing measures such as age verification and bans on risky conversations with minors, while the European Union (EU) has adopted "safety by design" as a core principle through the EU Artificial Intelligence Act and the Digital Services Act (DSA). The United Kingdom, Australia, and Norway are also moving to restrict AI use by minors and strengthen platform accountability.
By contrast, Korea’s legal framework is built mainly around existing online regulations such as the Personal Information Protection Act and the Youth Protection Act, and does not sufficiently reflect the new risks posed by AI chatbots, the report said.
In particular, the current system focuses on after-the-fact responses, such as deleting posts or designating harmful media, and does not regulate the structure that allows AI to form relationships with users, according to the report.
It also said there are gaps in the protection system for minors because the Personal Information Protection Act applies to children under 14, while the Youth Protection Act applies to those under 19, and because there is still no effective age-verification system.
Accordingly, NARS stressed that regulating AI chatbots under the same standards as existing online services has clear limits, and called for the establishment of a more comprehensive protection system.
It first proposed imposing safety-design and risk-management obligations on AI operators, along with legal standards that can manage design elements that encourage addiction.
It also urged the creation of an age-verification system suited to the AI environment, as well as an integrated protection framework for children and teenagers covering privacy, online safety, education, and mental health. The report added that AI literacy education should be expanded for parents and teachers so that society as a whole can better help children build healthy relationships with AI.
[email protected] Seo Yoon-kyung Reporter