Relational AI wins over people in their teens and 20s as ‘safety’ concerns grow with its popularity

[Financial News] As relational artificial intelligence (AI), which lets users talk with fictional characters and build emotional bonds, rapidly emerges as a new communication channel for people in their teens and 20s, questions are also growing over whether its safeguards are effective. Critics say measures to ensure safety are urgently needed as concerns mount over harmful conversations and content generation, as well as excessive emotional dependence, the longer users immerse themselves in the services.
■People in their teens and 20s left in an AI protection blind spotAccording to industry sources on the 12th, relational AI services such as Scatter Lab’s Zeta and Wrtn’s Crack are expanding their market share by focusing on conversations between users and virtual characters. Data released that day by Lee Ju-hee, a lawmaker with the Democratic Party of Korea, during the National Assembly Science, ICT, Broadcasting and Communications Committee’s audit of the Broadcasting, Media and Communications Commission showed that the AI character chatbot Zeta had around 110 million hours of monthly usage—more than twice that of ChatGPT. Nearly nine out of 10 users are in their teens or 20s.
Relational AI is characterized by enhanced interaction through conversations between users and specific characters they have configured. The more a service is designed to sustain conversations and deepen users’ sense of closeness to AI characters, the easier it is to attract users.
However, concerns are growing about potential harms, particularly among teenagers, such as treating AI-generated responses as if they were reactions from real people or becoming excessively emotionally dependent on AI.
At Zeta, sexually harmful conversations were generated even in an environment with safeguards intended to protect teenagers. The office of Hwang Jeong-a, a lawmaker with the Democratic Party of Korea, reviewed “Safe Mode,” Zeta’s environment for teenage users, and found that conversations depicting sexual violence and exploitation, as well as threats using them as leverage, were generated in scenarios featuring underage characters. A harmful-conversation detection model blocked around 200 problematic utterances during the review, but repeated attempts to bypass the filter resulted in some inappropriate responses being shown. There were also cases in which, even after a conversation was blocked, the AI suggested ways to evade the filter through phrasing and continued the harmful exchange. Scatter Lab CEO Kim Jong-yoon was selected as a witness for that day’s audit by the National Assembly Science, ICT, Broadcasting and Communications Committee but did not attend, citing an overseas business trip.
Following the controversy, Scatter Lab plans to introduce measures by the end of the year that limit daily default usage time to three hours for Zeta users aged 19 and under and set a default monthly payment limit of 100,000 won. It will also introduce more granular content age ratings and strengthen its harmful-conversation detection and blocking features. Starting next year, Wrtn will also set teenagers’ daily usage time at a maximum of three hours and reduce their monthly payments from 500,000 won to 100,000 won.
■Government and National Assembly signal overhaul of regulatory frameworkHowever, some say that time limits and payment caps alone will not fully address safety issues with relational AI. Since harmful conversations can occur even within limited periods of use, services need not only technology to filter harmful responses during content generation but also systems to detect dangerous contexts as conversations build up and counter attempts to bypass safeguards. Some also argue that services should accurately determine users’ ages and minimize efforts to evade restrictions by using different accounts or payment methods. In particular, harmful contexts such as sexual exploitation or threats can emerge as earlier and later parts of a conversation combine, prompting calls to prepare for cases in which users change their wording or make repeated attempts to bypass safeguards.
The government and National Assembly also plan to move quickly to revise the regulatory framework to close gaps in safeguards in the relational AI market. At that day’s audit by the National Assembly Science, ICT, Broadcasting and Communications Committee, Broadcasting, Media and Communications Commission Chair Kim Jong-cheol responded to a question from Lee Ju-hee, a lawmaker with the Democratic Party of Korea, who said measures were needed to prevent teenagers from becoming excessively immersed in AI. “I agree in principle,” he said, adding, “We will do our utmost to ensure that (measures to address the issue) are fully implemented through amendments to the Information and Communications Network Act, improvements to the AI Basic Act, consultations with relevant ministries, and other steps.”
[email protected] Jang Min-kwon, Choi Hye-rim Reporter